I think with the right problem and the right agentic loop it’s clear to me improvements will speed up.
Four top logical people in the world are acknowledging this. It is mind-blowing and we don't know why.
Several people had problems with Sonnet burning through all their credits grinding on a problem it can't solve. Opus fixes this — it has a confidence threshold below which it exits the task instead of grinding.
"I spent ~$100 last week testing both against multiplication. Sonnet at 37-digit × 37-digit (~10³⁷) never quits — 15+ minutes, 211KB of output, still actively decomposing numbers when I stopped it. Opus will genuinely attempt up to ~50 digits (112K tokens on a real try), starts doubting around 55 digits, and by 80-digit × 80-digit surrenders in 330 tokens / 9 seconds with an empty answer." -- Opus, helping me with the data
The "I don't think this is worth attempting" heuristic is the difference. Sonnet doesn't have it, or has it set much higher. In order to get Opus and some other models to work on harder problems that it assumes it is not worth attempting, it requires an increase of confidence level.
I'll finish writing this up this week. I'm making flashy data visual animations to make the point right now.
The Universe seems against free lunches… if AGI is possible, finding a manager good enough to get the AGI to update its timesheet will not be (in practice).
Originally LLMs would get stuck in infinite loops generating tokens forever. This is bad, so we trained them to strongly prefer to stop once they reached the end of their answer.
However, training models to stop also gave them "laziness", because they might prefer a shorter answer over a meandering answer that actually answered the user's question.
Mathematics is unusual because it has an external source of truth (the proof assistant), and also because it requires long meandering thinking that explores many dead ends. This is in tension with what models have been trained to do. So giving them some encouragement keeps them in the right state to actually attempt to solve the problem.
So which one are they?
Models are trained on human outputs. It’s not super surprising to me that inputs following encouraging patterns product better results outputs; much of the training material reflects that.
The answer is probably more straightforward than we think, e.g. “the user thinks I can do this so I better make sure I didn’t miss anything”
It's pattern matching on training material. There is almost certainly an overlap between positivity and success in the training material. Positive prompts cause the pattern matching to weight towards positivity and therefor more successful material.
Nice language often sorta does this for whatever model(s) they looked at, and is also something people are likely to try. Probably lots and lots of nonsense token combos would work even better, but who’s gonna try sticking “gerontocratic green giant giraffes” on the end of their prompts to see if it helps?
Positive or negative language likely also prevents pulling the probabilities away from the correct topic, being so generic a thing. The above suggestion might only be ultra-effective if the topic is catalytic converters, for some reason, and push the thing into generating tokens about giraffes otherwise. How would you ever discover the dozens or thousands of more-effective but only-sometimes nonsense token combos? You’d need automation and a lot of brute force, or some better way to analyze the LLM’s database.
Nevertheless not having a sense of time makes it really bad at planning anything. I used Gemini Pro.
I've seen this mentioned a few times though, so I think maybe it's more complicated than this?
https://tech.yahoo.com/ai/chatgpt/articles/chatgpt-fails-mis...
This is an ad for the upcoming IPOs because AI fails everywhere. Including in math, where there are no serious results and just the people who already loved tinkering with Lean before AI jump on the bandwagon.
Maybe mathematicians have looked jealously at software engineers who can survive by just tinkering with new tools all the time without making real progress and think: We can do this, too!
and https://www.simonsfoundation.org/about/ has "Since its founding in 1994 by Jim and Marilyn Simons"
https://en.wikipedia.org/wiki/Jim_Simons explains how Jim Simons got rich.
The book 'The Man Who Solved the Market' - https://www.gregoryzuckerman.com/the-books/the-man-who-solve... is a nice read.
HN discussion on a review of the book - https://news.ycombinator.com/item?id=29392041
I think AI-assisted research will likely have a very negative net impact on mathematics in the long run by lowering the average level of understanding within the community.
Also, research directions are influenced by what people can solve, and this will slowly shift research toward purely algebraic/symbolic manipulations that mathematicians no longer fully keep track of.
> When Ryu asked ChatGPT, “it kept giving me incorrect proofs,” [...] he would check its answers, keep the correct parts, and feed them back into the model
So you had a conversational calculator being operated by an actual domain expert.
> With ChatGPT, I felt like I was covering a lot of ground very rapidly
There's no way to convert that feeling into a measurement of any actual value and we happen to know that domain experts are surprisingly easy to fool when outside of their own domains.
> “2025 was the year when AI really started being useful for many different tasks,” said Terence Tao
I think I’ll go out on a limb and agree with Terrence Tao, I think the dude is well known in the math community, or something
Is AI his specialty?
> I think the dude is well known in the math community, or something
I believe this is called "appeal to authority." Which is why, instead of disagreeing with him, I suggested a more cogent endpoint that could be used to establish the facts the article's title suggests.
We cannot build one.
AI outputting axiomatically valid syntax isn't going to be all that useful. It's possible to generate all axiomatically correct math with a for loop until the machine OOMs
Physics is not math and math is not physics.