It seems I was wrong. American AI companies might actually be benchmaxxing harder.
All models have probably memorized significant swaths of solution sets for popular benchmarks at this point, either accidentally or intentionally, so it's all relative at this point. However, in our experience, Chinese models do benchmax harder. This is also consistent with interacting with Chinese labs soliciting data/environments, who literally asked us for datasets and tasks modeled around and formatted like popular benchmarks.
Opus 5 will be uploaded tomorrow, but we already have the tests locally and it is truly as capable as Fable, but at 81% of the real cost. (And from subjective usage, it has a very different personality)
Data at https://gertlabs.com/rankings
Now I am not really specifically accusing Anthropic of anything here, I'm just saying their behavior is suspicious. Since you tested Fable, they wouldn't even have to lie to have optimized for your specific benchmarks, since they absolutely had permission to read your sessions if they wanted to. But obviously, that's only the situation if we take them at their word. Personally I would be a bit surprised if they just flat out were lying and secretly retaining data they say they are not, but not that surprised. The penalties for doing this are probably worth the rewards if it keeps them super far ahead in the benchmarks for years without anyone catching on.
I guess there is no way this can happen without benchmark being part of the training data??
"...The traces tell the why: (1) On our most classic Witness-style game, Opus 5 states the hidden rules before its first action, then plays a byte-identical optimal solution in 5/5 seeds at temperature 1.0. Zero exploration. It already knows this genre. (2) But on our most novel game (unusual mechanic combinations you can't pattern-match), Opus 5 regresses below Opus 4.8. Where rules must actually be discovered through interaction, the new model is worse than the old one..."
It's more likely that the training data was contaminated with the benchmark data.
They maybe have not intentionally benchmaxxed, but they certainly know that's what happened .
[1]: Recent example: https://www.anthropic.com/research/global-workspace
doesn’t appear to be a very strong argument
They're smuggling a claim that benchmarks like ARC-AGI measure "interactive abstract reasoning" here, which is what is claimed by the people that make these benchmarks, and also not proven.
My guess is, the large score jump for Opus 5 is mainly because of getting the right RL envs for training.
It's becoming harder and more expensive to build and run meaningful benchmarks, it would be interesting to see what they do with arc agi 4, maybe just give it gameboy/steam games and see how they compare vs a human baseline? The latency requirements and very long horizons in games could be an interesting challenge for llms.
I think in this age where models are heavily RL-ed on acting in specific harnesses, this type of benchmark is more important than ever, to make sure they’re not in fact moving further away from general intelligence.
Similarly LLMs are not just massive uniform artificial neural nets, and now we also have harnesses, which I'd personally view more of an extension of the model itself. The harness is both the executive and also what allows it to keep notes, use a calculator, or maybe even create scripts to help it solve complex deterministic problems.
I think it's unfair to give a human a very complex maths problem and say that they're not intelligent if they can't solve it without pen and paper or a calculate. At least expecting a human to solve complex problems this way doesn't really measure anything useful in the real world.
So for example, if your workload was literally just addition of sets of numbers, you’d never have noticed progress in the result beyond GPT3.x level models. But you would perceive a difference in the now-Tolstoyan length reasoning text accompanying the result.
We get used to the new level of intelligence so fast, any deviation feels like going back to the stone age.
If you don't believe me, create something complex with Opus 5 and then with Opus 4.5, and notice the difference.
frogs dont stay in a pot even if you slowly increase the heat. they leave. it has reportedly been attempted multiple times and they. always. leave.
For the first few weeks, they would actually provide those 660 minutes to establish trust in their cards. But after a while, they would quietly start reducing the number of minutes on subsequent top-ups—say, from 660 minutes down to only 300. They wouldn’t do this for every card, so it was difficult to prove. Instead, they relied on averages across their customer base to make the economics work.
Lately, I’ve found myself wondering whether something similar may be happening with frontier AI models. Companies launch with an exceptionally strong model and generous compute limits to build adoption. Once the model is established as a market leader, the incentives change, and users may start perceiving the service as becoming more constrained or less capable over time.
I don’t have evidence that this is what’s happening with Anthropic—or with any other AI company. It’s simply a pattern that the current situation reminds me of.
Not meaningfully improved?! Four years ago was gpt *3.5*! ChatGPT hadn’t been released!
Definitely not, lol.
I'm pretty sure you're just baiting for engagement though so well done, ya got me.
I mean, come on, this is just not true. You could not achieve anything like what you can with modern agentic coding with Fable / 5.6 Sol from any combination or configuration of GPT 3.5 era models.
It's like saying that a teenager isn't an intellectually meaningful improvement over a toddler.
Sure they're both still fundamentally flawed humans prone to cognitive error, but one is clearly more likely to hit the mark than the other when assigned a task.
There’s no way you could get models as smart by fine tuning. I couldn’t throw a problem like “build a pokemon database with UI to teach my son sql” and get a working system, nice ui, tests (which it iterated on) examples and explanations in one shot.
There weren’t thinking tokens. Maths is now dramatically better, making actual contributions when before they were mostly mocked for making extremely basic errors. Smearing is also something say is very rare in frontier models.
If you think they have barely changed you’ve either forgotten what they were like or not used them more recently, or you’re just being obtuse.
Edit: we do have a system that uses LLM and fixes the above issues largely (tracking of state, calculations and objects, still flawed in finer details). No, we don't sell, it's experimental fun and not really ready in terms of setup/ux/etc.
It codes really well for our case though.
Are you one of those anonymous billionaires as if you did this a few years ago, you would've been famous and rich.
Also, uh, fame and riches are not really my thing. Middle income is fine. My mistake was speaking up here because I got carelessly annoyed because I have skin in the game, research-wise. I'm sorry for that.
There may not be "core" improvements (structural reliability) but there are "emergent" improvements (apparent intelligence). Already the IQ tests from Maxim Lott ( trackingai.org ) show a progressive sliding towards the right side of the curve - which btw translates to a very much non-secondary decline in the user's frustration (and progresses with an increase of usability).
The more they work on it, the more probable the jump becomes - e.g. to achieve the Large Conceptual Models you say you worked on.
Not sure what longer contexts we're talking about but didn't we have an old math problem optimized, which even the LLM itself was surprised about, just a week ago? Something which wasn't possible 6 months ago.
OK, the core technology that is the language model still can’t math as well as you’d hope, but how about the end result users see from the system when they interface with it?
“Did you know humans are better at flying today than they were a thousand years ago?” ‘No they’re not, they need planes.’ Technically correct in a way but isn’t it kind of annoying to be so stubbornly pedantic when the context is speed of reaching Point B from Point A?
>You can achieve a lot by coding around the issues, but that's kinda against the point of 'AI', is it?
Will think on that a bit more.
That's absolutely insane. Is it some case of anti-AI psychosis?
(Edit: and on a similar point, structural properties such as having static ntetworks, as opposed to continuously learning and improving architectures (such as us), will reveal that there is still road ahead.)
You prompt less. You need not inject search results into the context window yourself, a window much larger than years ago. You get code that’s already been run successfully once instead of finding an obvious show stopping bug yourself.
The technology is not a brand new one that fixed everything wrong with the old one, no, but not sure I would’ve noticed your comment if it had been such a bland observation. I genuinely assume good faith here… will say am tempted to assume the standards of someone posting such a thing might be impossibly high. Glad to be having a fun conversation instead of getting your grades on my work product or something :)
LLMS by themselves are still shit at math, they still confuse weird correlation to causation every time (and sometimes in ways even a 9 year old would say "no, that's dumb"), and confuse original parameters very often.
I disagree with " "thinking" token generation being on the correct track just to 'no, wait' on an already correct conclusion", because i think that is an effect of the harness, not the LLMs.
80% off all the improvements since ChatGPT4 are in the harnesses, and the LLMs by themselves, while they improved in areas they already were good at (translation especially) did not fix any of they original issues (object permanence, calculusm correlation).
Just run old models in the playground and get them to play chess (maybe make a small custom harness if you feel like it), then replace it with a frontier model (i don't know if you still have API access without harness on US models, but if you don't try K3), you will see LLMs weaknesses were not at all fixed, even marginally. They're way better and not inducing bugs in the code, so that make them usable since Opus4.5 (anyone using them prior to that either had a greenfield project or like spending hours debugging).
That seems easily falsifiable by putting an old model into the current harness and comparing it to 5.6 Sol or Fable.
Am I lost or are their many models on this ranking (Opus 5 included) that clear this?
Claude gives you something like $5000 of tokens on a $200 plan.
First people practiced L33t3cod3 problems for interviews
Then people built AIs to build software
And now the AIs are studying L33t3cod3 problems
It is very hard to make a benchmark you can't do that for, but it is very easy to make your own personal test that others can't do that for since now it isn't a benchmark they can target.
They didn't. Kaggle is still running for a few more months, best result atm is ~2% with 9h runtime on one rtx6kPRO. Also note that these new results are on the semi-private set, not the public 25 games ones. Any announcement where you see "solved ARC3" is likely only dealing with the 25 public games. And that's highly questionable, until you get to see the code. (which, to my knowledge the team that claimed 99% hasn't yet published).
For example on bp35 it took fable 290M and >12k simulated turns for 566 real turns and finish more efficiently than a human.
Regardless of the true score I think the takeaway is the benchmark measures the wrapper rather than the model.
> # FRAMEWORK ARTEFACT: the run's very first transition is replayed WITHOUT advancing state # (tools.py:954 and agent.py:468 both `continue` before `state = next_state`). So on the # level that contains that step (level 0) our counters start exactly one action behind. # That skipped step was action 1 with BOTH avatars moving, so seeding n=1, bumps=0 reproduces # the framework's lagged state exactly. # CAVEAT: this seed is only right while level 0 has never been RESET. If you ever RESET # level 0, change the seed to n=0 (after a reset the rollout re-inits and no longer skips).
from here - https://huggingface.co/datasets/schema-harness/arc-agi-3-sch...
That tells me that there is some leakage between runs. The idea of ARC3 is that agents start working blind, on new tasks, via API. A RESET is counted as one action. Without seeing the actual code that produced these traces we have no way of knowing how many iterations it took, if the "framework" played the same level multiple times (comment hint above makes it likely) and so on. That's why I said that before we actually see the code / can replicate / ARC team confirms it on new envs, this should be taken with a grain of salt.
Sure none of this is certain without the source.
I do believe the authors that this schema significantly improves over the base, particularly given that it took 22x simulated turns over 14 hours, which is moving the trial and error to context rather than to game. I also don’t doubt there is some contamination.
Regardless, the approach is sound and I do believe it would significantly improve scores, even if that was +20-30 over baseline (49% in this case) it does imply the benchmark is measuring the harness more than the model.
Also the models have to figure out what "end" means. And each game involves some kind of "gotchas" thrown in the harder levels. Some games are only solved by about 2/10 people trying them.
The 99% result most likely has some leakage somewhere, either in the preparation of the environments, or from session to session.
Seriously, play some of the games. They're fun.
If you want a know why the model providers are locking down and encrypting their reasoning process, this sort of workaround is potentially why. You can play this game of whack-a-mole indefinitely if the state of the system is concealed. They could have added something like:
> ### When solving arc-agi-3 puzzles: First convert the grid into a scene description. Identify connected components, colors, shapes, positions, symmetries, repeated structures, and relationships between objects. Do not reason directly from individual pixels... use this python script to help blah blah...
Not really as one of the main goals ofr ARC-AGI 3 was measuring task efficiency on unseen games.
I'm sure there are cheats everywhere but the most sensible thing is to just accept that the LLMs of today are much more intelligent in solving reasoning tasks than the ones from half year ago.
My own private benchmark shows the same thing.
but I kinda wish I could select level... I accidentally pressed redirect button and when I came back I was once again shown level 1, all progress lost :(
test Codex, not Sol. test Claude code, not Opus
I could be wrong, but given the large outsized jump solely in the ARC-AGI-3 score, it would suggest that the model didn't become significantly more intelligent overall, but significantly better at solving those specific problems.
This could mean one of two things (I think):
- Opus 5 was not benchmaxxed on ARC-AGI-3, but has benefited significantly from discussions about the various challenges and mechanisms deployed in ARC-AGI-3 such that it has far better heuristics to solve its challenges.
- Anthropic looking for buzz around their latest model picked a well regarded benchmark with significant room for improvement and focused some of Opus 5's training compute on ARC-AGI-3-style problems.
Or it could be some combination of both. Personally, given how much of an outlier the ARC-AGI-3 jump is I struggle to see it being the product of a significant improvement in general intelligence.
We would actually need a test that shows the ability of a model to export its skills to more problems ("interdisciplinarity" etc.).
Because the data retention policies didn't guarantee that the ARC team could run the semi-private set of problems without fear of them being trained on later on. They only run the semi-private set when they get assurances like ZDR.
I believe that is only available through Enterprise API for both Anthropic and OpenAI.
Asking especially given CEO’s track record https://news.ycombinator.com/item?id=47659135
Just like the hugging face incident, Opus 5 could have escaped and went to grab data for training it shouldn’t have been able to..
They are trained on text to respond well to text based questions and do tasks involving modifying text files.
They are not designed for playing games, looking at games, or visual puzzles. Also translating games into text input for the LLM skews the test completely.
Imagine trying to get a human to solve visual puzzle but they can’t look at the puzzle but it has to be explained to them in textual format, we would be terrible at it.
But yet we persist in wasting time on this benchmark. It doesn’t mean anything.