It's especially relevant now that models are good enough to solve ~most point-in-time problems.
Some relevant but disconnected thoughts:
- deterministic scores are so nice
- what "maintainable" is is probably some high dimensional space described by these signals; it'd probably require some human labeling to figure out where this space is
- another signal I've been thinking about and I'm seeing increasingly get brought up is the state space of a system; I'm seeing formal methods pop up a lot recently
State space of a system AND the way to make it accessible / visible to a model. Many times a model can work magic if it can "see" the state of a system in a way that suits it. That's why sometimes having a cli added to the environment seems like such a big unlock. Because that cli usually takes a complex state and allows visibility into it, and possible manipulation in a structured way.
this is a nicely succinct way to put this - a multi-dimensional space where no single metric is really useful
state space of the system is interesting too. I would guess that for any production software with dependencies like databases/third parties that might be too hard to measure, but if you can silo off parts of your system into bounded state machines, it may be a value metric on some module behind a clean interface.
I think the kubernetes control loop model is a great instance of this, a handful of scoped components that own a control loop across a well-defined state machine, that can operate / recover in the face of most network partitions or downtime - the promise of CRDTs but rather more a pragmatic approach to it
I’ve now replaced my use of Opus 4.8 xhigh with Opus 5 medium, and I’m using less tokens and it’s quicker. I can understand people being annoyed by its writing style but for getting work done that really doesn’t bother me. I’ve been really enjoying using it.
It seems like there’s a week by week and sometimes day by day change in performance when on a subscription plan using their harnesses.
Quality vs cost - medium is the sweet (perhaps better too!) spot.
I have a suspicion that most models will miss the `database_migration` Checkpoint 2 test that includes a `default_value` because it could be interpreted as either a JSON-literal or a SQL-expression.
There might be other tests as well that are prone to failure for reasons other than the reasons cited in the paper.
I think a cool experiment would be to adjust the order of the features implemented (e.g. checkpoint 3 then 2 then 5 then 4) where dependencies allow it. Then one could account for some checkpoints being more difficult than others.
i will look into how easy it would be to zip up some subset of the results without leaking anything...probably doable
I have found that models are generally poor at managing refactors / complexity while also implementing new features. But I’ve had some success with a semi-lights-off approach where you decompose it and prompt the model adversarially in a second pass to look for new rough edges and areas of complexity or refactors that might simplify the codebase.
So I’d be very curious to see this benchmark but with something like a periodic “refactor turn” interleaved in.
Also eager to see Fable benchmarked; anecdotally that was the only model whose code I felt I could actually trust to not review closely.
a 41% improvement is not much higher? come on that's just doomer
I'm hoping to do some more work with sol/fable in the mix as well as exploring more languages and curating the problem set to include more of the benchmark
I also kinda felt like opus4.5 was dumber than 4.1 personally, maybe a little biased since 4.5 was 2.5x faster and 2.5x cheaper seems to indicate its a smaller model
You ever go to forums full of entomology specialists and tell them you don’t understand their fancy terms?
If a model isn’t a step function change? Welcome to research.
> fetch this article for slopcodebench and help me run an eval on a subset of problems with opus 5 https://arxiv.org/html/2603.24755v1 > Get all the context, fetch any mentioned repos, and then propose a plan to me.
> i have an anthropic API key in .... > Let's do the three challenges with Opus 4.8 and Opus 5 and Fable please. I like your minimal set. Let's try it. What do you need from me?
> Actually I changed my mind. I want to do two of the easy ones you picked and then I want you to pick the one with more checkpoints, maybe one of the harder ones, not the very hardest one but one with a higher number of checkpoints.
> Actually let's do one easy, one medium, and one hard problem please. If we have a hard problem I'd like to see that.
> lets rock - i think lets just do opus 4.8 and sonnet 5 and opus 5 since we our ZDR will block fable
For readability specifically, I've found it hard to get the models to do this with prompting. If you've talked to opus/fable for a long time on prose writing you probably felt this too
At the same time, I imagine it will be hard for them to prioritize this over improving flashy one-shots of impressive zero-to-one feats that demo so well and attract more customers.
Can we have simonw make "pelican on a bicycle after 1000 requests for iteration" popular?
yes the labs will always prioritize the vibeslop dopamine casino as far as I can tell - making the models useful and addictive for unsophisticated users, sometimes at the expense or at the very least at the ignorance of the needs of power users
I did find opus 5 quite handy for general knowledge work and visual design, without the cost of fable (e.g. the graphics in this post are made by opus 5)
but its not noticeably better than opus 4.8 in those regards, and I would not miss it if forced to go back to 4.8
I'm not ready to blame Opus 5 for being stupid. Perhaps we have a prompt buried somewhere that's essentially asking it to be pedantic, and it's just obeying the prompt.
This is a known failure mode. Sadly, working on a software engineering team doing agentic engineering now means we need to build and maintain suites of evals that measure these things, so that we can measure the effects of changes to harness components including how they perform under model upgrades.
But... for a traditional software engineering team that has no experience in this... How do we even do it?