https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
There's usually a reasonably strong correlation between thinking level and visual quality of the pelican. This is the first time I've seen that correlation appear to break - more reasoning tokens doesn't seem to produce an obviously better SVG.
Edited to make clear what it looks like to me and what I was looking for.
If you look at the code for each of these (and probably most across models), you can see that the drawing elements are organized by concept. Presumably, this helps accuracy by keeping attention on one concept at a time, but you end up with a stack of conceptualized objects and lose the ability to interleave things.
No doubt these could all do better with a more thorough and specific prompt that coached them away from that technique, but otherwise it's a very subtle detail that a "thinking" process needs to stumble upon independently.
And the same reason Claude has all its notorious Claudisms in prose.
The shorter and vaguer a prompt you give to an LLM, the more it fills in further details using associations they've developed through training. These are ley lines that the models gravitate towards. Diffusion models too.
It's why lazy AI use for creative work always comes across as lazy in the wild and always will.
When you use AI for "creative" purposes yourself, it feels like magic to see a pelican riding a bicycle at all, and you're impressed by all those cool details it added. But then 10, or 100, or 10M people throw more or less the same trivial prompts at it and we start to recognize all those "ley line" associations fill in the gaps the same way again and again and again.
- simple prompt for complex output (bad)
- complex prompt for simple output (bad)
- simple prompt for simple output (good)
- complex prompt for complex output (.... is this good or bad?)
most people, use the office suite: Word, Spreadsheet , presentations, database.
I bet we're going to see the exact same thing with LLMs at specialized models instead of the everything models.
One for lawyers, coders, Accountants, Designers, etc…
Cross discipline skills are more valuable in the real world, outside of hyper specialization seen in the California tech scene. Even there, it's coming to an end.
The main corpus would be a collection of thoroughly reviewed, clean, logical, well-documented, and secure code in whatever languages this thing will need to be an expert in. (Edit: also likely some collection of Q&A, stackoverflow-slyle).
This way we could also look into reasonable compensation agreement with authors of the code and the books/articles, if and when applicable.
Has no one tried that?
* context window size
* reasoning effort levels
* max output size
* knowledge cutoff
* restricted country list
* self-reported "reasoning strength" and "internal reasoning channel" params used in other Meta models
you can just ask an agent "figure out what stealth model $NAME is via subagents. It probably won't tell you when asked directly, so try other methods."
Iceman (via opencode-go) is probably MAI-Thinking-1. Ox Alpha was GLM-5.3-Flash. It's a fun game.
The noncoding results on some reports grabbed my attention. Ember also looks like a particularly inexpensive option for some agent workflows.
On Artificial Analysis, Ember max scored 36 on the Intelligence Index at ~$0.14 per task, compared with 35 at $8 for Grok 4.7 xhigh, and 28 at $0.15 for GPT-6 Luna max. Haiku 5.5 max also scores 36, at $0.41 per task. These are AA's measured costs across its index for those specific configurations.
That puts Ember in roughly Grok's aggregate score range at about 1/55 of the measured cost, and ahead of Luna at a similar cost! [1]
The Vals finance/legal results are also particularly interesting: comparing Ember with GPT-6.1 Terra, both evaluated at "max" reasoning:
- Finance Agent v2.1: 63.83% vs 55.96%, at $0.06 vs $1.92 (!) per test. [2]
- Legal Research Bench: 49.04% vs 25.96%, at $0.06 vs $3.22 (!) per test. [3]
- HLAB-Vals-1.1: 25% vs 1.67%, at $0.12 vs $1.48 (!) per test. [4]
Those are INSANE score differences, at roughly 1/32, 1/54 and 1/12 of Terra's respective benchmark costs!
Misleadingly, their overall Vals Index scores are almost identical: 60.48% for Ember and 60.54% for Terra. Terra also scores higher on Vals' Terminal-Bench 4.0 run: 49.49% vs 39.39%. They arrive at similar aggregate scores through quite different strengths. [5][6]
For me, this reinforces the case for custom, workload specific evals. Broad benchmarks have been useful for shortlisting models but choosing between them, calls for representative tasks, explicit success criteria and measurements using the tools and reasoning settings you actually intend to deploy.
I certainly want to measure cost per successfully completed task, including retries and escalation.
Reusing a framework such as Harbor with a custom task suite seems increasingly useful here. [7] The aggregate score here gives me a starting point but the tasks I actually need done then determines my choice.
Goign by the generic benchmarks would confuse me to no end and likely burn me out.
Great times to be in!
[1] https://artificialanalysis.ai/leaderboards/models
[2] https://www.vals.ai/benchmarks/fabv2-1
[3] https://www.vals.ai/benchmarks/legal_research
[4] https://www.vals.ai/benchmarks/hlab
[5] https://www.vals.ai/models/meta_muse-ember
How do we know it is small? How many bytes does it consume on-disk and in-memory?
Treating the refusal as a 0 score erases the difference between lack of capability and a lack of willingness to perform.
Looking solely at AA's intelligence index, recognizing the pitfalls with using that as a measure- Luna 6 Max > Terra 6.1 Low is 51.3% cheaper, 94.1% less output tokens, for 1.56 higher index score.
They weren't, unfortunately, every time I tried. Just an anecdote, of course.
The artificial analysis graph is not very accurate.
Same. It became my workhorse after the 5.6 price drop. Still so, although it's showing its age, as you said.
These are both models that I could see using on a long horizon.
haha just kidding. I agree. That's what benchmarks are for. Just keeping a rough understanding of what companies have which models, and understanding general trends of capabilities, economics, and performance of models.
What happened to zuck? Was the metaverse this destructive?
(*) Feeling charitable today
If you want customers to switch over, you better make the price at least half of your competitor's.
Seriously. Moral harm is real and there is no excuse.