Given that the decrease in their margin and the fact they delayed the release of Grok 4.7 almost two weeks past the original date, XAI must not have been happy with the results for 4.7. And XAI also waited the day before Opus 5.5 is rumored to launch. I imagine Opus 5.5 will blow Grok 4.7 out of the water benchmark wise.
However, I have become skeptical of benchmarks. Grok 4.5 solved some issues setting up a buildroot system that Fable 5 couldn't do. I find the post cursor groks are phenomenal at frontend web development, though Claude is much better at backend ruby.
My favorite part of the new Groks has been how they speak in plain english. I simply cannot stand Claudish. Or even GPT, which doesn't have Claude's ticks but definitely likes to handwave explaining technical concepts. Still, nothing beats Claude 3.5 and 4 with explaining since it seems all models have regressed. I wonder if Grok 4.7 will also regress with English because of all the RL.
Grok has its own feel too. It's not as bad as Claude, but one of the things that bugs me is that it is far too terse.
It regularly seems to come up with terms and descriptions for things in its chain of reasoning and then uses these terms in its output assuming you understand what it's talking about.
I find I often have to ask it to re-explain what it means.
First, after a while it's just as grating as Claudeish. Second, my hunch is that it constricts the actual thinking of the LLM, like the same way that Newspeak does in 1984. It shrinks the range of thought that can be expressed if used as an input.
I think the real way to do it is to have another Claude entirely deal with the user as a liaison, but to keep the thinking in whatever format it came in.
Latent space reasoning, if you think about it, is exactly this to a crazy degree: why even formulate a thought as words if you can just keep it as matmuls until the user needs it? And then, if the user needs it, have it always specifically formulated for the user by another LLM rather than constrict its range of thought? Anyway, that's my take.
I do think an infrastructure where another Claude retranslates the output would be better. Oftentimes I forget to put it in the actual prompt and when I receive back 8 paragraphs of Claudeish I ask for it then.
I would have to disagree that it gets as grating as Claudeish though. Its just direct and professional instead of ring-around-the-rosy clickbait.
1. Is natural language holding LLMs back by some %? 2. Is natural language serving as a hard gate that will prevent LLM intelligent progressing past some specific point?
The answer to 1 seems like an obvious yes to me.
Your thesis says the answer to 2 is "yes." That doesn't feel right to me. Think about all of the humans who have pushed various fields forward: Einstein, Newtown, Bach, whoever. If natural language doesn't prevent an entity from surpassing humans in one intellectual field, why would it prevent an entity from surpassing humans in all intellectual fields?
(To be clear, I'm not claiming superintelligence will or won't be achieved; I'm considering your specific thesis about whether or not natural language will be a hard gate)
I don't think this is true.
They have to express themselves as tokens. The meaning of those tokens doesn't have to be text. See any model that can handle images/video. Also, I don't think math, svg, etc, are "natural" language.
And, only the final expression is tokens. The intermediate layers, with the encoded concepts, aren't "natural language".
But, to address your concern (which nobody can disagree with, since even humans can't fully express through text/pictures), potentially: https://news.ycombinator.com/item?id=49758615
The model isn't limited to concepts that can be expressed in natural language.
It's only once the AI gets to the output layers that natural language comes back into play.
After all, they're all made out of weights[0].
By the way, how good is Claude's Hopi?
It burns more tokens but is the only way to get tolerable text.
https://code.claude.com/docs/en/hooks-guide#agent-based-hook...
Literally every one, even 1-2 prompts later it starts to go back
That being said, I currently prefer Sol / Astra to Opus / Fable as I find both to be a better cost payoff to me.
I totally agree, it’s like that as models become more intelligent, they are less understandable by most of people... but aren’t we humans doing the same?
The weird thing is, that's not what AI models seem to be doing. The prose is just weird.
This happens most though when the speaker doesn't (or care to) understand their audience.
Eg i find effective communication requires expertise in both the subject matter domain but also the reference of the listener. Eg in ELI5 framing, if you don't know what information 5yr olds are expected to know you'll do a poor job at an ELI5.
It often feels like Claude does poorly at both framing the response relative to what it "thinks" the listener knows, but also the prose is... sideways, just weird as you said.
If I don't grok an elaborate explanation, I can ask for clarification. If it's explained to me in an overly simplistic or unnuanced way, I'll walk away with a false sense of understanding.
That said, I'm sure we all have very different concentrations of these types of people and problems around us. I've definitely met some engineers who seem to actively try to make their language incomprehensible
It is unsurprising that a LLM fails, without coaching, to effectively communicate.
Agreed. Do you think it's due to that EU issue of making AI text be identifiable?
When claude speak in convoluted mess, they are often going off on tangents in real work that you asked it to do, too.
Just because something is difficult to understand doesn't mean it's fraud, although if someone is trying to dazzle you with clever words and names of institutions you recognize because they are selling you something, there's a good chance they're lying to you in order to get some money from you.
Sure the explanation will oversimplify a lot but then you can expand it recursively if needed, you gotta start somewhere.
You just simplified most of the problems people work on down to cancer complexity. Ironic, isn't it?
That's also simply not the case, most people are building CRUD apps with some frontend code and some accessory stuff like build systems etc., which while complex, can still be expressed in very plain, easy to understand language for anyone who's a bit technical.
Does not excuse the Claude slop.
Solving the problem right in front of you is easy. Stepping back and asking: is that a problem to be solved, is infinitely harder.
I did not use Claude to write my comment, so I don't know where that is coming from.
I should try adding these tips to my system prompt. Is there a shorthand to describe such language use? I am not a native English speaker.
As for the wording of the prompt, you're pretty on point, I created a custom output style targeting mostly the first two you have there. Some people have wording that demands a certain technical standard or uses fancy words to describe what to avoid, but I haven't seen evidence those work better than asking plainly and I suspect the opposite: LLMs mimic the user to a degree so talking to it in terms of technical specifications and fancy words is an invitation to get them back.
Part of intelligence is knowing your audience and communicating efficiently.
Bingo! And on this axis many SOTA models fail miserably. These things are acting on my behalf under my direction. All the supposed intelligence in the world means fuck-all if nobody can understand it.
And like somebody else said… when meat-based humans talk like Claude does, it almost always means they either don’t understand what they are talking about, or are actively trying to conceal something and are a fraud. Not always, but almost always.
I am a huge fan of Gemini Pro for chat... gemini somehow just knows the most obscure stuff. I'll double check something Gemini said and find the source is deep inside a hard to access scientific paper. Google just has the best index of the internet.
It's best for brain storming, rabbit holes, and image recognition.
Let the big models do the heavy lifting for now.
Even if you aren't coding, you really need to double check its answers. Flash 3.8 hallucinated a Keyence camera's max operating temperature for me, last week, and backed it up with "references".
It's still my favorite model for most non-coding stuff, though.
I do wonder why a frontier model does this to be honest. It still does good coding wise, but it seems strange to me. r/Claude is full of "load bearing" jokes in every thread.
How representative that is of real world usage, I don't know.
In their benchmark GPT 5.6 Sol performs suspiciously poorly compared to the former models.
And like that grok4.7 cache reads are more expensive than sol's (at $0.40/mil).
Wonder if we'd benefit from a much more specialized + task-specific benchmarks to paint a clearer picture like this. A benchmark solely for frontend, ruby, hardware, etc.
I don't know if it's the plain english or what, but I really like Grok for legal research (as opposed to code). It's got a noticeable edge in getting to the point compared to Opus 5.
The Claudish is dead. Long live the Claudish.
Fable 5.1 is not there quite there yet.
They need to get that Sonnet 3.5 magic back.
// `HANDLE` is an opaque kernel handle (kernel32 validates and returns 0/FALSE
// on a non-console handle); every out-param is `&mut T` to a `#[repr(C)]` POD,
// ABI-identical to the Win32 `LP*` pointer (thin non-null). The reference type
// encodes the only pointer-validity precondition, so `safe fn` discharges the
// link-time proof. (`bun_windows_sys::kernel32` declares these with `*mut`;
// redeclared locally so the legacy-conhost cursor path below is plain calls.)
// Progress's terminal handle is the canonical `output::File` (vtable-backed
// stderr/File from `OutputSinkVTable`). The duplicate `ProgressTerminalVTable`
// from B-0 round 1 is removed; tty/ansi/winsize route through the new
// `OutputSinkVTable` slots so `bun_core` stays T0 (no `bun_sys` dep).
I don't want to waste money because my calculator is cracking jokes. They don't deserve their paltry 5% marketshare or whatever it is they have currently. I'm not even getting into Musk as a person or the horrid things we've seen Grok spit out on twitter. I just don't trust his companies with my data and I have seen very little evidence that it's ever the best tool for the job. I'm sure those cases exist but I can't imagine it's worth it.
4.7 is definitely slower & more expensive. It feels kind of like they really had it burn tokens to claw up the benchmarks. But it's not super clear to me whether it's above the line or not. A part of that is that it is so slow that i haven't been making fast progress today with benchmarking it.
Overall, it it gets above my intelligence line its a good release...but you can read the tea leaves and tell the Grok team thinks this was a miss.
For example?
4.6 made more mistakes than SOL or Opus overall. Gave up a lot. And in my opinion, the rate of mistakes is kind of more important than how brilliant it is.
I think 4.7 may still be better, but I was hoping for clearly Sol/Opus level and so far it just isn't there for me.
Designs: https://image.non.io/78795662-8bfc-4e14-8d72-3738392aa6b3.we...
Astra's build: https://html.non.io/annui/
Grok's build: https://html.non.io/Annui-grok/
Additional prompt instructions: "Add scrolling clouds behind the statues. Dynamically light the statues based on mouse position. Use diffui to generate the normal maps/depth maps/roughness maps of the objects, and to separate out the assets on to different layers."
Overall I find these models are getting good at following image as a source of instructions, but their refinement of the output varies heavily between the models. Astra's final output feels more polished, has better visual contrast, and the animations between the pages are smoother. Grok also chose to light all of the background elements, which imo overcooks it a bit.
Still though, for the price it's a great starting point.
Grok 4.7: $12.60
GPT Astra: $35.00
Here's reasoning level high: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
For some reason reasoning effort low and medium used similar numbers of tokens, and xhigh used less than high. I think I need to try without OpenRouter in the middle.
UPDATE: I tried again with the xAI API directly: https://tools.simonwillison.net/markdown-svg-renderer?url=ht... - not a great deal of difference between reasoning levels, and this time xhigh and low used the same number of reasoning tokens for some reason.
For comparison here's a fresh run against Grok 4.6: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
Grok 4.7 generations: https://threejseval.com/models/grok-4.7-high
Also go vote on https://threejseval.com so you can help evaluate how Grok and other model performs compared to each other!
It used to be a mess in various interesting ways. Now, almost every big release can draw something perfectly functional.
So the question - without a correct answer - given the prompt "Generate an SVG of a pelican riding a bicycle":
Does the user want the least lines of code to make it functional, or the best looking version?
xAI missed its chance, Ball is on Anthropic's court.
It's phenomenal at computer use and 3D stuff. I've been using it less and less for coding.
Best to stick with a high end model + low effort, do a manual pass on high effort and fix the bugs you know are reachable.
The two models are in completely different price tiers. Astra costs 5 times as much.
It seems like all you can judge about cars would be their maximum speed on an oval.
Based on Artificial Analysis Cost per Task, Astra is about 2-3x cheaper than Fable 5.1 at Medium and Low.
Consequently Astra could be cheaper than Grok 4.7, depending on the task.
Excited to try 4.7. I hope they fixed the "it's not X, it's Y" that showed up in 4.6.
By now AI should know of the DRY concept. But no. Hence the keys have a rounded rectangle for the key shape and another rounded rectangle for a clip path, to prevent text overflow. There are 72 * 2 = 144 identical rectangles, when just one would suffice (in the defs), with this being cloned once for the clip path, and 72 times for the keys.
I would not expect SVGO levels of optimisation (rounding numbers, that sort of thing), however, the human, if writing out the same thing for the 72nd time, might think 'is there a better way', to get the manual out. A graphics program such as Illustrator would not do that, but AI 'should' because AI.
The above is not criticism of your work, just an observation regarding AI SVG capabilities.
In cursor I have switch over to grok for planning a composer for coding.
"Privacy# All these models are hosted in the US. Providers follow a zero-retention policy and do not use your data for model training, with the following exceptions:
Big Pickle: During its free period, collected data may be used to improve the model.
DeepSeek V4 Flash Free: During its free period, collected data may be used to improve the model.
MiMo-V2.5 Free: During its free period, collected data may be used to improve the model.
Laguna S 2.1 Free: During its free period, collected data may be used to improve the model.
Ling-3.0-tiny Free: During its free period, collected data may be used to improve the model.
LongCat-2.0 Free: During its free period, collected data may be used to improve the model.
North Mini Code Free: During its free period, collected data may be retained and used to improve the model. Do not submit personal or confidential data. See the provider’s Terms of Use and Privacy Policy.
Nemotron 3 Ultra Free (NVIDIA free endpoints): Trial use only — do not submit personal or confidential data. Your use is logged for security purposes and to improve NVIDIA products and services. The logged session data for improvement purposes is not linked to your identity or any persistent identifier. For more information about data processing practices, see the Privacy Policy. By interacting with this endpoint, you consent to the collection, recording, and use of such information and the NVIDIA API Trial Terms of Service."
https://openrouter.ai/deepseek/deepseek-v4.1-flash?endpoint=...
"order": ["relace", "coreweave", "novita", "baseten", "together"],
"allow_fallbacks": false
But I can't argue with the lower off-peak pricing when using DeepSeek directly. The downside is they train their models on your input, which might be a deal-breaker for many users (as it is for me).
Well, at least I spent lots of dollars, and I had to use those models the same way I am using local and cheap models, with the same results.
GLM or Kimi are better for my own personal projects. DS? uhm. it just keeps doing dumb crap
Unless they produce the same token output on the face of it, it looks like they're trying to cover for 4.7 not having good model perf?
There for awhile it seemed like we’d have 3 big competitors but then Grok 4.2 or 4.4 was just diabolical while OAI and Claude continued their significant improvements. Grok was/is so bad that I was convinced musk was gonna shut it down and just fund Anthropic compute once they reached their compute agreement.
- grok 4.6 (xhigh): 97M (for 44 score)
- grok 4.7 (xhigh): 240M (for 46 score)
From their headline comparison:
Grok: $2/$6 per million
Fable: $10/$50 per million
What this doesn't say: Grok costs 0.50/M cache read, Fable $0.25/M cache read
Just makes Grok sound deceptive, and more importantly, reliant on user's lack of understanding of costs aka predatory (which in turn is more infuriating)
It is the same multiplier for Sol with subscription. For Astra though the multiplier is ≈20x, so half of Sol usage.
For Claude it seems to be ≈40x too for Opus, but less for Fable (similar to Astra in GPT).
All on the most expensive plan. Previously, Grok usage escalated linearly from the $100 plan to $300 plan. That would be a really good $100 plan if it is still true.
Some sources:
1. https://x.com/kunchenguid/status/2098256018836963382
2. https://x.com/stevenzhang/status/2092110386569089311
3. https://github.com/openai/codex/issues/43731
For me and what I’m doing that’s insanely good value.
I find grok build chews through my SuperGrok sub very quick - but I think that is due to it having the 500k context window which uses more credits. Cursor limits it to 256K (tho I see in today’s update for Grok 4.7 there’s now a toggle for context size).
Astra for deep dive investigations, Sol 5.6 at mid-level for day to day tasks, Grok 4.6 via Cursor for routine and low complexity tasks.
I'm excited for 4.7 although I share skepticism with other users whether 4.7 will be significantly better, since they didn't raise the price.
Output tokens from Intelligence Index:
- grok 4.6 (xhigh): 97M (for 44 score)
- grok 4.7 (xhigh): 240M (for 46 score)
Is there a metric for like... time taken when comparing these two? I see score and cost.
If Fable5.1 can knock it out more quickly on low but Grok4.7 might take twice as long to stumble through a problem (and leave behind a bunch of yucky comments or un-needed extra unit tests), are they really comparable?
Or like... the "quality" of the solution? "It works" versus "it's unmaintainable/very messy/hacky".
It will loop in thinking mode ("Let me implement those fixes: Fix 1, Fix 2, Fix 3 .... Fix 80, Fix 81"), ignore the AGENTS.md instructions, corrupt plan files, etc etc... I have 5.6 Sol as advisor/watchdog, and it blocks every turn, I never saw this. Quite a shame, 4.6 wasn't so bad.
- it allows different models within one session via roles (I only have API, so pay per token)
- it's much more likely (ime) to use the LSP over grep for determining how code fits together
But I agree a 20k+ starting context is way overkill.
I find it's very hard to get information on harnesses people are using. I have to stay model agnostic so I avoid claude, codex, cursor, etc. I've used and tried opencode, which worked well, but obviously lacks the above features.
Does anyone have a resource for following what people are actually being productive with? With so much vibe going on it's hard to separate the wheat from the chaff.
This explains why. Mentioned in another comment, but cursorbench explicitly tests with Cursor as the harness, and OpenAI doesn't allow them to use Astra in Cursor.
That said, I don't expect them to benchmark Astra in their Cursor harness given the situation.
If Cursor wanted to include Astra in CursorBench nothing would stop them, they could easily have spent half an hour vibecoding in OpenAI API key support - if it hadn't been convenient to neglect to do that.
Its because of this. You can't use Astra in Cursor, and cursorbench uses cursor as the harness. They can't actually benchmark it using their harness hence why its not included.
https://x.com/elonmusk/status/2102082011233931762?s=20
so it's likely about usage in Cursor specifically.
That it isn't the most efficient way to achieve the same end result is irrelevant.
These comments don't stay up much anymore and I can't tell if it's structural to the forum (flag weight + statistical mechanics of votes + guidelines) or if it's the userbase sentiment.
But I think it represents real malaise in the community. It's not a moderator plot, people here really just don't care and might even support this.
We really are in the minority of opinion for giving a damn about liberal democracy.
Between the guidelines + user thoughts (e.g. repetition, low novelty/new info), there's other reasons these types of replies might end up dead.
I am worried that it leads to people self selecting to other forums biasing the remaining userbase vote/vouch/flag distributions. In an exit vs voice situation, the voice kinda dies out. Then we end up other-izing people and homogenizing our communities.
But I concede it's also possible that the minority opinion issue could be the core driving force.
Nobody both worked and spent their money to get Trump elected like Musk. 300 million to his 2024 campaign [1]. DOGE. On-stage endorsements. Nobody even came close.
No, other big labs are not "innocent little virgins", but they're not even in the same solar system of harm as Musk. To hand-wave at the differences is to permit them.
[1] https://www.opensecrets.org/2024-presidential-race/donald-tr...
Handing corporate code secrets to his AI model is... unusually trusting.
And methane is a large percentage of all power production in the US. So again that also applies to all the other data centers. (And FWIW they've been winding down and shutting down the on site methane generators.)
And no corporate code was handed to AI models.
Even with his successes (Tesla, SpaceX) he has built them up in large part by bending levers of government to his advantage.
Can you provide specific examples of where Elon has bent the levers of government?
So what? Thats called being a maverick. He is very very good at executing on making money which is the point of business.
Also pushing technology forward.
Anthropic: 1.25B/month
Google: 0.92B/month
Unnamed customer starting in december: 1.1B/month
Starlink monthly revenue is ~1.5B/month
If anything, they voted for reduced debt burden and they got the opposite. DOGE failed at pretty much every single one of the goals that the public arguably gave it a mandate for.
Ah, yes, democracy!, except for when the public is wrong.
Who decides when the public is wrong? We do! Who decides "what the public voted for"? We do! So we are the rulers? No, of course, not, this is democracy.
You want to become the decider of when the public is wrong and of what the public voted for? TYRANT! TYRANT!
This is simply epistemologically incorrect. It's obviously incorrect in this case because voters writ large do not have any idea how the government is administered and how to improve it, so even if they claimed to be voting for that, it would not necessarily be an endorsement of any particular approach.
More specifically we know it's not true in this case because there are polls. Voters didn't even claim to care about this! "How the government is administered" was not a high salience issue to voters. Simple as that.
Nonetheless, I didn't suggest anything about overriding their votes. It sounds like you have some sensitive spots to work through (someone obliquely criticizing your idol for sucking at his job?)
half of voters don't pay any attention to politics until the week or two before voting
Sheep often like to think themselves the wolf or coyote, it would seem.
Fuck, it is like the denial around Jan 6th. Those idiots we’re live streaming that shit. I watched it go down live. Now they say they weren’t violent.
We can’t have discourse when we have legit video evidence and people refuse to open their eyes and choose to deny reality
Which Nazi ideologies do you think he embraces? How do you reconcile all the Nazi ideologies he rejects?
The personality is bland and it doesn’t work nearly as hard or even tries to help.
I don't use Grok, but do you want your LLM to have a personality? "Personality" is exactly what people don't like about Claude.
There are ample reasons to believe that Elon Musk is running his mother's account and that the photos weren't even real putting in question that he even had a birthday party.
If you ask Grok about what this means it will always take Elon's defence. It will vehemently deny that Elon would be capable or willing participant of such a thing even if you point out that he faked being a world class gamer, buying accounts that had done all the work and showing none of the skills when live-streaming.
Sounds like a plus. Guess I will give Grok another try...
Until you ask it to start generating horrific imagery and then it's best in class.
The value of the internet is that people can share whatever they want, and use software how they want. This will mean that some people will abuse that. This is the tradeoff of a free society.
But also Xai doesn’t seem to care about user experience and long term support.
For daily one off questions I prefer it because it is fast enough and I like the way it responds. I also use it for basic research like “find me a battery drill for this and that”.
Kimi and GLM feel extremely coding oriented. I use them for code reviews basically. I hate the way Anthropic models talk. GPT takes too much time and effort for that kind of stuff for some reason.
Grok happened to be a nice middle ground.
As a technical point of reference to compare against other llm stuff, sure, I'll glance at a report or benchmark but I really couldn't care less about anything to do with the project and it could blow other options away and I wouldn't touch it.
You probably shouldn't cut off your nose to spite your face.
What's superficial about refusing to use a product from someone like that? Or are you one of those 'technology isn't about politics' people? That's a superficial take if you ask me.
All technology is political, and understanding that is a deep, not superficial take. It requires systems thinking which unfortunately many people building technology seem to lack, despite software being a sophisticated complex system.
I like to follow them and look for benchmark for each LLM release.
Maybe I'm in some kind of bouble but I have never met or talked to anyone who has used Grok.
Not sure if I'll hold the subscription but I could see myself working with it more.
As a chatbot it’s totally fine, virtually indistinguishable from Gemini or ChatGPT or Claude.
For coding it’s… okay. I tried 4.6 and it feels similar to Opus from 12 months ago, or maybe Sonnet from 9 months ago. YMMV.
It's just an observation but so far a pretty solid correlation. Musk has so severely poisoned the well in terms of his UK reputation that the only people who are open about using Grok are... well, wankers is as good a word as any.
FWIW among the AI-using people, it mostly goes Claude Code, then Codex, then whatever runs on their Mac. The only Cursor user I knew has jumped ship to OpenCode.
I can't think of a single dimension grok is winning on (capability, cost, voice), but want to stay open-minded -- anybody want to vouch for its capabilities in any domain?
I think it's winning on UI for normies (grok bot) and they made some claims about being pareto SOTA (lowest cost per task completed) a while back with 4.6.
I find it to be a perfectly capable model for implementation (there are many in this class--deepseek flash, spark1.3, luna, etc). I find the usage to be very generous w/ supergrok. I find the model to be just fine for 90% of what I want to do, but I use a smarter model to plan complicated things.
The voice is the same AI slop as the others imho.
(This is about Grok 4.6, I didn't test 4.7 yet).
edit: clarified I mean agentic coding tasks