I got it via the "Pro" plan, exclusively for personal task usage. So no agentic programming.
For starters, the limits are dreadfully low. Five messages on medium is already enough to trigger a 4 hour cooldown. Secondly, the best part is the integration with Google search and Maps. Makes it seamless for trip planning. Lastly it's the only "SOTA" model in a long time that has actively hallucinated details. I don't know if it's because the Gemini harness is not asking it to verify information, but I've had some real Deepseek V3-like lies in responses.
Either way, I find it difficult to get invested in Gemini because nobody will have access to Carbon for a few months anyways.
Honestly the fiction writing was pretty impressive!
But my experience is that the agent matters a bit more than the model. It is fun on ChatGPT because of its recent (?) bias to always be creating images for conversations around fiction.
Capabilities are also improving so fast that this fomo doesn't really keep people excited. If I can't run it, I'll sorta just click off. And given how they handled the Argon release, I've not even bothered look at this Carbon one. Fully expected it to not be usable yet.
they don't really care how end-users will receive these models, they're just announcing for internal targets.
You are assuming that Carbon is ready for release, deployed to their datacenters, that all scaling and other issues have been worked out, and they are confident in all of the above. Not to mention additional internal constraints like alignment testing or other safety procedures their own policies and guardrails require.
If they are smart, which they clearly have some smart people if they are making the best frontier model, they would account for the above needs in their timeline, and announce the model when they can prove they have it but also before they are ready to release it. This is so they can be at the top of the frontier model lists, boost their brand for both Gemini and Google, get the bump in stock price and investor confidence, steer adoption decisions for people making decisions for what platform they might adopt, etc.
The target is to make a celebratory announcement.
In my limited usage, it has fixed issues which has not been correctly fixed by sol-6.1 (medium) - https://github.com/auto-gnufied/markdown-modern/commit/a189a...
I am also using it via Google pro tier and did not immediately run into quota issues. But as someone else said in previous discussion of this - Google Pro tier already includes lot of stuff (including Youtube premium etc), the AI plans are just a nice addon.
From Google saying that Argon would be 'available soon' (as they did just now for Carbon), it took them 13 days to release it. So... unclear that Carbon won't see the same trajectory.
That's the problem for me. Google can have the world's best model; it doesn't matter to me, because I effectively can't use it.
Both Opus 5.5 and Sol 6.1 are extremely good and provide a lot of inference on their $200 plans, and then there are the Chinese models - DeepSeek 4.1 Flash, MiMo, and StepFun all offer a lot of value per cost.
At the moment, I don't see a niche where it makes sense to use Google's models, regardless of how good they are.
for coding .. it works ok but i prefer the openai or anthropic models, much better results.
I feel like you're being overly dramatic here. Argon was 13 - 15 days.
Argon was announced September 30, made available on October 13 to Ultra everywhere, and Pro in the Gemini App. Pro plans got Argon on October 15 in AGY.
This is the first time they've slow-rolled a model. Who knows how long they'll wait on Carbon.
The Ultra-lite plans ($99/month) get some pretty good quota where you can actually use Argon some more. Maybe Google shouldn't be grouping together plans, but it should also be noted that $20/month also gets you 5TB of storage on Drive. If you paid for Dropbox, you get 3TB for $18/month (without any AI model usage).
(googler, opinions are my own)
By the time Carbon comes out, I fully expect Anthropic, ChatGPT and/or a Chinese lab to have its answer ready to ship in less time than it took Google to approve a release.
Gemini cannot create "my maps". I like creating a map of stuff for travel and now I ask it to create an importable file (which it doesn't always get right) but it should honestly do that itself.
And I don't mean that from within the Gemini app, I'd like a shareable list of places, which you can do via a very outdated maps UI
my whole family of 5 people is using in total 160 GB. personally I use about 15 GB
It looks like the best sort of bang for your buck is still GPT Sol and GPT Astra. Claude and Gemini have just been going for that number one intelligence spot by sacrificing a lot of efficiency.
It does sort of show their hands a bit that their larger goals might be outside of just consumer sales but also seeking hyper intelligence that could bring value in new ways of discovery (if it ends up being a legit outlet for such things consistently)
Gemini models always suffered from this. They were almost always best in terms of intelligence, but one of the worst to use in practice, because they have a lot of failure modes on simple tasks that simply give factually incorrect answers, or completely break and output gibberish.
I don't know what it is about Google models, but they are the worst at hallucinations. I don't understand how they can so consistently be this bad.
Yesterday I used Google AI-Search (so Gemini Flash 3.5?) for opinions on a traffic accident and it continously constructed law-interpretations and then referenced the wrong paragraphs. It talked about rules on road-crossings and cited paragraphs pertaining to alcohol intoxication as sources.
the problem with gemini is the whole app itself, not the model.
when you use other chat app, like chatgpt or deepseek chat, the app will use web search for everything to verify. and if it can not find anything useful, or the current fact is disputed, it will try to call web search tool again and again until the consensus.
meanwhile gemini will refuse to do web search if possible, and only do it a single time if demanded. worse, it uses the same result from google search, so the top ranking answers are almost all SEO garbage.
since then I just stopped asking gemini again. and I don't think the newer model will change the situation.
Since then I tried it for experimenting on diverse fields. For example, this week there was a discussion here in HN about the non-continuation of Python funding on the JIT, so I "played" again to removing the GIL and moving to the JIT in Python, it did it. I played with many other things, even the Riemann Hypothesis (it just removed their rational parameter rounding only moves 0.875 towards 0.87495702, nothing significant though).
Many other relatively great things were done this week in parallel. I have some budget for research.
Trip planning is the most often used use-case for agents/bots I see in marketing. I'm lucky to travel once a year, but if I did have a trip, I wouldn't feel comfortable letting an LLM take the wheel. Too many variables for wrong flight or room, closed locations in itinerary, etc.
Now, critical server infrastructure? Sure, have at it Claude, make no mistakes!
Plus, it's great at researching the history of places you're visiting, so you can find cultural experiences a bit easier.
I don't travel often, but I do live in Europe where it's easy to visit a new city or country in the weekend. This makes this functionality a quite common use for me.
Could you expand on this? I’ve never seen a frontier model that doesn’t hallucinate details fairly often, what do you mean by “actively” here?
I used to defend Google's models months ago, but switched to Luna for less intelligent models.
But with this bs of not making it even public, pushing off release of newer models late, I lost faith in Google.
Also, their tooling still is complete shit. I'll not be forced to use Antigravity thing even if it is available and clearly better than Opus or Astra. Their subscription is useless. API can be used with anything you got but that's gonna cost.
I never use any google models for any agent work any more, in case they block me.
Started migrating gdrive, and gmail to other providers at this point after horror stories in HN, that they lost access to their gmails, etc.
Prove me wrong.
It's been provided to AA under clandestine circumstances, no you can't verify it yourself, this might not even be the model you get.
People on HN for some reason "wow this thing that I can't use is so great, really excited about all these ads I just looked at"
Having a great time with an ice cream brand at a store is no assurance against hollowing out the brand tomorrow.
Especially with the recent addition of graphic visualization widgets
It manages to strike the perfect balance between depth and style of prose that other Frontier models just can't quite seem to match
1. Sleep debts effect on muscle size and strength gain
https://share.gemini.google/Gvdb5Q02INrl
2. Fish oil and extra cellular vesicles
https://share.gemini.google/fem9EBCe4mfy
3. Pharmacology question about human-equivalent dose
I'm trying to figure out how heat dissipation works. Specifically, I have a block of aluminum with extruded fins (as a heat sink) and I'm trying to understand how many 30-watt LEDs I can place on it. I have a fan as well, and I'd like to understand how fast I need to run it at different power levels (the LEDs are dimmable). I can gather measurements empirically (which I'll do), but I also want to understand the basic mathematical basis I should use for estimating these values.Its the first model ever that I have been worried about a successor losing the "feel" that it has.
https://artificialanalysis.ai/models/qwen3-8-flash-next
https://artificialanalysis.ai/models/glm-5-3-flash/
edit to add: It's very disturbing, because these open models (GLM particularly) were among the best of all LLM's on their benchmarks. On the intelligence/cost frontier, specifically. Archive.org remembers that GLM Flash was in their green quadrant just a few days ago, next to Haiku:
https://web.archive.org/web/20261014031624/https://artificia...
It's completely gone now. AAI (as far as I've seen) haven't publicly commented about why they did this.
I also think Google should stop "releasing" models that aren't actually available. For their own sake, because by the time Gemini 4 Carbon becomes available, nobody will notice. There won't be a new press release or much excitement around it.
My experience is that Google is aiming to create a generalist model – but a truly generalist one! In fact, I find it particularly accurate in general terms and even in long conversations, maintaining excellent consistency and a wealth of information in response to complex questions.
I’ve tested it both as a coding agent – though, precisely because of what I mentioned above about its ‘generalism’, I don’t think it measures up to Fable 5.1 or Astra – and I’ve really appreciated its qualities when supervising my agents’ activities. It was able to keep track of them, suggest alternatives, and monitor the entire lengthy process in parallel without ever straying too far from the request or losing the thread – as often happens with Claude and OpenAI (you only need to test it in Claude Code itself, in both Ultracode and the agent team mode – albeit experimental – to realise how, once the task gets underway, both of them lose their way).
Without us being able to access Carbon when it is announced, there's no way to know the Carbon we get a few weeks later is the same thing as what has been benchmarked here. I think benchmarks should verify what is benchmarked is exactly the same as what's been released.
[1] https://news.ycombinator.com/item?id=49901736Since then none of the Google models have competed well with the others at their price points. Long context performance drop, instruction following issues, unnecessarily complex and sloppish outputs.
Argon is no different, it feels just an iteration better than 3.8 flash. I don't trust the AA ranking yet.
Also the guardrails were implemented in a very bad manner. By default the models used to raise a default guardrail error on basic queries.
Until it got lobotomised. Then it lost most of it's capability.
Similar situation happened to Anthropic's Fable. It was an amazing model until Opus 5.5 came out.
So now, I'm not even looking at a "new" model until it is a month or two old. By then we will know if they will destroy ability to make it cheaper to host.
* Definitively very good at domain specific knowledge. It can read sloppy 19th century danish gothic handwriting with faded ink just fine. Not perfectly, but probably better than I could have done with a day of struggle for every page.
* Expensive.
* Bad at actually doing what I said. I had told it in the prompt to be careful not to miss text, but I hadn't explicitly told it that some images may have text in double-column layout. So on those images - over half of my 300-page document - it produced an excellent transcription of the first column, and then considered itself done. Want to try again with a more careful prompt? That'll be another $20.
I also found that it ignores other instructions sporadically ("Don't use letter variants or ligatures in the output, use the normal Danish/Norwegian alphabet when at all possible", still a few Unicode tall s'es in the output.)
It's good, but it's so frustrating since "worse" models manage to follow such instructions much better.
That seems arguably pretty cheap compared to the human alternatives.
I do get super annoyed at how specific you've got to be with these tools. They are like almost like malicious compliance geniuses lol.
They're like a monkey's paw or evil genie I think.
You'll get exactly what you ask for, but lots of weirdness you didn't expect.
To be fair to the model makers, deciding what to do with an under-specified prompt is hard, particularly in an RL environment where you can't just ask the user.
For stuff I want to run for a while, I'm defaulting to telling it to ask me the questions upfront to reduce the incidence of these kinds of problems.
- Two days ago AA announced their v5 Index which added a few new benchmarks and tweaked a few things.
- Today the newest model announced from a big lab tops the charts.
I'm not saying that AA is definitely taking money from the big labs and changing the benchmark to make the labs' new models look better, but it kind of looks that way. This pattern of behavior has happened before, where the index gets tweaked right around when a new model comes out, and the new model is on top. AA does not, AFAIK, provide a way to see how the new models would have scored under the older index versions. You could probably calculate it yourself, but I'm not going to do that much legwork just for an HN comment.
It's interesting because they publish the index methodology, so you can't really accuse them of not being transparent, but I don't really believe that the choices to update the index were completely independent of the model they happened to be evaluating at the time.
Related: The DXOMark camera scoring method also had similar behavior, where they would release new testing methodology right before flagship smartphones were released. Again, they are technically being transparent about the process, but it kind of smells like the "independent" testers may not be as independent as the average user would hope.
Check out vals.ai for an alternative. Not sure how independent they really are either, but it's at least good for a cross reference.
Example problems are:
- https://hub.harborframework.com/tasks/terminal-bench-science...
- https://hub.harborframework.com/tasks/terminal-bench-science...
I have no doubt that these are very relevant task for people in those respective fields, but I suspect they have next to no relevance to some 99.9% of model users. Hence why especially smaller labs with smaller models didn't really train for such use cases and you can see it.
Personally, it's not something that matters at all to me, as I don't work in cheminformatics or bioinformatics.
This is also arguably more of a test of a model's knowledge base than reasoning ability.
But most importantly, there is also the question to which extent the aggregate score then remains a meaningful result when you mix up too many too disparate benchmarks.
The coding agents subindex, in contrast, looks at least superficially plausible [1].
[1] https://artificialanalysis.ai/agents/coding-agents#coding-ag...
At the previous iterations they were accused of favoring OpenAI. Now they are accused of favoring Google.
I do think the test a new model and realize their index is broken- the new model isn’t on top, so they change the weights to make sure it’s on top.
The difference is that you seem to think they are doing so out of some innocent feeling that the new model is actually better, while I think they are doing so because they were paid to do so. I think my interpretation is more likely, but that’s just me.
Or are you implying that the Chinese can't afford the amounts paid by OpenAI/Google.
So ... why bother to cheat?
I think part of it is to justify capex. "Yeah, we spent x billions on this, but look, we have the best models, we're beating OpenAI and Anthropic, we're winning, it's all worth it!"
My hunch is that Google will end up being best for non coding tasks. Google can concede coding to other vendors who pay Google for infra.
True, I use LLMs as glorified search engines to unearth material to read myself, but Gemini is much more useful if you decide to learn yourself. It also offers interesting expansions to your questions.
While I don't offload my work to LLMs (e.g.: Write this project, I'll be watching TV), I find them useful for closing my knowledge gap by collating information, and Google's Gemini is good at that.
I'm on $20 Claude plan, and it's also ample for my needs, for now.
They have top of the line research, I mean, they invented the thing, and they are still doing stuff that goes beyond making a bigger LLM, as it seems to be the only thing the others are doing, research-wise.
They also have hardware capabilities no one else has, they invented the TPUs, and have lots of them.
They also have the brand recognition, ecosystem, user base and lots of money.
So why are they lagging behind on the actual products, and by lagging behind? I mean they are always in the position where they are just behind the frontier, but still in the game.
It seems almost deliberate. Maybe they are playing the long game, waiting for the bubble to burst. Maybe they don't want to compete against the more profitable parts of the company. Or maybe their researchers are not interested in making an actual product unless being forced to.
Even if these numbers are correct, they don’t have any useful product for coding or search. Not sure what you’re referring to
I wouldn't even bet that Google is better than OpenAI and Anthropic for these categories, but they seem to be well used outside of our direct space.
I think what's happening is Chines models genuinely try to benchmax on it and hence they suffer when a new version comes out.
I read somewhere on Substack that a lot of this was driven by the inclusion of more knowledge work benchmarks, where the US labs are stronger, versus coding which is where most Chinese models appear to put their energies for now.
Here's mine for coding in Rust, and I definitely see the same pattern of "most chinese models do much worse than on public evals that have been around long enough to train on" that AA v5 surfaces: https://slopcop.com/power-ranking
Let’s see, maybe they’ll surprise this time.
I mean, that's the main reason. How would you expect to see the kind of hype that Claude or GPT can generate when nobody can use it? The hype of those models are primarily generated from people using them to build impressive things and share those results. Can't do that with an unreleased model, right?
Beyond that, it seems a lot of people have learned that these agent leaderboards really don't tell the whole story. They're a good litmus test, per se, and there's plenty of hype being generated around these scores as-is. But anybody using AI in practice knows that benchmarks can only say so much, and one model scoring better than another doesn't necessarily imply that that model is actually better in the ways those people care about. You can see that in this thread as people are pointing out that even these benchmarks seem to suggest that Opus 5.5 is still leading in coding (which is what a lot of people, especially on HN, ultimately care about).
>just not enough publicity or hype as the other labs do?
Well, it's effectively both. Google is certainly not that pressed to generate as much hype as these other companies. They can afford to take their time. It's also worth noting that Google is definitely targeting different consumers, investors, etc., so their strategy will definitely be different.
For what it's worth, I've been using Argon since it's been available in Cursor and I really like it (at least at its current price). I think Google really did close the gap a meaningful amount (maybe not all the way, but enough to prove they're still in the game). Carbon being significantly better while being the same cost is pretty exciting to me. But until I can use it, I don't really care about it.
Want a private benchmark and see where you'd rank? $$$.
Want to show up day 0? $$$$.
Want to show up day -0? $$$$$.
I take this opportunity to give feedback on Argon. The limitations are a problem. I have a Pro subscription and Argon already stops working after 3-4 prompts with long delays afterwards. It's virtually useless right now even when you pay monthly for it. It is better for some tasks than 3.1 Pro, though. Not much in my experience but it can sometimes fix errors that 3.1 has problems with. I never use Flash models since they often give abysmal results.
So I'm stuck with using Gemini Pro 3.1 for now.
Is this API-only or US-only for now?
"We're taking a deliberate approach to rollout and are actively engaged in the U.S. government’s voluntary process for pre-release model testing. Starting this week, we will make Carbon available to a subset of cyber defenders and will gradually phase access to additional trusted partners through our Fairwind Program. We’re continuing to gather feedback and iterate on guardrails before making Carbon available to developers, businesses, and consumers via our product surfaces as soon as possible."
I feel like the headline here is "the Artificial Analysis index is becoming an increasingly poor measure of capability, having previously been quite good".
So here's a meta-benchmark: at what point does the act of choosing an LLM product for a reasoning task become less reliable (in the sense of "predictive of future success") than hiring a human being for that same task? Have we already cross that threshold? I think we might have.
And once we do... what happens then? Will the truly elite models be hidden in a sea of mediocrities simply because the goofball "managers" doing the hiring are too dumb to recognize their genius?
No, you can't use it. You can't even see it, and I won't provide you any information. But you must trust me that I am the genius who made the best model in the whole world.
Google's model release is pure PR puffery. It's no coincidence that it was released on the day of its earnings call.