The author observes that a call to GPT-5.6 Luna is only 4-5 orders of magnitude more expensive than grep, and then predicts that at current rates of progress, calling an LLM will soon be cheaper than a grep. I think this is a good time to invoke Stein's Law: "If something cannot go on forever, it will stop." These efficiency improvements won't continue forever. It's more likely that the per-call cost of high-quality, compiled software like grep will be a lower-bound that LLMs asymptotically approach, rather than a line that they blow past with perpetual exponential progress. (Barring a true breakthrough in something like quantum computing or room-temperature superconductors.)
non-backreferencing grep is not very difficult to implement in an ASIC either. But it's probably not worth it because of how relatively rarely you use it and of the data transfer costs.
LLMs are great candidates for ASIC-burning because they're slow compared even to network speeds and run all the time. The issue is that you don't want to burn a specific model or architecture that then becomes obsolete.
So you've got two possible futures, and both guarantee large price drops: (a) LLMs keep getting better and better and better, so ability/$ keeps rising; or (b) LLMs plateau in ability, in which they will start getting ASIC'd.
Inference costs cannot keep falling forever, but they do still have a long way to go.
They don't need to plateu for that to happen. There are companies already building AI on ASIC, and IIRC they were approach 12 months lead time. A 12 months old frontier model (Sonnet 4.5, GPT-5, Kimi K2) for 1% of the price is still a rather good value proposition.
But I would probably take a long horizon bet that the grep implementation on my machine will remain cheaper than an equivalent ai task, even though I think those ai tasks will become far cheaper over time.
I just think the original comment's model of asymptotic approach is probably more likely to be accurate than the model of the line blowing through this grep-like cost level.
Yes, because it was a largely random undirected process.
Then you have to climb to another branch to get more fruit. The biggest issue with most problem space discovery is you're doing it blindfolded.
you could make hardware implementation of grep and store the file itself next to it in some ROM but that's not a very useful grep ... while hardware LLM is exactly as useful as software LLM only orders of magnitude faster
https://iepathos.github.io/ripgrep/performance/#work-stealin...
That’ll be just as true for an ASIC NN as it is for good ole grep.
NVMe v1 wasn't released until 2011 and there was no hardware available until 2012. Before that the fastest drive interface was 6 Gbps SAS. Then we got 4 GBps (32 Gbps) with NVMe v3, 8 Gbps (64 GBps) with v4, and 16 Gbps (128 Gbps) with v5. V6 is slated to double again. v7 is currently planned to double that. This is less than 15 years of progress.
In the 1990s, we went from 33 Mbps to 66, to 100, then 133 with IDE. We went from the 40 Mbps of 1986's SCSI 1 to 80 of SCSI 2 in 1994. Since then parallel SCSI has gone to 80, 160, 320, 640 (Ultra 2 wide), 1280 (Ultra 3), 2560 (Ultra-320), and 5120 (Ultra-640) Mbps.
SAS has also gotten faster, with 6 Gbps, 12 Gbps, and 22.5 Gbps. NVMe is still lower overhead and lower latency.
Memory bandwidth can also be addressed. Stock servers with EPYC are 16 channel DDR5. That's 409.6 GBps. Four channels of DDR3 was only 102.4 GBps. Video and accelerator card memory bandwidth is also increasing.
At some future point where LLM hardware is cheaper than simply running grep, then grep equivalent would benefit from those selfsame hardware improvements and be cheaper to run as well, probably still by the same ratio.
Is running LLMs (or some other ML workload) on/with quantum computers expected to bring efficiency gains?
Won't that also help grep and then move the asymptote down more?
From a computational standpoint this is obviously nonsense, but from an attentional one I'm not so sure. It may already be more attentionally expensive to use grep in some cases, such the moment you need to remember a non standard arg. And if this applies for performing a simple http operations, then it certainly applies going up the complexity chain.
The only aspect that is poorly analyzed by the OP is business model viability. All players are investing insane amounts of money in infrastructure with the expectation that their future profits will justify all that investment. The winner or winners in the AGI race, they believe, will find the proverbial "pot of gold at the end of the rainbow."
The OP glosses over questions of business model viability with a brief qualitative discussion and very little hard data. For example, to earn an annual return > 10% on every trillion dollars of capital sunk into infrastructure, the owners of that infrastructure must earn free cash flow (operating profit less investment) in excess of $100 billion per year in perpetuity. Is that feasible? Why? How?
The OP does not really consider such questions.
You may be right. I'm not so sure. Inference looks like a viable business model for those operators that have SOTA infrastructure in place, but the investment required to have it is enormous, and appears to be never-ending, because if an operator stops investing aggressively, its infrastructure quickly becomes non-competitive, and customers will quickly leave for alternatives. SOTA infrastructure is a moving target.
But yeah, I definitely don't have high confidence in any of this!
Free cash flow (operating profit less investment), actual cash coming in, is deeply in the red.
EBITDA can be a sensible measure of profitability when there isn't much need for additional investment. That doesn't seem to be the case with these operators. They need to invest aggressively to avoid losing customers to competitors. All of these operators have made multi-year commitments to invest more in infrastructure. In addition, they have guaranteed quite a bit of debt to fund it.
Maybe it all will work out fine (and I sure hope it does!), but I didn't see any hard data from the OP, or from you, supporting that view.
Those real debt obligations are going to want to be paid back.
"It is not too much to expect that our children will enjoy in their homes electrical energy too cheap to meter,..." Lewis Strauss
https://en.wikipedia.org/wiki/Too_cheap_to_meter#Origins
Oddly enough my power bill was metered and big.
Of course, perilous territory for future irony depending on how your prediction plays out.
The cost of a given level of performance often falls fastest right after that level is first achieved, that is, when it is state of the art (SOTA). We see this pattern on three of our five main benchmarks of AI capability. Averaging across all five, cost falls 66% per quarter (75× per year) for performance that has just debuted as SOTA. Two years later, prices fall half as fast, at 32% per quarter (4.7× per year).
but the analysis itself has more nuance and is a quite interesting read.
[1] https://epoch.ai/publications/the-plunging-price-of-thought
I think Nvidia is under the same pressure as Anthropic/OpenAI. Nvidia will dominate research and probably keep dominating training, but the real volume is in inference. And for inference Nvidia's lead is only a few months, similar to the lead frontier labs have over open source. Nvidia will sell a lot of Rubin CPX's, but their margin on that will be a lot smaller than B200 because there is so much more competition in that space.
I don't disagree with the thesis here, I just don't think costs are coming down quite that quickly.
- Energy is not infinite, neither energy efficiency is. - Datacentres neither. - Benchmarks are an abstraction of real world problems!
On top, there is an overall "economic" aspect that most of the people miss: every change carries a certain degree of risk (lose money, reputation, customers, death of people, ecc) that very few want to take and a lot of changes(e.g. rewrite some piece of SW in another Lang) don't produce a positive economic impact.
Of course, for collecting better telemetry using local AI for analyzing video from camera and audio from a microphone.
I just stopped reading at that, for anyone else, Please find a better source and take everything in here with a grain of salt.
IMO
The number one improvement that mattered for local AI was llama.cpp, partial offloading to system cpu/ram. The next was quants, being able to take fp16 and turn it to q8, q4 etc. The next IMHO is unsloth dynamic quant, that have been able to do mixed precision so we have UDq1/q2 that is actually pretty damn coherent. Allowing individuals to drive K3 locally even if it's at Q1/Q2. Then MoE changed everything for everyone, cloud and local. The other is integrated GPU, Apple, Strix Halo, DGX Spark. Then all the extra improvements like MTP, DSpark, etc. Of course there's many other additional things that have mattered too
> Generally, models intended to be run locally will be much smaller, such as Muse Glimmer or Qwen3 Coder.
That is such an interesting set of models to use as examples here. One being essentially obsolete on release a month ago, and the other being completely ancient in LLM time. I really wonder how they landed on those two.
Then we have these articles that say tokens will get so cheap that labs won’t know how to make profit.
Who is correct?
Token pricing was a small minority of customers up until this year, when all the labs started trying to force customers onto token-based billing. Within the last week, Anthropic repriced my team's plan from a temporary "50% extra tokens" to 25%: https://support.claude.com/en/articles/15910845-claude-code-...
The fact that all this is ongoing within such a short timeframe should make you suspicious of any analysis that claims to be observing "statistical trends" like they've discovered a new Moore's Law out of 6 months of pricing data from 2 companies.
AI is not a bubble, IMO, though we may see a retrench and some companies with sky-high valuations will crash to more reasonable ones. But data center demand is probably a bubble, and the main driver will be reduction in the actual amount of power and data center space required to serve escalating demand.
I think hardware and model improvements will pace or maybe outrun demand and then when demand starts to saturate will keep going and leave a lot of orphaned data centers.
When people say "AI is a bubble", they mean economically as a whole, which includes data centers.
Perhaps we need better terminology for "product useful; numbers nonsensical"
You can't simply wave Jevon's paradox at things. Thousands of miles of canals were dug in the UK that couldn't be sustained and were abandoned. Thousands of miles of railways were laid that could be sustained and were abandoned. And those are potentially durable investments, unlike cheap walls, pillars and roofs laid over a levelled concrete slab full of fast depreciating IT equipment.
But yes, if sold for a negative margin Jevon eventually stops because the decreasing supply will drive up prices.
> things are made that will sell for less than the cost of construction
Price is set at the marginal cost. Capital costs aren't in marginal costs.
You'll need a better counter-example than UK railways which suffered from Parliament price-fixing.
I'm so glad the tide here is turning on this talking point, brought on by exactly the same people beating us over the head with it for months while no progress is made towards it materializing.
Many, many people who post here are capable neither of real analysis nor distinguishing real analysis from memes. They aren't hackers, they are adherents of a cult that happens to focus on the same subject matter as hackers.
Yeah okay bud, anyone checked in with the state of consumer hardware recently? Not the author, evidently.
>oh in 3-6 years this will all be over
Yeah I'm sure Samsung, Nvidia and sk hynix will all be very calm with lower volumes and lower margins.
RAM prices will crash when demand drops even a little. They'll probably crash to a lower (inflation adjusted) level than before. This has happened before.
Industrial scaling in general often looks like a sawtooth: price spike, capacity investment, crash, repeat.
Part of what's keeping prices high a little longer is that everyone knows this and is a little reluctant to plow resources into chip fabs for fear of having the bottom fall out before they recoup or sell that to someone else to hold that bag.
Graph the average compute and RAM in a mid-high end laptop at an inflation adjusted price point for the past 40 years. It's very exponential and hasn't slowed down much.
On a long enough scale you are right that prices will likely normalize to a better level, but before 2030? That would mean the factories are built quickly once they begin.
They all collectively chose NOT to increase supply with increased demand. So if the bubble pops, they just go back to previous prices without oversupply driving the prices to rock bottom.
except cxmt who is plowing resources in like crazy