Assuming you’re willing to drop a fat stack of cash on the upcoming Mac m5 ultra with 512 gb unified memory, you can even run it locally, quantized to 4 bit. Whether it’s even slightly reasonable, well, my wife would probably skin me alive but maybe yours is more understanding.
* LLM usage is new for the world
* Models are evolving quickly with high worldwide competition
* Hardware is evolving despite RAM shortages
Is investing a huge sum of money in equipment for local inference a wise use of money? Or are M5 Ultra and equivalently priced local inference hardware future-proof enough to be worth it relative to how the market is evolving? Maybe it’s all a question of what you’d spend otherwise on serverless or dedicated GPU spend…
However there are other reasons (e.g. privacy) that might make it worth running locally for some people.
The models are currently improving much faster than the hardware and this doesn’t seem to have plateaued yet.
The object permanence of not having to reinvent the world every time a model gets sunsetted has value.
If you're paying Anthropic or OpenAI to use their models, harness, governance, etc., I could see the local inference potentially coming out ahead. They're already starting to ratchet down what your money gets you on their platforms, and that can be expected to continue as the leaders of those companies continue to seek the road to the El Dorado that is being a trillionaire.*
If you're looking to get into the guts of AI development instead of having it handed to you by a provider, that's where it gets murky. I'm wanting to write some sort of agent that does things and get into making outputs consistent in the like, and I'm not sure whether to host something on GCP or buy an M5 Mac.
*Note: El Dorado is a mythical city and many people died trying to find it.
There are several things I do in my life that only pay off in the event of a big disaster, like an extended internet outage, civil unrest, supply chain disruption, war, etc.
I like to be able to do the things I do even if offline for weeks.
I spent a lot of money for more flash in my iPad Pro so I can keep all of offline wikipedia and OSM in it, for example, along with tons of books. It’s sort of like being a digital prepper. (Being a prepper is a spectrum, from anyone who keeps food in their pantry to people building bunkers under their house - how much you invest is a personal prudence and threat modeling decision.)
Also, privacy. And when I got the Mac Studio the 512GB was only $15k, which is dirt cheap for that much VRAM.
The surprising thing for me is how much work you will need to cool the banks if you’re near your memory ceiling. My memory starts soft throttling at about 74C (dies may be hotter, that’s the bank temp) and will turn down speed to try to stay below 80.
Happy to send my llama.cpp config settings if you want it.
That said, I am guessing my problem is not enough RAM - but this poor consumer platform struggles to do memory training with 128 GiB as it is.
Now I surely regret not having gotten Threadripper and 256 GiB of RAM in the before-times.
For what it's worth, all of my hardware was used; I think, all-in, I'm probably at around 3k-4k USD? Not cheap, but also not the worst for something relatively versatile.
Which is completely insane for a ten year old configuration.
Does inference make full use of the memory bandwidth in a NUMA system?
Using an example of llama-server on a 256GB (each CPU is attached to 128GB) system running qwen 3.8-flash-next Q8, which takes up about 188GB with the model and context, it's not that the inference engine 'knows' about NUMA, but if you use --load-mode mlock to load the whole thing into memory at once at the time llama-server launches, the kernel will distribute it relatively evenly across the memory connected to both CPUs.
Though I appreciate how good it is at "solid" grunt work and at that price (in fact I am paying the grandfathered subscription price; mostly).
I am planning to let go for my Claude AI subscription which I now use only for "planning" and maybe use that via Open Router as PAYG (at to try how it ends up). But god glm is bad at "talking" and "responding" anything prose. Not only quality but it's almost impossible to tune it and make it let go of its habits and biases and enthusiasms which often result in too many too and fro.
So I sometimes wonder at what point that starts becoming the cost and mental hassle. Maybe it's not there for me yet.
- It is trivial to extract samples of the training data that was used, which can bolster existing lawsuits/foster new ones.
- Older models are not as safety-hardened, so it is easier to coax unsafe behaviour out of them, which is a PR risk.
- It may be possible to divulge proprietary secrets from the model (e.g. architectural details that may still be relevant).
For these reasons, and more, it's unlikely that GPT-3/similar models will be released until these concerns are no longer relevant (e.g. when they become a purely historic concern, similar to the open-sourcing of other proprietary software from decades ago).
The weights arent the truth tho, maybe a timecapsule-vhs but i wouldnt trust llm weights more than more hardcore deterministic media that might get preserved to infer facts from an era.
The companies doing the training are becoming the "winners" that are "rewriting history" as they train their models.
What's the point of publishing it when it'll likely be outclassed by gpt-oss?
AI policy is being shaped somewhat by the things Sam and Dario say. So even if you're not feeling vindictive, it's probably good to keep a track record of the previous things they have said as a Bayesian prior. People who don't know better listen to these people, and maybe they shouldn't.
I think the release of kimi k3 is definitely arguably dangerous, we're already seeing consequences of elite-tier cyberoffense capabilities.
According to the book The Thinking Game, lower quality models at that time were considered less safe, because they could be easily tricked into doing harmful stuff. In the book, Dario (of Anthropic) was the head of safety at openAI and was responsible for pushing for 10x scaling in training to make the models safer.
It does make sense, a smart model is going to be way harder to jailbreak into telling me how to synthesize anthrax (or something).
In the times of GPT-3 I'd scoff at the idea of an LLM doing any hacking; today, I'm running several AIs on my code before publishing, and they are finding (and demonstrating!) RCEs on my localhost server.
For example, one found a missing check in a third party JWT library which allowed full account takeover, which I'd have never even looked at.
Hence I don't believe a single word coming out of these people's mouths. Their "beliefs" are just marketing.
According to me, this is nonsense.
I cannot stand using gpt-oss, but I miss some of the creative spark of GPT-3 davinci dearly.
We're nowhere near a Fable-class model IMO, but things are going to get interesting in this next year.
If it's significantly larger than GLM 5.3 (I've heard some insane guesstimates out there like upwards of 5T params or more), that would prove rather embarrassing for Anthropic.
I feel like most benchmarks cluster on a reasonably limited area of human knowledge
You do pay for the tokens, but in theory on a smaller model each token is cheaper.
I should do that myself, come think of it.
Not really, in that you just work with different constraints.
Anthropic and US labs in general has maybe 100s to 1000s of GPUs per person to experiment. Zai and Chinese labs in general have 1-10.
The priorities are different.
z-ai/glm-5.3: also Z.ai, Novita, Atlas Cloud, IO.NET
My visit to your website went like this:
1. Visit models page
2. Try to find GLM-5.3-Flash (which is among the ~5 models that 90% of people currently care about)
3. Give up scrolling (which would have taken OVER 50 SCROLLS!!!) and use Ctrl + F
4. Try to find input/output/cached price
5. Scroll all the way up to find out which column is what
6. Notice that output price is cut off
7. Notice that the scroll bar is over 100 scrolls further down the page
8. Use Shift + Wheel to scroll horizontally (most visitors probably won't know this trick)
9. Notice that cached price is missing
10. Conclude that this is probably not a serious offering and bounce
There are probably more issues later on, but this is how far I got.
I would suggest you to:
- Deslopify all pages that a user may visit before conversion
- List important models first (see OpenRouter rankings)
- Move the most important information (model name/input/output/cached price) to the left
- Disaggregate the prices per provider (maybe subtables per model? not sure)
- Measure cache hit rate and compute effective price per provider (see OpenRouter)
(- Optional: Fix the broken link on your HN profile page. Currently, the only way to get from this comment to your website is a search engine.)
I've seen dozens of conversations about it in last 24 hours, and every major inference provided added in first 24 hours. I think it's gaining plenty of traction.
We saw the same issue with GLM 5.2 when they still published publicly who the providers are on their website. Most ran FP8 but one was doing FP4, so you had this issue where one moment you had the better FP8 and another session you had the FP4 provider.
You can check the internet archive, it was in the FAQ part before they hide/removed it. So if you looked up the providers, and the published quants, yea, ...
Given that a lot of complaints are coming from people that felt OpenCode Go Flash feel like a step down compared to old OpenCode Go/DeepSeek API directly, it smells of a quantized down provider is mixed in.
I use DSV4-Flash on Max through Deepseek's API. I have been using GLM-5.3-Flash on High through Openrouter which I thought had a 50% discount. I must be doing something wrong for the costs to be off this much.
There are other providers with much faster inference, like BaseTen at >100t/s: https://openrouter.ai/z-ai/glm-5.3-flash#performance
I hate how difficult it is to compare prices when looking at subscriptions.
Would $20 in open router, using models like GLM get me more or less?
Z.ai does have their own subscription, but I haven't used it because their privacy policy was pretty buns last time I checked.
What did you find objectionable? I looked at it when I subscribed almost a year ago and I was fine with it (e.g. they don't train on your API inputs).
I burn through my current Google AI pro sub for the week in about 2 and a half days so wanted something extra to add to it, but don’t want to buy any expensive ultra plan. Flash models have got me about 98% of what I need, but quotas are still a bit low.
- price: DS4 wins... $0.0235 vs $0.0242 for ten tasks
- latency: GLM wins... 108s total against 154s
this is for a personal use-case where i'm detecting ads in a written transcript. sticking with ds4-flash for now since latency is not a critical factor
