Particularly, I had one team member who was extremely sceptical of AIs/LLMs/harnesses and refused to use them. One day he said "Well, I have an RTX 5090 doing nothing... should I try to get something up on it?" and a few minutes later he had 3.6-35B loaded up, running OpenCode.
It continues to be a workhorse to this day, running on both my local Mac for various types of jobs, an AMD R9700 at the office, and said teammember still uses it on his 5090, although in practical terms we do a lot more with DS-V4-Flash-0731 these days.
- You at least need a capable machine, so that's not 0 monetary investment. - You need to spend at least an hour decicding between ollama, llamacp, mlx, etc. - You need to find the correct quantized version of the model that works for you based on the architecture. - You need to figure out the correct context window size to get reasonable performance. - You need to setup a harness that works against your model - You might need to setup additional websearch tools, image tools, etc since harnesses like pi don't come with the model. Ofc you can't use codex and claude code, because those aren't opensource and you are anti-AI.
Or, you could sign up for Opencode for $10 and just be productive.
I'm particularly calling out the hypocrisy of the original comment. Being Anti-AI, and then spending hours on setting up a less than frontier AI model.
It is 0 monetary investment if I already have said machine lying around doing nothing.
Which is exactly the story OP talked about.
Pi has a nice guide on it (https://pi.dev/docs/latest/llama-cpp) and it is really not that hard.
How is that hypocrisy? Self hosting is somehow anti AI? Its not anti AI. Its literally using AI!
…and honestly, at a higher technical level than slapping your wallet against a token provider and running prompts in a hosted sandbox you can't even see the prompts in.
Local: You need to invest $thousands into GPU and/or very-high-end CPU+Memory hardware.
Vendor: You can use any existing device, even a phone or tablet. A very low-end laptop is fine.
> takes literal minutes to get started
Local: Typical scenario is hours just to download the software, the model weights, and then faffing around with CUDA and matching your GPU drivers.
Vendor: Free-tier available instantly on a web URL. Even local agents have free tiers from multiple vendors. Install is a single command and/or download and "next,next,next,finish" wizard that takes ~1 minute.
> you can just `rm -fr` it and forget the whole thing existed.
I'm still cleaning up multi-GB model weights floating around in hidden subdirectories under my user profile from months ago when I was experimenting with local models!
Meanwhile I simply... stopped using Gemini. That was the entire process: I no longer actively use it. They stopped billing me for my token usage, because it is now zero. That's... it.
You have it totally backwards.
Are you trying to say that local models are hard to use because... you're having issues handling files properly? I am not sure I get the argument.
I get the rest of the comment: local models require an investment upfront, and it is less convenient. It doesn't say that it is not cheaper, though.
I guess they were using ollama, which does not tell you where it puts the models it downloads.
Same story on an RTX 4060 Ti 16 GB. MTP is a serious boost to tg.
Downloading the model is a simple hf command that HuggingFace's web UI even gives you.
llama.cpp is trivial to use, and so is llama-swap, if you want to use other models too.
If you don't know what arguments to run it with, you download ggrun and use that.
Local LLMs are incredibly capable and don't need expensive hardware. A $500 GPU will do. Or even cheaper.
This is all trivial.
That's one of my main issues with AI anyways, the thought of having all my data go through some sketchy foreign (to me) entity with questionable motives and under a questionable regime.
Local AI solves for all of those.
If you're learning about model inference, then it's a different and you are definitely not anti-AI in that case.
It's not that billing is complicated, but learning to set up a local AI is a lot more useful and more rewarding.
I have an RX 6700 XT with 12gb vram and 64gb system ram. running dense models like 27b is difficult, but i can run IQ4/IQ5 qwen 122b-a10b or 35b-a3b at ~20tok/s
Isn't that just the definition of MoE vs dense ?
On the other hand having 90t/s with any local model is nice and Pi with loop police extension can prevent looping a lot.
I honestly think that with my electricity prices running qwen 36B myself is more expensive than hitting the cache rate at deepseek.
I gave it a try for a few days (pi + openrouter + deepseek-v4-flash via deepinfra) and ended up paying ~$18 for rather light usage. Yes it's still cheap, yes it's fast, but i feel i would still get a better deal with a Claude subscription plan.
https://openrouter.ai/deepseek/deepseek-v4-flash-20260731#pr...
You want to use the newer quantization formats like Unsloth's UD quants or oQe, where the weights are selectively quantized using a calibration dataset so that important weights are left at/closer to full precision.
The weaker point is prompt prefill, which starts at 1,400 tokens/sec but decreases significantly at high contexts. That said, for agentic scenarios, if you're using a harness that doesn't needlessly bust the cache, it doesn't feel slow.
I really hope they release a Qwen 3.8 35B, although the lack of a mention seems ominous.
Kind of slow, but using only 14W of electricity so the Wh per task is twice as good as using my i9-13900 laptop with 4060 GPU.
I personally have the Framework Desktop, but there's also systems from other brands like Bosgame
Huge fan of that thing, it's th e Linux MBP I've always wanted.
Tolerable and usable for some things, though thinking makes it take about a minute to reply in many cases. But getting this kind of thing to run on 8GB of VRAM is the main benefit of the mix-of-experts setup: it can do partial GPU loading and get a ton better throughput than a similarly-sized dense model (like 5-10x, sometimes more).
Kimi K3, GLM 5.2 and now Qwen3.8-Max - open weight models.
DeepSeek V4 Flash outperforming Gemini 3.1 pro, probably DeepSeek V4 Pro update is also coming soon
Chinese labs are cooking very hard. US closed weight labs are probably hard time to resist not calling Washington DC for more AI regulations
Not sure how Qwen3.8-Max is going to be licensed, hopefully it'll be Apache like the smaller ones.
There seems to be quite a gap between the small ones and the enormous ones these days.
Just so that we know what 3.8 would be like.
I currently have about 150 Tabs of Antirez posting on AI and running local model I haven't had the time to read. And there are probably some prerequisite reading or other research in between as well. I just wish there are some very high level overview and news coverage on all these.
IMHO this is a difficult question to answer. Part of the power of paid models comes from the software supporting it. With local models, you have tons of workflows that can severely influence the quality of the result.
In my personal experience, the SOTA models are way more consistent and can handle more complex questions. Part of that is (probably) because I don't let my local model access the internet, while paid models do use the internet to look at docs etc.
What's even their end goal? Open source models make sense, if profit is not the target, but for OpenAI and the rest, once they achieve "AGI", don't they basically become useless?
All requests to an LLM are idempotent, for every API call you need to send it the entire conversation history so that it can process it. LLMs do not learn or remember anything, which makes it super easy for users to switch LLMs on the fly. Most popular AI frameworks, make this a one-liner change these days.
And that makes me wonder if the trillion dollar valuations for OpenAI and Claude are even justified. Cause if that is justified, then Kimi, Qwen, Deepseek etc are also valued at a trillion dollars. Or all of them are worth a lot less. One of those statements is true.
Also this makes me wonder if the next iteration of LLMs would be based on fine-tuning, where LLMs actually learn from your past behaviour so that it would grant some amount of stickiness to the product. OpenAI used to offer fine tuning runs for GPT-3.5, but they don't seem to do that anymore.
https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...
A more appropriate term is “stateless”. LLM responses are certainly not idempotent, as they are not even deterministic.
https://earendil.com/posts/session-portability/
“Just take the session thread to another provider” might not be feasible anymore soon-ish.
Generally speaking they do, at least from my experience when switching from one model to the other - their performance decreases, and they often do large refactors outside of the requested scope as they try to bring the code closer to 'their' style.
Which makes sense imo - they'v been trained to iterate over the code they wrote, and not code that was modified by someone else in the interim.
They don't. The moat will mainly be the tooling around AI, not the AI itself. You don't hear any company claiming their moat is the Internet.
Recursive self-improvement changes the parameters a bit, especially for the market-leaders, and it's the one thing that makes me wonder if they'll be able to extend their lead faster than the smaller labs can keep up, but it's an option available to everyone.
This isn't without it's challenges however.
1. This will increase costs drastically, since they would need to a run a sandbox per use to ensure data isolation.
2. Increased latency, and this directly limits how much of the harness can be moved to the cloud before the users notice sluggishness
It’s more like a bunch of people are placing different bets. Only a few bets are going to generate a return, possibly only one, but the profit on that one bet will make it all worthwhile. That’s the theory, anyway.
It is clearly distilled with an aggressive pro West bias to increase sales in the West.
Original designs: https://image.non.io/257dc9cb-9e6b-4e00-8f12-23ea5e073649.we...
These are fairly rich, pattern-heavy, nuanced designs. I've asked each to create it as a SPA where the map flows behind it.
Opus 5 results: https://html.non.io/opusAcmeBooks
Qwen 3.8 max results: https://html.non.io/qwenAcmeBooks
Same prompt for both for the conversion. I used OpenCode for the qwen version, but I encountered a significant amount of errors / timeouts while it was running. Claude finished in around 16 min, but I spent close to 2 hours shepherding the Qwen build. For the implementation, there were signs it had good vision, but the timeouts make this very hard to use in a production setting.
That folder didn't exist, I asked Qwen where the files were, and it attempted to create them again.
Stopped it, and asked what happened to the original files. Qwen Desktop apparently doesn't have access to the local file system, but continued merrily along without warning about that.
https://image.non.io/07a153cd-c31c-4f7b-a89d-363faec05a91.we...
It did a self update and it worked for a single request (me just saying hi). Pasted in the prompt to build the page and it 401'd.
I'm quite surprised at the difference.
I've noticed another type of AI slop that's prevalent in all the popular models; font sizes and variations like you wouldn't believe. It's very hard to instruct LLM's not to do this.
However, that would just change the weights values and not their dimensions.
Imagine our LLM as this one-dimensional matrix (instead of unfathomably many, many more dimensions):
0.728201017
-1.9293839303
2.93020202929
5.82290101728
-0.2844920917
…
And then, all of these values are multiplied by some scaling factor, essentially vectors pointing in some direction (in our case, only on a one-dimensional line instead of in some higher-dimensional space).
Now imagine it’s possible to “remove” the portion of the weights’ directions that’s responsible for some undesired property like history knowledge, or another programming language:
0.428201017
-1.6293839303
2.63020202929
5.52290101728
-0.5844920917
…
(For demonstration and faster-to-type-out purposes, I’ve changed each vector by deducting a constant amount of 0.3, which doesn’t make much sense. In reality, it’d modify probably all digits in some direction.)
Now we can see: the number of rows has stayed the same, but the value has changed. This means the required storage (or computational) space remains the same.
It probably sucks for you if the proposed powerline goes through your house, but the need of the many come first.
Having cheap and plentiful power is a huge market advantage. It was one of the factors driving Norway out of poverty 100 years ago.
Also they really don’t have the chips to supply those data centers, and the government isn’t so crazy to go that aggressive on AI, they have lots of other priorities as well.
EDIT: This was obviously meant to be facetious with the intent to highlight the negative effects on the general population that datacenter buildout has in the US.
States have had trouble delivering expected EV charging capacity btw
Why aren’t we considered energy limited?
I have not looked into it, but I have heard a lot of new data center have to run on site gas turbine on trailers to have enough power.
If the US is not energy limited, why they do that?
The need of the rich, you mean. In these kinds of situations it's never about the many, if it was about them different choices would be made.
volunteers in the open source community use whatever is easiest and cheap.
open weights is not open source. there is no "trust".
I don't understand. That's dated today, but:
https://twitter.com/alibaba_qwen/status/2078759124914098291
> Qwen3.8 is launching and going open-weight soon! [...] You don't have to wait to test it. Just now, the Qwen3.8-Max-Preview made its debut on Alibaba’s Token Plan, Qoder, and QoderWork.
That was on July 19th. I used it to draw this pelican: https://simonwillison.net/2026/Jul/20/afraid-of-chinese-mode...
So what are they releasing today?
I believe they mentioned around the preview announcement that they'd be releasing improvements to capability, which I assume means continued training.
3.8 Max Preview ran 90% discount on QwenCloud exactly as when Moonshot couldn't keep up with all the signups.
Here's a pelican I just got out of the new model. It took 11 minutes and forgot the wheels! https://tools.simonwillison.net/markdown-svg-renderer#url=ht... (scroll to bottom)
The reasoning trace is pretty great:
> More additions: basket with fish in it? Cute detail — a fish poking out of a basket on the handlebars! This adds charm and pelican context.
If the price is $2/$6 that cost me 17 cents: https://www.llm-prices.com/#it=90&ot=29734&ic=2&oc=6
> Qwen3.8-Max was asked to create the oh-my-cli project from scratch and, over a 10+ day long-horizon autonomous coding run
10+ days of building what exactly? Is that a shell prompt customization toolkit? Or a coding harness??
Neither - well, sort of the second. The poorly named thing is a self evolving coding harness, the self evolving part makes it a big deal.
I wish they had picked a different example.
There's Mistral. Although they've been surprisingly quiet for the last 6 months.
Open weights is more beneficial to Europe spring-boarding a catch up than defending Claude/Anthropic.
They seem to be oriented more toward customizing models for the concrete needs of a company, on-prem deployment, proprietary knowledge-bases, etc.
I agree, but you are underestimating EU regulatory incompetence
Level 2: ... or with anyone who does.
Level 3: Anyone caught with unauthorized model weights is subject to the same prosecution they could expect if they had been caught with CSAM.
Then they call your local law enforcement with an anonymous tip about this suspicious dude who's long overdue for a totally random home inspection.
Making it illegal even without having a realistic means of enforcing it still chills behavior, and so just because it isn't practically enforceable in the general case doesn't mean it isn't without consequences.
They can also restrict US providers from serving these models. Looking at OpenRouter, there is currently 11 providers for 'DeepSeek V4 Flash 0731' with the only non-US one being Deepseek themselves, which aren't even the cheapest.
Interesting.
If I were a western lab, might I be tempted to subsidize at least one faster & cheaper provider for Chinese models to keep even more data from leaving the US?
(I know in this case there can just be more efficient providers, am asking more than speculating)
Crazy how all the 5 eyes nations banned Huawei within like 18 months of each other.
The open models are now good enough for what I want to do with them, let alone any future improvements. And factoring in efficiency gains, a model in the ~70b range starting to satisfy my needs would completely obviate the need to pay others for inference. This does not seem far-fetched to me, comparing with where open models were at this time last year. What am I missing?
So just like in IPhone vs Android, you could end up with a situation where Chinese firms compete and get most of the revenue and usage with low profit margins but OpenAI and Anthropic capture a premium side of the market and still get a lot of profits.
I have numerous cases where Sol failed and only Fable could solve a problem. For example yesterday I was merging a Q2 curved with a Bezier curved face in 3D using OpenSCAD. I tried for over 2 hours with Sol 5.6 high and x-high.
Fable two-shotted it in about 30 minutes.
In my experience open models (or GLM, DS and Kimi) are radically worse than either of Claude or ChatGPT at these tasks.
I think there is a huge "long tail" of tasks like this where the frontier labs are ahead, and I think this long tail is quite important.
Unless you are spending more than a max subscription (200 a month+) its cheaper to use the cloud.
But things are priced cheaper in the cloud now to lock you in and restrictions around hosted models are getting worse.
If you only have a $300 dollar laptop its probably not worth the upgrade.
I'm personally excited by local AI but the experience for the average isn't the same. I'm willing to get .5/s running on 10-15 years old machines but what I can do with it is limited.
Their marketing department :-) . I'm only half-joking; those guys are hard at work finding the best product-market fit for ChatGPT/Claude. "Product market fit" means "strongest revenue", which is not necessarily going to bring the best tool for you or me, but the one that can either get more consumers to shell off money, or more enterprises to cough money for licenses, and in both cases those consumer basis will be narrowed down to what legal and geopolitical circumstances allow OpenAI and Anthropic (and this is why they want to ban the competition!). It also means dark patterns and enshitification, of which I'm already seeing some both in the Codex interface (it was just renamed "ChatGPT"!!!) and in Claude Code (which also is just "Claude" now and can't '@' properly any longer). So in the medium run most people will be better off running an open source harness that can use any model.
you can't create a new drug by running a model on a laptop. You can't serve a customer support bot running on a laptop. You can't generate video in bulk for many users on a laptop. So there is still a case for paying others for inference.
Does it justify the valuations? No idea, but some major use cases are still there. That's why they are rushing to implement, OpenAI creating a "deployment company", Anthropic having some pharma rumors, etc.
I think GP was asking specifically about the use case for closed models.
Or, the company itself can rent the GPU and host the model of their choice on it.
> But they would have to keep up with how good the private models are.
Why? The whole point of the thread is that the marginal gains of using private models is a fraction of the additional cost of using those private models.
IOW, if you really need that improvement over the open models, you can go ahead and pay it. What we are seeing though is that the improvement in general is barely noticeable but the cost is 4x as much.
LLMs are useful. We can all see that in agentic coding. But replacing everyone’s job? Hardly. And what’s with the scare tactic of trying to get the US government to ban foreign models?
LLMs are useful, and dare I say they’re on par with the internet. Making them cheaper and affordable is good for everyone. The fear mongering from Anthropic and OpenAI looks like an attempt to corner the US market into using only US models so they can keep the profits, especially since China has proven that LLMs are a commodity. US AI labs should work on making LLMs cheaper or better harness. Altman and Dario are not trustworthy.
> "Anthropic believes that they are the ones who should have final say over how Anthropic is used; given that they think only they should be developing leading edge AI, they by extension think that only they should have final say over AI generally. When you further combine this realization with the company’s pronouncements about AI’s ability to conduct all economic activity, you realize that Anthropic’s leadership effectively wants to have power over everything and everyone."
"Concrete Problems in AI Safety" by Dario and another founder was published 2016, anthropic was founded 2021. Among a bunch of other examples, including other founders.
If safety is seen as a joke now, it was really seen as a joke back then. The reason they keep shooting themselves in the foot with bad PR is because they truly believe in risks.
> “My scorecard, at the highest level, would be we’ve been roughly right on technological predictions and pretty wrong on the social and economic implications” https://www.cxtoday.com/ai-automation-in-cx/sam-altman-softe...
I agree re: Dario quietly pushing for government control. He also said LLMs would replace a lot of entry-level information jobs, doubling the unemployment rate from 4-5% to 10%.
Yale did a study recently showing little impact on employment in high-AI exposed jobs https://budgetlab.yale.edu/research/ai-probably-not-yet-reas...
Yet here we are, still driving our own cars, and laughing at silly behavior of most advanced self-driving systems.
I am certain we'll get to something of similar value, but I expect we'll have to improve infrastructure too (smart roads).
Their base models and architecture has quickly become the go-to for local inference and fine-tuning, even when they introduced some tricky things like GDN, so many people use it, that it was matter of days/weeks until lots of OSS frameworks adopted it.
Edit to answer my own question:
License file [2] states:
> If the Licensee or any of its affiliates operates a Model as a Service business, and the aggregate revenue of the Licensee and its affiliates exceeds 20 million US dollars (or the equivalent in other currencies) in total over any consecutive 12 months, the Licensee must enter into a separate agreement with Moonshot AI before using the Software or its derivative works for any commercial purpose.
[1] https://huggingface.co/moonshotai/Kimi-K3
[2] https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE
Just like we have professors with specialisation wondering if AI models can also be so.
Intuitively it feels like focused models should be better models, but human programmers are ‘better’ knowing programming alongside general stuff. We’re not reduced by reading economics or Tolkien, removing such knowledge would be premature optimization.
>we have professors with specialisation
Yes, but any professor with a specialisation is an erudite with vast knowledge across the board.
I'd argue we can have a meaningful BIG model with a specialisation but not vice versa.
PS: all of this is about a model that is expected to be able to execute development tasks on a human level. Obviously we can have small models that are very capable in things like TTS or STT.
If you throw general purpose model at a codebase - it will look at the table and data logical connections beyond what is explicitly declared. It will figure out on its own that Salaries should be displayed on SalariesTable.php and it will "know" that your prices should include vat and so on.
A human knows that VAT and price go together and are related, full size LLM does too, stripped one - doesn't.
Nice!
> Qwen3.8-Max comes with the official support for reasoning_effort, which can be used to adjust reasoning depth and control cost:
> xhigh (default): for complex tasks demanding thorough analysis
> medium: balancing accuracy and speed
> low: efficient reasoning optimizing for speed and cost
I hope this is significantly cheaper. I've been loving Deepseek for it's nearly free usage costs, hard to justify switching from cents per day.
> Pricing: Input: $2.0 / M tokens Output: $6.0 / M tokens Implicit Caching: $0.25 / M tokens
Surely Alibaba is leading by example here by reducing work hours per week while keeping pay the same right? Right?
Ironically it's Electron BS, so actually, Sol could probably build itself...
However, they are not at the point where they can effectively train themselves, nor did they are capable of researching their own method of learning. SWEs in mid-corps on my country are right now relegated to reviews and sanity check, basically babysitting the LLMs and making sure they're not spouting nonsense. If you think about it, that's basically QA and can also be delegated to another AI. If Bun's rust rewrite that they tout as fully LLM-led can pass the test of time in a year or so I think that's it.
I believe all that is now constrained by compute and capital, not tech.
Are you not already experiencing this? I think this is fairly common for people using AI now, though the time may not always go into hobbies or sports. It's common for me to setup Claude with an hour+ task while I catch up on housework, or while I'm getting ready in the morning.
In the last couple of weeks I've unfortunately had multiple family illnesses - it has been helpful to have Claude keep up with much of my product development programming work while I visit my mother in hospital and check on my father's recovery. I'm able to give more time to family without worrying that business progress isn't keeping up. The overnight Claude sessions while I'm asleep have been particularly helpful.
It's infinite work, I just did more work while codex was doing it's thing in the background.
Already 38% of Americans aged 16 or older do not work and are not looking for work and yet are not dying from hunger or exposure. This would have seemed like a utopian dream to someone from the 1800s.
Being 16, 17 and 18 and not working is expected, given they are still at school. A good percentage of people aged 18 to 24 are studying full time.
This stat seems a bit misleading and doesn't support the "you can live and not work argument". How many of those people are retired? How many of those are under 18? How many of those simply cannot work? How many of those live at home or are on food stamp?
... these days. 100 years ago you'd have a job at 12 in many cases, 14-16 if you were more lucky.
I'm not even talking about peasants from previous centuries.
Lmao I love their video with the idea that people will be able to do their hobbies while ai does their job.
Before AI:
I have 10 tasks that take 1 hour each to do.
After AI:
I have 100 tasks that take 10 minutes each to do.
Same amount of time spent working, maybe even more stressful, just more productive.
You can always try out this model for free on Qwen Chat. Alibaba Cloud has too much friction for me.
The last thing I want my personal agent to do is to write up code and run arbitrary commands. That is practically a legalized RCE.
I mostly just need it to mostly be a very advanced NLP model that is able to figure the set of tools that it needs to call with what argument. Then it will just follow a predefined decision tree.
How are you all justifying economical use of these local models right now? What's the cost efficient way to do this and do better (even with models evolving over time and losing now vs later) than the big labs?
The point of local models is privacy, offline use, and maybe no guard rails.
* Not talking about enterprises that buy DGX racks and host Chinese models for internal use.
But also, control and consistency. A local model cannot be changed out under your feet like an API model can be.
We all know that is hugely subsidized, and I guarantee that OpenAI and Anthropic are looking to enshittify that ASAP.
The enterprise users, however, are not subsidized like that. They pay per token. And some developers in those companies are chewing down a lot of tokens. Self-hosting an open weight model could be a massive savings very quickly. It also gives them negotiation leverage when talking to OpenAI and Anthropic.
For all we know, inference might be dirt cheap, they might just be hiking the API prices so high for us to think subscriptions are subsidized.
Now, the one wildcard in all of this could be Google. They are on the eighth generation of their TPU and have been holding their cards extremely close to the vest. I don't think anybody has a good read on exactly how much capacity they have. Most things you can kind of figure out the overall business numbers and what's going on in Google--the TPU area is one of the exceptions. I know a couple of big customers and even they don't have any visibility on that front.
There's a lot of AI models that each 'win' one week over the next. The pure definition of a commodity market and not a 'winner takes all market' as valuations would imply
i might end up cancelling claude, anybody else thinking of the same ?
They don't explain how successful that went but it's a bit hilarious seen that an Anthropic dev explained that it's been 15 days Claude was hard at work --with nothing to show yet-- trying to rewrite itself in another language.
"You rewrite Claude Code, we rewrite oh-my-pi."
"You're nowhere after 15 days, we do it in 10."
Sure, it's apples to oranges and all that. But part of me thinks they know fully well what they did there.
Does this mean they distilled Claude? Sounds like what Claude Code will often do.
What is important is how long it can go without requiring human intervention. Not just that it's possible to run on its own for a time.