In May of this year, I was running qwen3.6:35b-a3b on my MacBook (bought in 2024). Obviously not as fast as, say, running a model on Cerebras, but a year ago it wasn't really feasible to have a local model running on my 2024 laptop with vision support. (Concretely, I was passing apartment diagram pictures to Qwen and making it compare different apartments for which ones would feel the most spacious while optimizing for initial moving costs and other factors.)
This was back in May and I wouldn't be surprised if there have been significant improvements since then.
Overall, I think it's fair to compare a workflow like "use llama.cpp locally to upload some pictures and ask questions" to "open the ChatGPT app, upload pictures from your phone, and ask questions". Sure, you can't run a model like GPT-5.4 locally, but the model is mostly an implementation detail here. What a user will care about is: "when I go with the llama.cpp option, am I getting useful information from my conversations?"
What you can possibly audit is probably data sovereignty. For instance, I would not be surprised if Mistral's customers demand concrete evidence that their data is held within the European Union. But that is a distinct issue from training on input tokens.
You are missing out a lot.
Try that with Hermes or Opencode or Deekseek Harness , even Qwen 3.8 27b works really well for that kind of that.
I just ask it to install windows as a vm on my linux and install vs Community 2019 on it , and then build a legacy vb 2019 project on it. and sleep
When i wake up :
It installs Qemu , setup a vm , inside vm download and install windows 10 on its own , clicking next next next as needed , typing in things , writing powershell , python scripts , that run automatically after install by baking into CD that includes ssh server , reboot , it logins into ssh , trigger pythons script that continue installation of vs 2019 community , which includes a driver that click the installation steps , installs nuget , install all depedencies and then build the project into exe after i woke up.
That is with 100% pure local AI .
I also needed a new device anyway - and having this much system memory to run virtual machines has been amazing.
Am paying subscriptions as well tho lol.
Local isn’t strictly about NOT lab. It’s rapidly becoming apples (though not just macs) to oranges to compare the to.
Which is why the premise is silly. To be underwater it would need to be a real comparison. It’s not, and the claude fartifact doesn’t make it so.
What part of my brain is contained here? Sure, the conversations have back and forth (some have dozens of exchanges), but, like, that's not the secret to me. I don't think it can replicate me, and even if it could… okay?
Are you worried they're going to target ads? That the government will steal something? What?
Claude Code has information about my home server, but google or DDG would also have the broad strokes (torrents). I don't know. Maybe others are working on more sensitive things at home.
Its the same point used against privacy. What's so secret you are doing that you need privacy. I think in the end, its about privacy and not trusting these model companies with your data. Facebook manipulated people behaviors with all the data they had, no reason AI companies wont someday decide to do that same, and they have far more intimate knowledge.
When it comes to coding, I also don't like the idea of them taking my money and potentially at same time potentially using as dataset generator.
When Claude is used in a professional setting, any or all of:
Proprietary intellectual property (a.k.a. system code)
PII[0] of the employee, customers, or both
HIPAA[1] data known to a system
Internal communications not meant to be publicized
Sensitive data, such as SSH keys and the like
0 - https://en.wikipedia.org/wiki/Personal_data
1 - https://en.wikipedia.org/wiki/Health_Insurance_Portability_a...
I understand people’s hesitation but the business agreements are different. The business risks of misusing HIPAA data is not only being banned from a massive enterprise market (last I checked, there were about 1.2m jobs related to claims billing and adjudication) and significant legal repercussions.
Other companies like AWS also handle HIPAA data. Are we afraid they’re stealing it? I don’t believe it, nor that Anthropic is training on HIPAA data.
OpenAI… I will never trust them.
And Google's original goal was to organize the world's information.
How did that turn out?
The proof to the Navier-Stokes problem.
I paid $1350 and threw an R9700 in an existing machine. That's a 4 month pay off or so.
Plus, I can feed it sensitive data all day and not be worried where it's going.
Idk about the quality of this setup but just pasting it here as an example. https://explainx.ai/blog/heretic-llm-abliteration-guide-2026
When does the average person actually need to do that?
I expect this is only going to get worse. "Censorship" isn't just going to be about who you vote for and which political party the model will say nice things about and which it is more likely to say bad things about. It's going to become about whether the hoi polloi are allowed to have effective AIs at all. Like the 1990s internet, AI has outrun a lot of power structures but that is not going to continue indefinitely.
So I can certainly understand why someone would want the guardrails gone.
Case in point, last week I was poking Opus 5 into writing me some RPi-pico firmware for driving a small e-paper screen. Font was built in right into C code as hex constants. Space being tight, I asked if there is some clever compression that could be applied. Claude thought for good 10 minutes, then guardrail kicked in telling me that was "cyber", and refused to continue.
Local inference makes sense for speciality needs, or very small models. But if your model is bug enough to span GPUs its excessively wasteful to hoard those GPUs for yourself without piggybacking hundreds of other conversations on top of all that memory bandwidth and matrix multiplies.
-- Warren Buffett
I don't know when we'll have an open equivalent to Fable, let alone whatever (insane) hardware you'd need to run it locally.
For each usage level, it lists the quickest pay-back in each capability class, with each model on its quickest machine and one click into the calculator to change the assumptions. Short version: at 1M tokens/day the best Sonnet-class option is Qwen3.8 27B on a Mac mini M6, 8.3 years. It only drops under a year if you're running agents at around 20M tokens/day.
At 7 tokens/s (Mac mini) you max at 600k/day so you couldn’t hit those higher amounts like 4M where it says 2 year payback?
You're competing against companies that get tax breaks, locate themselves optimally, and have large economies of scale.
Also, if it did, the hardware would be bought up, raising the price until there was no economic profit again.
If you can find a unique application for it then maybe?
Having said that, it will never pay for itself. A simpler more absolute math is, if I buy a Mac and use it to sell tokens on OpenRouter, will I make a profit? And the answer is no.
Mac mini can also build iOS applications. I think if you’re a mobile dev, you can have concurrent builds for your agents instead of everyone waiting on a single machine to finish.
AFAIK, a limiter for iOS engineers (and AI agents) for concurrent feature development is the xcode environment and hardware limits. BE engineers can easily have 3 agents working on 3 different microservices (or gitwork trees), but iOS devs can basically only manage one version of the code at a time, due to externalized state (like derived data and bundle ids).
The premium is not having your million dollar prize and career stolen by billionaires.
Some improvements that I think would make this more useful:
1. Allow manually setting tokens per second, or as an alternative, let me jack up the number of tokens a day.
2. A sort of backwards flow "if you want to run this, at X tokens per second, with Y context, you'd have to spend Z."
3. Add support for configuring multiple RTX 6000 variants.
When I was making heavy use of DeepSeekV4-pro I was burning somewhere around 1.5 billion tokens a month, and that was just using it in my free time on random projects. It was something like $24 at the time because of the initial discount/promo period. I don't think there's anyway in hell I could ever run that (on current hardware) for less money.
I think the calculator shows from a purely financial standpoint what we all know... that yeah, it's definitely not worth it if money is your only concern. That (cost per token) will eventually change. Models will get better, more efficient, VRAM prices will come down, VRAM capacity will rocket upwards, and the economics of it all will change. It would just be really cool to have the calculator show me exactly how cheap they'd have to get for it to make sense.
I need to finish up some work and make dinner, and if no one else beats me to it (anyone is welcome to) I'll ask Fable or Opus to knock that out.
I sell those to business who need local air gapped requirments and I make a lot more money!
I can run the alliterated models where none of the service prvoider even dare to provide.
THose benefits outweights a few K.
And show me an api provider that allows me to run 10x agents concurrently for 5 days straights .
Any of them on a Max/Pro plan as long as you are smart about model selection? That's my main objection to local inference, I'd need a whole rack of GPUs to do as many things in parallel that I can do for $400 a month. I do plan on setting up some local inference hardware, but...RAM and GPU prices alone are $$$$