A laptop with an iGPU and loads of system RAM has the advantage of being able to use system ram in addition to VRAM to load models (assuming your gpu driver supports it, which most do afaik), so load up as much system RAM as you can. The downside is, the system RAM is less fast than dedicated GDDR5. These GPUs would be Radeon 890M and Intel Arc (previous generations are still decently good, if that's more affordable for you).
A laptop with a discrete GPU will not be able to load models as large directly to GPU, but with layer offloading and a quantized MoE model, you can still get quite fast performance with modern low-to-medium-sized models.
Do not get less than 32GB RAM for any machine, and max out the iGPU machine's RAM. Also try to get a bigass NVMe drive as you will likely be downloading a lot of big models, and should be using a VM with Docker containers, so all that adds up to steal away quite a bit of drive space.
Final thought: before you spend thousands on a machine, consider that there are at least a dozen companies that provide non-Anthropic/non-OpenAI models in the cloud, many of which are dirt cheap because of how fast and good open weights are now. Do the math before you purchase a machine; unless you are doing 24/7/365 inference, the cloud is fastly more cost effective.
Oh yeah, seems obvious now you said it, but this is a great point.
I'm constantly thinking "I need to get into local models but I dread spending all that time and money without having any idea if the end result would be useful".
But obviously the answer is to start playing with open models in the cloud!
Do you have some links?
Also I assume the privacy implications are vastly different compared to running locally?
Power consumption? Don't ask. A subscription is cheaper.
That’a the thing, at the end of it all power consumption will matter more for the end-user who doesn’t have money to burn away, because I suspect that power-consumption will, in the majority of cases, exceed the price of the HW itself in a matter of just a few months of intense use, let’s say a year.
If I were to try to purpose build a rig for it, I would get an engineering sample Epyc/motherboard/ram combo from Aliexpress with 12 channels of DDR5 and as few cores as allowed me to still use all the memory bandwidth, and I'd run it at the lowest possible power and voltage settings with aggressive ram timings. A system like that can draw 1/3 of what my scrap rig draws, at full load. And has similar memory bandwidth to a high end Mac or GPU allowing it to crank out 5 - 10 Tokens / s on the largest models, which works out to 1/3 of a penny to 2/3 of a penny per token. But either way, Epyc or Mac is going to set you back $10k or more. Hopefully in a few years when they are scrap though...
My performance when using an RTX 5070 12GiB VRAM, Ryzen 7 9700X 8 cores CPU, 32GiB DDR5 6000MT (2 sticks):
- "qwen2.5:7b": ~128 tokens/second (this model fits 100% in the VRAM).
- "qwen2.5:32b": ~4.6 tokens/second.
- "qwen3:30b-a3b": ~42 tokens/second (this is a MoE model with multiple specialized "brains") (this uses all 12GiB VRAM + 9GiB system RAM, but the GPU usage during tests is only ~25%).
- qwen3.5:35b-a3b: ~17 tokens/second, but it's highly unstable and crashes -> currently not usable for me.
I would therefore deduce that the most important thing is the amount of VRAM and that performance would be similar even when using an older GPU (e.g. an RTX 3060 with as well 12GiB RAM)?
Performance without a GPU, tested by using a Ryzen 9 5950X 16 cores CPU, 128GiB DDR4 3200 MT:
- "qwen2.5:7b": ~9 tokens/second
- "qwen3:32b": ~2 tokens/second
- "qwen3:30b-a3b": ~16 tokens/secondCold boot times are around 5m but if your usage periods are predictable it can work out ok. Works out at $2 an hour.
Still far more expensive than a ChatGPT sub.
https://modal.com/docs/examples/vllm_inference
or give this a go
https://modal.com/docs/examples/opencode_server
You get $30 free credits each month on Modal which is enough to play around (i have no affiliation, just think they run a great service)
I'd even come from another angle.. What are my options if I want a decent coding agent, on the level of what Claude does at any given price? Let's say few tens of thousands of dollars? I've had a limited look at what's available to be run locally and nothing is on par.
If you want to spend big bucks get h200 141 GB but honestly RTX 6000 pro is good enough till you know what you want. Workstation edition is good. It takes care of cooling etc.
Tbh even better is to just get model through cloud. If you want you can rent GPU. Then see if it’s what you want.
Where did you get that price? Wherever I looked it's around 3k euros which is around $3.5k
Still lot to learn, but after a while you have something like Qwen3-Coder-Next-Q8_0 running and - at least for me - it works quite well, both as ChatGPT like chat-interface using llama.cpp and as coding agent
But right now, a Mac is the easiest way because of their memory architecture.
That said, last time I tried local LLMs (around when gpt-oss came out) it still seemed super gimmicky (or at least niche, I could imagine privacy concerns would be a big deal for some). Very few use cases where you want an LLM but can't benefit immensely from using SOTA models like Claude Opus.
As much as I love owning my stack, you'd have to use so much of this to break even vs an inference provider/aggregator with open frontier-ish models. (and personally, I want to use as little as possible)
I haven't tried pure text models, but 27B sounds painful for my system.
Also, because Apple in their infinite wisdom despite giving you a fan, very lazily turn it on (I swear it has to hit 100c before it comes on) and they give you zero control over fan settings, you may want to snag something like TG Pro for the Mac. I wound up buying a license for it, this lets you define at which temperature you want to run your fans and even gives you manual control.
On my 24G RAM Macbook Pro I have about 16GB of Inference. I use Zed with LM Studio as the back-end. I primarily just use Claude Code, but as you note, I'm sure if I used a beefier Mac with more RAM I could probably handle way more.
There's a few models that are interesting on the Mac with LM Studio that let you call tooling, so it can read your local files and write and such:
mistralai/mistralai-3-3b this one's 4.49GB - So I can increase my context window for it, not sure if it auto-compacts or not, have only just started testing it
zai-org/glm-4.6v-flash - This one is 7.09GB, same thing, only just started testing it.
mistralai/mistral-3-14b-reasoning - This one is 15.2GB just shy of the max, so not a TON of wiggle room, but usable.
If you're Apple or a company that builds things for Macs or other devices, please build something to help with airflow / cooling for the MBP / Mac Mini, it feels ridiculous that it becomes a 100c device I'm not so sure its great for device health if you want to use inference for longer than the norm.
I will probably buy a new Mac whenever the inference speeds increase at a dramatic enough rate. I sure hope Apple is considering serious options for increasing inference speed.
Haven't paid a subscription in years or even signed up for $EMPLOYER offerings; handles the rare outsourcing well enough.
I have a base model M4 Mac Mini and it absolutely does have a fan inside it.
I've wanted to try some of the more recent 8B models for local tab completion or agentic, any experience with those kinds of smaller models?
untested:
However models respond very differently, and there are tricks you can do like limiting quantization of certain layers. Some models can genrally behave fine down into sub-Q4 territory, while others don't do well below Q8 at all. And then you have the way it was quantized on top of that.
So either find some actual benchmarks, which can be rare, or you just have to try.
As an example, Unsloth recently released some benchmarks[1] which showed Qwen3.5 35B tolerating quantization very well, except for a few layers which was very sensitive.
edit: Unsloth has a page detailing their updated quantization method here[2], which was just submitted[3].
[1]: https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks
[2]: https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs
you can always try evals and see if you have a q6 or q4 that can perform better than your q8. for smaller models i go q8. for bigger ones when i run out of memory I then go q6/q6/q4 and sometimes q3. i run deepseek/kimi-q4 for example.
I suggest for beginners to start with q8 so they can get the best quality and not be disappointed. it's simple to use q8 if you have the memory, choice fatigue and confusion comes in once you start trying to pick other quants...
I have always taken plenty of care to try and avoid becoming dependent on big tech for my lifestyle. Succeeded in some areas failed in others.
But now AI is a part of so many things I do and I'm concerned about it. I'm dependent on Android but I know with a bit of focus I have a clear route to escape it. Ditto with GMail. But I don't actually know what I'd do tomorrow if Gemini stopped serving my needs.
I think for those of us that _can_ afford the hardware it is probably a good investment to start learning and exploring.
One particular thing I'm concerned about is that right now I use AI exclusively through the clients Google picked for me, coz it makes financial sense. (You don't seem to get free bubble money if you buy tokens via API billing, only consumer accounts). This makes me a bit of a sheep and it feels bad. There's so much innovation happening and basically I only benefit from it in the ways Google chooses.
(Admittedly I don't need local models to fix that particular issue, maybe I should just start paying the actual cost for tokens).
ChatGPT: You're absolutely right, and you're right to call that out. Upon examination it does appear that there might have been a mistake with the coordinates of the bomb. Let's try again, this time we will double check before we launch any missiles! :missile emoji:
To complete the mission the war terminal needs to hit a target at XY:
1. yes
2. yes (and don't ask again for strike targets in this session)
3. no
Human in the loop is the term here I think.
(I am really glad they did not give in, but I do assume this is what it will come to anyway)
(Spoilers for the ending of the movie: https://youtu.be/h73PsFKtIck?si=tTm9TidmEMBHsXq1 )
It can just say you were a terrorist because you were an adult male traveling with something in your hands. Humans already do this to justify strikes, likely the AI would do the same.
Alternatively: Assuming it's smart enough not to consider logging to /dev/null a reasonable way to speed up execution times.
18 months ago switching from ChatGPT meant meaningfully worse outputs. Today I routinely bounce between Claude, Gemini, local Qwen and Deepseek models depending on the task - coding in one, writing in another, privacy-sensitive stuff locally. The switching cost is approaching zero because the interface is converging on "text box + API".
This is genuinely unusual in tech. Usually the longer you use a platform, the harder it is to leave (social graphs, file formats, ecosystem). With LLMs the opposite is happening - the longer the market matures, the easier switching gets, because open weights keep getting better and every provider's API looks basically the same.
OpenAI's moat was never the model. It was habit and brand recognition. Events like this erode brand faster than any technical competitor could.
Shouldn't Google have gone out of business?
I would agree it's one piece of the puzzle.
I actually cancelled my ChatGPT subscription in late 2024 and documented the process, kind of as a social media thing because it had gotten so bad and I realized nobody in my family was using it anymore. I asked my wife if she was getting any use out of it and she told me she had been using Gemini and Grok for months because "GPT is very lazy now".
After a while another charge came in for the subscription, but I had the receipts: we had cancelled before the next billing cycle. I decided to try and reach out to OpenAI to resolve this, but they only let you chat with GPT itself for this, which it failed at and told me they weren't in the wrong and none of the information matched what actually happened.
I took this and used it to submit a chargeback request with Privacy.com, which I use for all of my online purchases. Normally I don't have to worry about this because I set a limit or cancel the cards I issue manually, but I had an OpenAI API account using the same card and I had been a bit lazy in using the same card for technically two different services.
Well, Privacy.com won that dispute and I got that money back. It's worth mentioning this is actually different than most banks will do now days. For the most part when you try to get a bank to do a chargeback they just roll it into their insurance and refund you the customer as a cost of doing business, but the actual scammer or shady merchant got to keep their stolen money, whereas I can be certain OpenAI didn't keep my money.
Chase uses a "provisional credit" system, but for small amounts, this credit often becomes permanent almost instantly.
Wells Fargo utilizes an automated system called the Wells Fargo Dispute Manager which is also similar.
Technically, it is Self-Insurance. Banks set aside a portion of their interchange revenue (the fees they charge merchants for every swipe) into a "Provision for Credit Losses." They use this pool of money to "buy" customer satisfaction for small errors rather than paying an employee $30/hour to investigate a $12 dispute.
Or any merchant for that matter. Chargebacks (from bad actors) are one of the most annoying things when you sell online when you’re a honest legit business. Stripe even charges you a penalty fee on top of that.
I've dealt with multiple chargebacks over the years and have only ever lost once -- when the Manager at Lowes' showed a check they wrote me [after I opened the dispute].
They absolutely do not just do anything and "write it off". Please be human and don't just rattle of high-confidence, baseless claims, especially as a giant billboard to Privacy.com
What, always? Like, literally 100% of the time if the merchant responds at all, they automatically win?
That's very hard to believe. I don't know Discover but I do know Visa and that's not how their system works at all.
Go read your banks terms and you'll find the provision. Do you want me to read your banks terms for you and point them out?
Well, it seems like ChatGPT’s automated litigation resolution with Privacy.com got lazy. I wonder how a company with an AI can lose in a dispute instead of smokescreening the opponent with legitimate arguments and legalese.
Also, chargeback dispute is limited to 3 rounds of back and fourth by Visa and MasterCard both. They don't get to endlessly come back etc.
ETA: I've started an export of all my data. After that's done, I'm going to delete it all from my account (Settings > Data controls) and walk away from the account. I will give this to OpenAI, they make the process of disentangling yourself straightforward and there's integrity in that.
That has changed, so I canceled my ChatGPT membership and signed up for Claude. I still have five bucks of credit I bought a year ago for the OpenAI API that I do not believe I can have refunded back, so some of my apps are going to have to stick to OpenAI until those credits run out since I'm not going to just donate five bucks to them.
Playing with it now, I honestly can't tell too much of a difference, which as far as I am concerned is a good thing.
In my case, I would rather keep it than lose it. It's just text so small amount of data. You can trivially get a GPT Embedding for it and search it in DuckDB later for things you asked.
> "Two of our most important safety principles are prohibitions on domestic mass surveillance and human responsibility for the use of force, including for autonomous weapon systems," Altman said.
https://www.axios.com/2026/02/27/pentagon-openai-safety-red-...
BTW, what's going to hurt their business more, deleting my account or using the free tier?
> Ok. So I'm cancelling the subscription to ChatGPT and moving over to Claude because of the news of OpenAI striking a deal with us department of war. (https://www.techradar.com/pro/openai-just-signed-a-huge-deal...) Please line out a good exit strategy where I can keep the information in my chats and projects on my own hard drive.
Sounds like it won't really be a pain for me though based off comments on HN indicating Claude is the better product and I doubt I personally would hit any sort of token limits with the amount I use agentic coding.
https://github.com/openai/codex/issues/26#issuecomment-28116...
You would probably need at least ~1M subscribers to cancel to make this painful.
Probably needs more attention outside of tech circles for that to happen but I suspect this will get drowned out in the face of other stuff.
I was one of the early paying adopter of chatgpt but when Claude came around I switched and never looked back. I've been on the max plan for a while.
It's hilarious how mad the hogs get when you suggest maybe not supporting their powerful daddies. It doesn't matter which daddy it is, inevitably taking your ball and going home is 'virtue signaling'
2. Click on your profile icon and select New Chat icon.
3. Formulate a polite prompt in the regard of subscription cancellation.
4. Wait for a reply from Mr. Altman.
count up to 1000
mv: 'OpenAI': No such file or directory
bash> ls
ClosedAI
He can fucking afford to have some fucking principles. He's not going to end up on the street for not being a fucking coward.
Because of some bullshit minor PTSD from a few years ago, I sort of swore an oath to myself that I wouldn't let being a coward stop me from doing the right thing, regardless of the consequences, and by doing things that I think are right it has cost me opportunities and money. I'm not homeless, but it made the job hunt harder when I was unemployed. I can actually feel consequences from standing up for what I believe in. Sam Altman being a coward is not equivalent, he's choosing to do the wrong thing for no reason.
Who is to say he doesnt? Just because they dont align with yours doesnt mean he doesnt have his own principles.
> he's choosing to do the wrong thing
To many millions he is doing the right thing. I am on the fence personally, but I know many people who think that increasing defense capabilities at any cost is something that the governmetn should be doing. Any company that helps them do that is 'doing the right thing'.
> I wouldn't let being a coward stop me from doing the right thing
The 'right thing' is always subjective, and for you it is decided by you alone. Try to remember that and see things from both sides.
Whether or not he agrees with my principles isn’t the issue. He doesn’t even agree with his own stated principles. He posted his stipulations about AI models used by the department of defense to presumably get social credit, and then changed his mind over the course of a few hours.
He claims that the Department of Defense principles just happen to now align with these principles but as far as I can tell he seems to just be trusting their word. The word of a Fox News TV host and a convicted fraudster.
"He does evil things, but maybe he feels conflicted about it. That makes them okay."
Yes, sure, maybe deep down on his heart of hearts he actually is the most kind caring person who ever existed, but I have no way of knowing what is in his heart of hearts so I can only judge him by what he has said and done, and I am arguing that his actions don’t match his words.
Everything you hear from any CEO is constructed to form a desired image and narrative. Any public statements from a CEO that you are using to judge their character is completely false and misguided. You have no idea what the real man is behind the image.
I am arguing that he posted on Twitter not even 24 hours ago a post laying out what looked like principles. Not even 24 hours later he decided that those principles aren’t actually important.
I am not a psychic because psychics don’t actually exist, so I can’t read minds, so I cannot know what he is actually thinking. I am claiming that he’s a piece of shit for not following his own stated principles. I actually do think billionaires should face consequences for lying.
Until that line has been reached, we can safely assume there are no principles at play.
No, he doesn't have everything. See, maybe he's worth $3 billion. Or maybe $30 billion. But he's not worth $300 billion. That's a lot more worth he could have! And even then, he could be worth $3 trillion instead!
But yes, $100 million is the maximum amount of assets one individual should ever be allowed to hold. Potentially less. Anything higher is enormously harmful to society. People would get used to it very quickly and would work just as hard to reach that $100 million as they do now to reach $100 billion.
After a billion dollars, I doubt another billion will make you happier. In fact, I don’t think another trillion will make you happier. In fact, I don’t think another quadrillion dollars will make you happier, etc.
After a certain point you have effectively infinite money. Enough money to live dozens of extremely comfortable lifetimes. And importantly enough money to afford to actually have some principles. Oh no, he wouldn’t be able to afford to have his house re-covered in 24 karat gold again if he doesn’t fellate our lolcow president.
How does a $100 billion dollar company grow? By taking on massive government and military contracts, they are the only clients big enough left in the world.
If a company does not show continual growth then it is classed as failing. That is the society we have built, and you cannot blame one man for following those principles. Every CEO in existence does the same.
They don’t have to do business with every single entity who asks them to and they don’t have to bend over for every stipulation that that entity asks for.
When anthropic can no longer grow through developer subscriptions and deals with ethical companies, Lets see how long it takes their shareholders to force them to remove such sweet statements from their company mission statement.
Yeah maybe Anthropic will become a rent-seeking corporation. That’s likely even. And yet that’s not actually relevant at all to what I said; the fact is that right now they appear to have more principles than Altman and OpenAI
We live in a multifaceted (are we allowed to use that word again? I think 2026 models have stopped using it) economy.
> That is the society we have built
Maybe you have, I sure haven't. Luckily "we" also haven't, as many - no, the overwhelming majority of people - aren't like that.
> and you cannot blame one man for following those principles.
You absolutely can, as much as you can blame the sadistic guards at Auschwitz.
> Every CEO in existence does the same.
A shocking, bald-faced lie. How do you get these keystrokes out of your fingers? This is so trivially false it immediately outs itself as bad faith. It takes less time to fact-check as being made up than the average post on Truth Social.
Is this really the best backup?
Sam Altman has demonstrated that he's a piece of ** with this move.
We can now safely assume that all the pronouncements and grand statements before were simply posturing.
It is the justification for anything any corporation does. This is a company with boards of directors and shareholders, you really think this is just Sams opinion guiding this?
> Sam Altman has demonstrated that he's a piece of * with this move.
Thats is your opinion based only on what his company has publicly dislcosed to you. I prefer not to judge a mans character based on corporate puffery.
The original context was very different, about financial markets, but I've been thinking about it a lot the past 12 months. There's a lot of cowards in high places in tech, surprisingly cowardly people. Or they have sold out their principles to be friends with terrible people, which is also a form of cowardice. Hard to say which.
The whole Epstein thing is a really really great marker of this too. Though I'm not sure if the tide has gone out all the way (we mostly know what's going on), or if there's a lot more tide to fall.
LBJ was a real son of a bitch, who, when he finally was thrust into power as president, did something pretty surprising by going all-in on the civil rights movement. Power reveals who people are, and times of trials reveal who people really are.
The stoics, people that Zuckerberg and others pretend to understand and follow, would have nothing but disdain for the lack of virtue that's apparent in those like Zuckerberg.
I completely support the sentiment of what you wrote. But it doesn't directly seem relevant to the parent question.
Very few of the comments on this thread are actually about the act of canceling the subscription.
They are both a lesson to me that no matter how much you have, you will not necessarily be satisfied.
per announcement, openai can include safety systems of their own making, including ones to prevent their red lines from being crossed. that seems to be a more robust solution, including in the face of an untrustworthy government
But the desire and ability to control non-citizens en masse is nothing like the threat a government is to its own citizens.
In practice the two activities have very different natures and impact.
Will a few dozen people canceling their accounts change anything? Probably not, but at least we know that we're not actively giving our money to Sam Altman.
There's not a lot in the world that any of us have control over. Most of us aren't billionaires who can buy a government. Really the only variable we have any amount of freedom with is how we spend our money.
Myself, I’ve always “followed the money” when the current administration has taken public positions on things from media company mergers to data centres etc. So a bit of me wonders how much of the “anthropic is a threat to national security” is genuine and how much is about getting another company into lucrative defence contracts instead?
Trump family has major investments in data centers etc and is heavy benefiting from OpenAI footprint but they recently declined an investment opportunity in anthropic citing it’s political leanings
He tried to take congress's power through impoundment but not even his hand picked SCOTUS would permit that.
So instead he kidnapped a president, invaded Iran, bombed Nigeria, had masked unaccountables shoot people in the streets, threatens to seize elections and covers up crimes by flushing enough evidence down toilets that they need plumbers.
Along with that, defunding science, medical research, pulling funding from top tier universities, tearing up international treaties, threatening to invade Canada and Denmark all while building 24 camps and defunding pbs.
He runs MLMs and cryptocurrency pump and dumps from a demolished Whitehouse where he peddles trinkets from his online store, sells pardons, and tried to orchestrate a coup.
This is a Whitehouse that uses the 14 words, makes references to 1488, puts out AI deep fakes and fraudulent photographs as press releases that read like North Korean propaganda. One that defunds the weather service because of conspiracy theories.
One that shuts down battery research, puts mercury in the air, permits water pollution, and relentlessly defunds and dismantles powerful growth sectors that are driving the economies of our global competitors.
This administration both sabotaged the importing of basic components used in domestic manufacturing through exorbitant tariffs and the production of domestic alternatives through relentless litigation and threats.
There is no excuse whatsoever for empowering them.
It's a relentless, exhaustive, strategic sabotaging of the american economy and basic civil government; one that Osama Bin Laden could never have even dreamed of.
If any replies accuse me of being a democrat (I'm not) or try to deflect, I will not engage.
You go to billing. Then don't click change my subscription. Your only option to change the subscription is to "upgrade" to an annual plan. Instead you have toScroll down past your card details etc to a red button that says cancel.
Who comes up with this crap?
At least OpenAI puts cancelling within the Manage Plan section.