I have an older M2 Mac mini that does the OCR and visual description of all my screenshots. Screenshots are stored on my NAS.

I like to screenshot things as a quick way to remember. They are things that I would not be comfortable sending a cloud provider (customer data, prototype screenshots, bank dispute details).

It runs Qwen3.5:9b and glm5.2-ocr with Ollama and uses about 10GB of RAM. It automatically releases the models from RAM after 5 minutes of inactivity so it is pretty seamless to leave running in the background.

All the details are stored in a simple webapp with a SQLite db that I can search through.

The $20/month subs are much stronger than the local models you can run, even with how far local models have advanced lately.

The appeal of local models is that the data never leaves your network so you can feel safer putting sensitive content into it. It also feels “free” to use when you’ve already paid for the hardware.

But it doesn’t perform better and if you do the math you’re probably not saving money either. It’s helpful for things that you can’t or don’t want to outsource to a 3rd party.

We also built some QA agents that are always playing our games from the same builds a player would and flagging things to fix/improve; that alone needs the game focused and front-and-center so it can properly screen-capture for deciding what inputs to take next (and for screenshots/replays), which also means we can't really do any hands-on work at all on the machine when it's running.

Having a separate (and tiny) machine for all of this has been great. We don't bother with local models because, you're right, the $20/month sub is way better than anything that can run on small consumer hardware atm.

I recently built a minimal Dark Software Factory out of an N150 Mini PC. It uses three models; Sonnit, Sol, and Gemma.

But, I have a LOT of instructions about how I prefer the software it builds. Gemma doesn’t handle all my instructions very well. But it’s close!

I’m running gemma-4-12b because I have limited RAM and larger models were too slow.

I do two types of jobs: planning and prototyping. It has done fine at some of my planning rounds.

I still consider it experimental and don’t use it a lot but I think we’re getting there.

I realistically costs $5-10k to replicate a ChatGPT like agent. And it doesn't scale.

That's still really close. And models and quantization etc keep improving.

I'm absolutely positive that I'll be switching to mostly local AI in the next 5 years.

I just ordered a new Mac Studio M5 Max 128GB $5899 ($6400 with tax) to be able to run the bigger "consumer size" models in the 70B parameter range (~96 GB). That said, I have no illusions that this expensive setup with a Qwen Flash coding LLM will be comparable to a $20/month subscription. Even upgrading to an even more expensive Mac Ultra 256GB for $10000 to hold a bigger model still won't be comparable. Apple hasn't shipped my Mac yet and I'm still considering cancelling it and downgrading to a smaller 64GB RAM config ($4299) to save $1600.

Why did I initially spend the extra $1600 if I knew ahead of time that it wasn't as good as cloud AI? Because I thought I could use some local LLM for the easy tasks or when I hit cloud rate limits. No issues with privacy so that wasn't part of the motivation at all. I just wanted some local AI capability to augment a subscription. I've not totally convinced myself of the cost/benefit of this.

Based on today's consumer hardware landscape, you're paying very high prices for crippled capability compared to the cloud AI subscriptions. We're also in a transition period where the next iteration of hardware improvements have some compelling features for local AI. Apple's upcoming M7 (2027 or 2028) is anticipated to have better GPU and neural engine to help with prefill TTFT. AMD Strix Halo is about to release 192GB system which is a big upgrade to their current 128GB ai pc. Maybe apply my $1600 savings towards those newer products. Those future products will still be very expensive but maybe the cost/benefit will be better.

I've spent $2 in the last 2 weeks on OpenRouter. I've been trying to only use the medium sized models that I would otherwise be able to run on a nice local setup. That nice local setup would cost ~$4k. I don't know what the operating cost would be, but I would be concerned that my home electricity would cost more than at a datacenter. It just doesn't make sense right now except for privacy reasons.

I'm probably going to hoarde open weights models in the ~31B range until memory costs fall in a few years. Then, I'll buy some hardware to run at home just so I feel more sovereign over my stack regardless the cost/token speed.

I have been eyeing a 512 GB Mac 5 Ultra to run full DS4 pro locally, which I expect would be pretty amazing as far as quality/recall. The only downside is that the speed is a lot slower than something like 27B on the 5090.

What they are doing that you might not be is context caching, retaining a user profile about you in vector db, and adhering to strict output templates - often invoking multiple different prompts to assemble the end response to the user.

You’ll notice the mainstream ones always kinda answer in the same way. They are also able to mix code and images with text (multi-modality) because they are breaking down and routing your prompt to different models then assembling a result.

That’s all just software, has nothing to do with the model.

More people imo should be experimenting with it rather than chasing better models. This area is where all the differentiation is happening in the services, and what gives an AI provider like Gemini or Vidu or Singify an edge over running the equivalent locally

Though keep in mind not being beholden to shenanigans from said cloud companies (and interference from government entities!) is definitely worth something intangible.

Local llms don't suffer from cloud availability issues. Anyone that used Google models know that sometimes they just don't have capacity whatsoever, at least that was the state of things some months back when I used them. Just bear in mind if needed, cloud providers will prioritise API and corporate customers over subscriptions if availability degrades more.

Also they don't have the same guardrails as the other models, so for hacking, reverse engineering and black coding (piracy etc...) these local models might be the only options.

But currently it's really hard to beat anything offered by the cloud companies. And the cost and complexity of setting it all up, just to barely (if at all) touch on Opus-level intelligence makes it seem like we're not quite there for the common man (enthusiasts are a different story.)

I am very excited for open source local models, and we're nearly there, but it's still too complex and expensive to be my daily driver (yet).

It's still frustrating as hell to come down in the morning, having given it a list of tasks to do overnight, with tests to pass before they're "done" and find that it worked for about 20 minutes after I went to bed, and decided that it would stop at "3am" (it wasn't) and "not do significant work this at this late hour". Like WTF ? You're an LLM. You don't sleep.

Bloody training data full of humans demanding sleep. I tells ya...

That said I have an RTX 5090, not a Mac Mini, so it's not exactly the same level of performance... The latest open models run at 200 tpm at around 30B params.

What you'll learn pretty quickly from said engineering is that there's a lot more to a good LLM than just the weights themselves. You need a good search provider (also self-hostable, but sounds easier than it really is). You need (well, it's debatable) a memory system. You need a good system for up-to-date library references like a Context7 (also self-hostable but the options are surprisingly not that good). You need a good set of specialized subagents that can perform various tasks well -- for the sake of "doing things well" but also managing context efficiently.

When you've got all that, local models can be _extremely_ useful. But there's one other important thing and that's decent hardware, unfortunately. A lot of people try out local models using small consumer GPUs or Macs and are rightfully unimpressed with the performance. And if the performance doesn't get them, usually they have expectations that they'll perform at Claude levels out of the box. Getting in that neighborhood, like I said, definitely requires some work.

So yes, they are genuinely very useful, but they are not yet a full replacement unless you have more powerful hardware and or don't need more intelligent ai.

What's an RTX 9070? Do you mean the RX 9070 or RTX 5070?

My vibes were that Apple wound down the “actual work” side of their operations (including machines like Xserve), because Ives couldn’t handle the unsexiness and unpredictability of business requirements in hardware.

He was self-indulgent and only wanted to work on things that “vibed” with him, rather than what the customers needed. It’s easy to be creative when you get to do what you want to do, it’s hard when you have hard constraints.

Frankly even the entry price is a bit high - I remember buying one for my son a few years ago (M1 mini) and it was a few hundred; now we're up to $900 for the base model.

A year ago you could get an M4 Mac mini for $399 on sale and now the same one used goes for over $700. The general AI RAM/SSD spike is part of that but there was also a huge demand spike for small, powerful, desktop machines that could be easily configured with these workflow tools.

https://www.canirun.ai (five months ago: https://news.ycombinator.com/item?id=47363754 377 comments)

64GB+ or dedicated 48GB (2x24 on GPUs) is IMHO absolute minimum.

32GB is not enough RAM. I don't even own a device with less than 36GB at this point, and that device I only have because my employer is being cheap. 64GB is a reasonable starting point for running local LLMs + normal tasks. 128GB let's you really run most smaller models like Qwen 27B and 35BA3B with good context. Even Qwen3.8-Flash-Next runs in 128GB with a 4-bit quant.

32GB would be limited to running models like Gemma4 12B and smaller dense Qwen versions like 9B unless you were using very small quants which damages quality of response.

What makes you think that? There's a lot of data centers that sell you access to colocated Mac Mini's, they have added FileVault unlock via SSH in the boot process which also makes things easier. There's not that many reasons to run a Mac in the cloud unless you have some very specific Mac related workload.

Nvidia has CUDA, AMD has CDNA, and Apple has... compute shaders, I guess?

Your letter will help feed the boiler that heats the building and is much appreciated.

[1] https://tech-insider.org/ca/steam-deck-price-increase-2026/

This unified architecture makes Apple hardware very good for AI work, where latency between RAM and CPU is very important

Even the SSDs (which are normally more strippable) use a proprietary hardware form factor.

I think the article mentioned the real draw:

> Apple noticeably promoted the ability to link multiple Mac Studios together into a single, more capable system for running large frontier AI models, a feature aimed at business and developer customers rather than everyday consumers.

(Couldn't read the source cited, might have more info: The Information)