but i maintain https://github.com/Arthur-Ficial/apfel so i might be biased
Here's what you get when you run it... https://gist.github.com/robgough/7893602895e7580117475076198...
Meet Core AI - https://developer.apple.com/videos/play/wwdc2026/324/
Dive into Core AI model authoring and optimization - https://developer.apple.com/videos/play/wwdc2026/325/
Integrate on-device AI models into your app using Core AI - https://developer.apple.com/videos/play/wwdc2026/326/
Does this completely replace the previous API, CoreML? [1]
[0]: https://apple.github.io/coreai-optimization/
[1]: https://developer.apple.com/documentation/coreml/"If your app uses model types other than neural networks, such as decision trees or tabular feature engineering, see Core ML."
- Core ML is for models designed only for Apple platforms
- MLX is for models that don't need to be fast
- Core AI is for models that run everywhere already and also need to be fast
I've been on claude's opus 4.5/6/7 for work for a couple months, and I finally got back to running Qwen A3B 35B... it's incredibly performant and quite capable on semi-reasonable local hardware.
I get ~150 tokens/s on dual nvidia RTX 3090s and can fit the whole 300k context into gpu on a UD-Q4-K-XL quant gguf.
Combined with Pi as a harness, and I'm surprised to find that it feels about as capable as claude did 8 months ago (their 3.x models).
It's not Opus 4.5 levels yet, but it's good enough for a LOT of basic work. I actually downgraded my personal anthropic subscription because Qwen is absolutely fine for implementation work. I still let a better model write a plan, but then I can just switch over to Qwen to implement.
I don't think we're 10 years away from opus 4.5 levels running on cheap consumer hardware. I think we're probably closer to 18 months away, and I suspect it'll be in the 30-60b range, not the 256b range.
PC manufacturers also seem to be betting on local, with a LOT of focus on 64 to 128gb unified RAM machines.
Local is a pipe dream . If you can run it cheap occasionally why commercial companies can’t run it cheaper 24/7 and lower the costs ? The answer is simple. Use cases are more demanding and hence you need more from model not less .
Sure if you task is to do a narrow labeling task on 1m records small optimized model is good . If you want to do complex things , it shifts with models advancements
I'm seeing some impressive results from folks that can afford 10k+ GPUs right now. But those GPUs will all be hand me downs in 10 years. So pipe dream? Hmmm...... that's not how this industry works.
Qwen 3.5 was released 3/2/2026. It includes models up to a 397B-A17B model
https://huggingface.co/collections/Qwen/qwen35
A day afterwards, a high-up technical leader working on Qwen was let go
https://techcrunch.com/2026/03/03/alibabas-qwen-tech-lead-st...
The more recent Qwen 3.6 was released on 4/16
https://huggingface.co/collections/Qwen/qwen36
This does not include any particularly large models. But the models it contains (Qwen3.6 27B and Qwen3.6 35B-A3B) are the local models people have been very excited about lately. So they didn't release any larger models, and the models people praise so much are from this most recent release.
I think what people didn't realize was, just because the GPT-4.5 model didn't get better on the benchmarks, didn't mean the model wasn't different than the earlier models. It was being compared to thinking models that were being developed at the same time.
The GPT 4.5 model still has some of the most "human" like abilities in communication even though it isn't particularly good a problem solving. It hadn't under gone the same type of reinforcement training.
I still use GPT 4.5 sometimes, in creative exercises it can be surprisingly effective. The model is still available.
I use small models exclusively. They aren't a replacement for large models. You need decent hardware to run those models efficiently, as smaller parameter models plain suck and are still slow on macbooks. And affordability of higher end hardware is very limited.
Even at non VC subsidized $/token prices, its still much cheaper to run cloud based models.
On a price-per-wattage level, this is not true, people have done the math on /r/LocalLLaMA many times over[1]. Local models, while not as good as premier models (GPT 5.5, etc.), are like ~80%+ of the way there, and often converge to a similar solution after a few dead ends.
[1] https://www.reddit.com/r/LocalLLM/comments/1kshq4f/electrici...
Might be time to sell, to be honest. It's fun to have that at home, but I can't justify having $10k (with memory, mobo, cpu, etc) sitting in my basement without being fully utilized.
Where do these improvement curves go? Does the gap close, do they intersect for practical purposes (factoring in cost etc)? Or is the local curve always just a translation of the hosted, lagging behind, or indeed does hosted just pull ahead?
Nobody knows, but it's a very open question I feel, and it certainly appears like the answer might quite reasonably be that yes they intersect on that kind of short-ish term time horizon.
Nowhere.
Large models haven't seen that much improvement, just small unique tasks performance which is all special cased RLed to game metrics
For local models, its the same story. You can download Gemma 3 QAT from last year, and it will be just as good as Gemma:31b on the average. Qwen also boasts that its better, because again, they RLed it to game some metrics. Its better in coding then Gemma, but Gemma is better in more creative thinking (again, all RL)
Fundamentally, you need detail in the gradients for the models to pick up on the smaller details. If you don't have those, your output is gonna suck. No amount of clever architecture is going to fix this.
The only way to improve local models by training them to fetch context, and then their job becomes much simpler because all they need to do is reinterpret the fetched content and provide an answer. But fundamentally, if you are trying to keep things in house for advertising purposes like what all companies do with search, you want them to go to your service, which means running on your servers. And its not really that much extra per invocation (i.e excluding initial hardware costs) to instead just offer a large model as a service, which will be way better than any small models.
I expect I'll probably keep paying for whatever badass high IQ model is running on inference servers at that point
I haven't seen any sign that the framework fragmentation problem is going away anytime soon. NVIDIA wants everyone to do all training and inference with CUDA and to deny that NPUs have any usefulness. Everybody making an NPU has a different framework tailored to their architecture and the limitations they inherited from hardware designed before LLMs existed, and most of them have a another framework for targeting a GPU. And the OS vendor has one or two frameworks they would prefer you use rather than something hardware-specific.