Qwen3.8-27B on the iPhone 18 Neural Engine at 15–70 tok/s (1.5 bpw)(twitter.com)

28 pointsby anemll3 hours ago5 comments

Retr0id2 hours ago
OP, asking for upvotes is against the HN guidelines: https://x.com/anemll/status/2111531629529456771
createful2 hours ago
Not Qwen, Bonsai. Although it is based on Qwen, and achieves similar scores, it is not the "full" Qwen.
Last I remember, when trying to run a model above 1-2 GB on iOS, it would refuse to do so, at least from Locally AI. iOS only allows apps to use about 1-2 GB of RAM before being killed. Though using their Neural APIs might bypass this.
analog_daddy2 hours ago
Hi. Could anyone share any recommendations (hopefully with actual links) for App Store apps to try running such local models? I’m not really interested in side-loading since it currently seems (to me) more of an hassle than it’s worth.
App Store search is broken in a sense that there might be so many apps claiming to be local but once you read reviews, it’s just another scam where it’s it’s usually sending your query to remote server.
- simonw2 hours ago
  My current favorite is this one: https://apps.apple.com/us/app/locally-ai-by-lm-studio/id6741...
  It's free, not annoying, and has a bunch of model options.
  - layer82 hours ago
    Non-app-store link: https://locallyai.app/
    However this uses the GPU, not the Neural Engine like the submission.
  - jdgoesmarching2 hours ago
    Note that the model discussed in this article isn’t available in this app running on the phone. Still, it has some good local options like Gemma 4 and can link to the desktop LM Studio app which is cool.
- woadwarrior012 hours ago
  I'm the dev behind the Private LLM app, which was the first on-device LLM app on the App Store. Don't download it yet. I'm currently finalizing a brand new v2 rewrite which can run models that the copycat app (locally) cannot run, like Bonsai 2 27B, Gemma 4 12B etc.
  https://x.com/private_llm/status/2101769928252473488
spacecoder23 hours ago
Anyone know how hot an iPhone gets running the Neural Engine flat out for 10 minutes?
- GeekyBear2 hours ago
  The neural engine is optimized for energy efficiency, and the GPU is optimized for performance.
  You're going to get much better battery life and much less heat running on the neural engine.
- anemll3 hours ago
  It's warm but not hot. On M6 it's about 50% less energy consumption in my tests vs GPU. I think iPhone should be similar
anemll3 hours ago
[dead]