Ask HN: Is it feasible to run a model on device for complete privacy?
3 pointsby mazinz4 hours ago2 comments
  • mc7alazoun3 hours ago
    Feasible but too expensive! I get that privacy is a priority for you but unfortunately if you want quality models you'd still have to maybe use frontier closed models..
    • mazinz3 hours ago
      No open source model that’s any good?
      • vitalyan12343 hours ago
        the Gemma you tried is tiny, there are 31B and 26B (A4B) variants. there's also Qwen 3.6 with 27B and 35B (A3B) variants, reportedly pretty good. try them on open router or something. these require 30-40 Gb of memory to run between RAM and VRAM, less if quantized beyond near-lossless 8 bit.

        there are near-SOTA open models, but they are 1T+ parameters, i.e. they require over a terabyte of memory to run.

        • 3 hours ago
          undefined
  • benoau4 hours ago
    It's technically feasible, really just a question of whether this is worth $10,000(s) to you and you're willing to spend it.
    • mazinz3 hours ago
      Why financially crippling? It’s free to run on device. The native Apple Intelligence works well for smaller context windows and text only.
      • benoau3 hours ago
        You can get poor results "for free" from your laptop, but the devices you need for the large models are very expensive.