Launch HN: General Instinct (YC P26) – Frontier models on edge devices
18 pointsby guanming07172 hours ago5 comments
  • rdksu2 minutes ago
    Have you run ablations on the actual effect/impact of on-policy distillation on contributing to the performance ? Just Curious ! As Unsloth based mixed quantisation methods on MoE models are widely used with great community rep.
  • BoorishBears16 minutes ago
    I like the technique described here around distillation to recover from quantization, but I don't understand why we keep performing lossy compression on LLMs then using benchmarks that were nearly saturated before post-training to measure the effects.

    You could erase the gains from literally half the compute going into some of these recent models and barely make a dent in MMLU-Pro and GPQA-D.

  • XenophileJKOan hour ago
    I'm still kind of surprised that people are targeting edge deployment of MoE models. By definition they optimize for computation cost at the expense of memory efficiency. We generally need the opposite on the edge.

    I'm hoping to see more work in the other direction with cyclic/looped transformers and other memory dense approaches.

  • VikRubenfeld2 hours ago
    You've likely heard about this - he'd probably like to talk to you and might potentially give you some good PR.

    https://www.youtube.com/watch?v=rAzT5lcezPs&t=467s

    • smokelan hour ago
      For those too lazy to watch someone talk on video for ages to make a point:

      The link is to a famous YouTuber called PewDiePie and he uses a local LLM to parse his email, to save time with that. They have an autoreply system and get notified about urgent matters.

    • guanming07172 hours ago
      Thanks for sharing! I'd love to chat with him. Would you be open to introducing us? :)
  • rohansood15an hour ago
    Have you benchmarked against other 3-bit dynamic quants like Unsloth? I am sorry but this framing against a full precision, newer, smaller MoE just seems misleading. Also, Gemma-4-26B-A4B is not the SOTA for edge. Even at launch, that would be the 31B.
    • guanming0717an hour ago
      Yes I did, with other SOTA quant methods like HQQ, AWQ etc. You can find more info in our blog :) https://general-instinct.com/blog/frontier-moe-sub-4-bit
      • rohansood1532 minutes ago
        I can't find it. Can you state your performance versus comparable 3-bit quantization from Unsloth/Bartowski? Edit: I appreciate that you seem to have open-sourced the quantization pipeline. This is not to question your work, but to understand where the outputs stand relative to the SoTA for quantization.