Running Qwen3.8 at 255 tok/s on a single 5090(meshapplied.com)
11 pointsby Ir0n4 hours ago4 comments
  • jkingsman4 hours ago
    I struggle with approaching this as I do with many quants, in that there's not really a good measurement of how lobotomized by quantization they are other than "a bit but it's not too bad". Ternary weights immediately make me think "unusably quantized" but I'm presuming that's not the case if someone's standing by this release without massive caveats.

    I struggle with deciding whether to take the speed hit on q8 vs q6; I wish it was more standard to benchmark quants or have some way to indicate how much intelligence you're leaving on the table.

    I don't mind waiting twice as long for twice as smart; I don't want to be passing up the opportunity of 4x speed-up with 98% intelligence retained. But I have no way to differentiate which is which with quants in any vaguely objective way.

    Frustrating, as a hobbyist.

    • MozerJ4 hours ago
      That is true, the lack of good benchmarks will be with us for a while I think. Ternary is not a good quant algo, but because it's so drastic, it allows people to use enormous models on worse hardware, or just increase throughput speed. In the meanwhile, there's research going into how this quantisation can be executed while correcting for the loss. In this case we're not just quantising but also post training the model, and we can "trade" some capabilities to others (ex. post-train the experts for math but lose on biology or whatever).

      With that said, I find nvfp4 to be far superior. I think right now all you can do is try out the models, maybe run your own benchmarks if you have.

    • Ir0n4 hours ago
      we achieved this on stock nvfp4 model which is within 5% of fp16 version

      we then apply lossless techniques only

      dflash2 as spec drafter and patched ninfer inference engine (down projections for short token lengths and adding tool call related schema to drafter shortlist)

      see: https://github.com/meshapplied/goethe/pull/4

      benchmark numbers are measured using see bench verified traces at avg input 28k and concurrency 1

  • 4 hours ago
    undefined
  • ranger_danger4 hours ago
    How would this compare speedwise to Bonsai 2 on the same GPU?

    https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf

    • MozerJ4 hours ago
      It's faster and more accurate. Bonsai was a way to demonstrate how effective ternary quant is, and how much it accelerates throughput. If you try coding with it you'll find that it often makes mistakes with tool calls and other syntax related tasks.

      This model has higher retention because of the weight complexity and the quant method we built, and the speed loss is compensated via kernel optimisations and speculative decoding.

      • ranger_danger2 hours ago
        Have you actually observed these issues specifically with the newer Bonsai 2 (and not the original Bonsai)?
  • BizarroLand4 hours ago
    The title doesn't apply to the link.

    The website is not as good as it needs to be.

    There's no information being shared other than these models exist and I don't want to drill down through 3+ links trying to find what you're saying about them.

    Who would run a random script on a random page with no other information given?

    I'm not saying this is bad, but they definitely need to explain what the heck is going on to the point where a 15 year old can understand it, and provide logical and clear breadcrumbs to navigate through their pages.

    Even a FAQ would help.

    • matteohorvath4 hours ago
      thanks for the feedback, solved the linking, and will provide more info on the webpage asap.