Gemma-4-26B-A4B with 39-69% less thinking at same accuracy(bottlecapai.com)
76 pointsby armcat6 hours ago4 comments
  • arjiean hour ago
    It feels clear to me that smaller models are moving parameter-level thinking into inference time. i.e. they use more tokens to perform computation because they don't perform it in the model itself. There's some conservation of difficulty for tasks and you either do the computation in the model or you do it in the looped iterations of the model.

    This is why the massive tok/s you see on many small MoE never manifest as worthwhile outcomes. Sonnet 5.5 can be faster on tasks than DeepSeek 4 Flash Vision simply because it gets there faster.

  • PaulKeeble3 hours ago
    I have tried the Thinking cap and the Swift model for QWEN-3.8 27B and my experience was its not as smart and capable, the results on the same task were less impressive. For the amount of tokens saved it might very well be worth it but I wanted to run the xhigh precisely because I wanted the peak result. I haven't tested this but I suspect it ends up much the same trade off, in theory it does well in the benchmarks but in practice on your own tasks its not as capable.
    • dofm3 hours ago
      Yeah. I didn't find these any good either.

      However, Gemma 4 26B is a reasonably good general chat inquiry model (it's also not too bad at commonplace code questions, if not as good for agentic code).

      So this could be a good, fast model for that, in some situations; I may test it.

      I still broadly just prefer working with Muse Glimmer, to be honest; we get along OK. I am trying out the Nemotron Bolt 74B/A9B MoE and I really do think it could be quite good, but it's still slow on Metal.

      • solarkraft2 hours ago
        > I still broadly just prefer working with Muse Glimmer

        Welcome to our very exclusive club of now two people! Ever since it came out I’ve been trying to find a better model, but just can’t.

        • dofm2 hours ago
          It is hard to explain why it is better. Benchmarks etc., context length, it should be worse than Qwen 3.8 27B and I am sure for long context work it is.

          But on my old M1 Max, with "Reasoning strength: low" in the system prompt, it's just kind of nice to use. It's not really slow (despite being a dense model it is faster to actual solutions than the Qwen 3.6 35B MoE), it writes well, etc.

          So I use it as a developer-friend and researcher at the moment. Though it is actually the only model I've deployed code from! In general I am still writing my own (because I'm stuck in a bespoke framework on my main project) but it wrote me a plugin I needed for another app without any difficulty.

          It's the first local LLM I've considered building hardware for.

    • Foobar85682 hours ago
      But Qwen 3.8 27b is unbearable slow, you get 10minutes of "let me think" over any trivial stuff.
      • simonw2 hours ago
        Qwen 3.8 27B defaults to "xhigh" (its maximum level) for thinking, which makes it diabolically slow. Setting it to "medium" or "low" speeds it up a lot.
        • dofm2 hours ago
          It does. It's quite good with the low setting. I prefer Muse Glimmer for, I dunno, hard to explain comfort reasons.

          I had to hack the template of Qwen 3.8-27B to get it to go into low reasoning mode. Not sure if that is fixed.

          Have you tried Nemotron 74B-9B? (Slow on Mac with llama.cpp due to a missing tensor type)

          • electroglyph32 minutes ago
            that nemotron model is such a dog, like all the other nemotrons
            • dofm5 minutes ago
              I have found it to be OK on sort of general purpose overview code stuff. At the moment it's too slow in llama.cpp on Mac for me to bother testing further.
  • serfan hour ago
    "ThinkingCap PRO: want even better efficiency? The PRO version of this model uses up to 87.6% fewer thinking tokens, 62% on average. Faster answers and lower inference costs at scale. Contact us at enterprise@bottlecapai.com."

    lol what's old is new again, software selling performance options like it's the beginning of the multi-core days again.

  • syntaxing2 hours ago
    I tried using thinkingcap Qwen 27B for about two weeks. It definitely does dumber things than vanilla 27B but isn’t too far off. Swift 1.5 is noticeably dumber that I stopped using it after one day. Oddly, I switched back to regular 27B because the thinkingcap one is noticeably more censored. Anything explicit gets instantly shot down by thinkingcap while vanilla 27B will chug along.