"The machine immediately taught me that capacity estimates are just admission tickets."
"Useful in production, poison in a kernel comparison."
Please don't publish writing like this, it's exhausting to read. You can edit that stuff out.
The best part of this is the "Five xhigh artifacts from the finished model" section at the bottom, I suggest either moving that up or at least prominently promoting it at the top of the article.
Claude seems to speak and write more in Claude-speak with every release.
Long form though? Still pretty bad. Probably getting worse in practice, as people have them write larger and larger chunks of text without paying any more attention to the result.
llm gets them some of that sweet_llm_padding and readers smell the slop and attack the writing :)
people, go write "boring stuff" - its more interesting!
I know this doesn't exactly match your request, but I'm happy with my 3.6's performance. I've seen people claim they can get 120+tps on a 3090, but I'm not impatient when it comes to streaming text faster than I can read it.
As I understand (useful) concurrency for MoE requires very large batches, where about every expert gets activated per pass.
With dense Qwen 27B on 3090/llama.cpp I get:
- no MTP: 1x42, 2x33, 3x24, 4x19 t/s
- MTP: 1x50, 2x30, 3x33, 4x30 t/sNo idea how that compares to running a larger model and context though.
I’ve sweeped concurrency across many models and many different kinds of hardware, and the only times I saw similar results to you were when I didn’t configure it correctly.
I might need to finally make the switch away from it, but it is so convenient, especially as a chat interface for system prompt experimentation.
In my testing I got 150 tokens/sec with a single 5090 RTX.