While qwen3-4b is smarter than I remembered it, newer models seem to be way better, K2 Horizon 3.7B for example. Is there a reason why that qwen model was choosen?
My last question would be if the same optimizations could be made to a Jev-like model. I mean Jev is already fast and probably has a high throughput per H100. So could we maybe get to >2B tokens/minute/GPU with such a model?
We chose the Qwen model for the initial and core results because it's a simple architecture, which made the roofline cost modeling very easy and trustworthy. I'd already done related work with that model before for another project on speculative decoding, so we had a reference impl as well.
For brevity, this post focuses on a subset of the results. In the companion post on the Full Stack Data Lab blog[1], we go into more detail. With any luck, that'll be a nice VLDB-bound arXiV paper soon!
We have some more results on DiffusionGemma that we tacked on after Jev started popping off. Hard to say much about the specific Jev model, since the details are not known -- even to me, working with their team as they scale up on our platform.
But generally, I think a Quail-like query planner-based approach is a slam dunk for this broad set of classifier models. Quite likely that the benefit is primarily for backend/analytics queries though, where the superior KV management has a chance to shine.
2B tokens/minute/H100 happens to be roughly the speed of light for Qwen 4B. Probably unattainable, for the reasons we indicated (and others, e.g. manufacturer overstatement of peak rates), but I wouldn't be surprised if we could squeeze another 20-30% with CUDA PDL or a megakernel.