Thanks for sharing your work!

Please how to select the model? I downloaded tinyLlama, put it in ./models, changed reame.conf but I get:

(No such file or directory)

If a prefix cache is good, why not keep it in primary mem?

The persistent KV cache is interesting; I’d love to see how much of the speedup remains when requests share less structure.

Why qwen 2.5 everywhere? Why not 3.5?