I'm testing it on my old iPhone 13 Pro (A15 Bionic, 6GB RAM) with Ling-mini-2.0, a 16B MoE model. The first tests are showing around 13 tokens/sec, which is pretty encouraging for a phone from 2021.
It's still early, and I need to test memory usage, longer generations, and how performance changes as the phone heats up.
I'll keep posting updates here as I run more tests and see how far I can push it.
I guess I'm not really sure what the point of this (100% AI generated) article is.
You have a magic engine with zero details, that you're comparing against a (to me at least) obscure github project.
I guess I'm not really sure what the takeaway for me is supposed to be. The benchmark setup is not detailed enough to make meaningful conclusions from it, and I guess the big takeway seems to be that a bounded cache that an engine process owns behaves better than relying on the system cache?
Paste this into the claude session: "Add a section explaining the architecture of this engine, and how it compares to TUFF, EdgeMOE, etc SOTA implementations. Explain the novelty of this implementation. Show how experts are imported / evicted, and how that decision works. Also add more detail in a benchmark section about the exact testing you ran. Give test runs, upload to a git repo that links to this article. Include some charts showing test to test variance. "