145B parameters with only peak 29GB memory usage - really impressive, truly something you can actually run on a decent spec MBP
And I assume the metrics comparing qwen are using a quantized version - from the blog they compare memory use against qwen's "standard 4-bit deployment". Which might explain the improved performance over that base model they show - it might lose less from quantization.
Also, why ship both a 145B and a 120B size? That's basically the same size class.
This looks like it would actually work on my 64mb M1 MBP. That would be fantastic.
It is strange that they also benchmarked an M2 Ultra, but they are specifically not releasing a Mac runtime today. Maybe that is still under development.
This score is roughly on par with 5.6 Terra max, which in my direct experience is perfectly capable of driving a cloud agent harness that can do pretty complex knowledge work.
Definitely worth an evaluation!
Damn, guess I can't run this on my desktop. /s