But, while very impressive (and it is!), that's not (IMHO!) what's so great about this repository... what's so great about it is the following list that starts out as follows:
>"This improvement in training speed has been brought about by the following techniques:
o Modernized architecture: Rotary embeddings, QK-Norm, and ReLU²
o The Muon optimizer [writeup] [repo]
o Use FP8 for head, and asymmetric rescale and softcap logits
[...] (etc., etc.)
See, that's like a who's who (or a what's what, as the case may be!) of
all the latest LLM training optimization techniques
In other words, it is a very information-dense list about optimization techniques for LLM training.
Think Mike Abrash, but 2026 and for LLM training...
In other words, a very high signal-to-noise ratio here!
Worth studying and re-visiting in the future for anyone into speeding up LLM training and/or people who just like algorithms in general... what a great list, and what a great tool!