Great writeup, especially liked the line by line explanation :)

We actually did manage to get dynamic dispatch working in Compiler explorer with Matt's help: https://gcc.godbolt.org/z/KM3ben7ET

Maybe interesting to also refer to Gemma.cpp's version: https://github.com/google/gemma.cpp/blob/dev/ops/ops-inl.h#L...

We recently added a FastExpMinusOrZero, only valid for inputs <= 0, which is true for softmax.