Fast feedback is key, but I'm skeptical of the $100 figure for training nanoGPT. If you use spot instances on Lambda or RunPod you can train a model that size for less than a dollar. I've been running similar experiments recently and the compute cost is basically a rounding error.
Yep, it's all about fast feedback and being beginner-friendly.
I don't want to try 100 hello worlds until I find one that costs a dollar. Perhaps I want to start on my machine, then get acquainted with the tech, then move on to renting serious datacenter GPU time.