I think there's more to the linked post than just putting too much emphasis on "we've got to get LLMs to hallucinate less". And I agree that next token prediction will lead to text divorced from reality sometimes.

What the author didn't hammer on enough was his student's complete inability to think. At all. The example kid (who I suspect of being a composite) had relinquished all thought to LLMs. This is a failure of society in one sense. When so much emphasis is placed on the grade or the award, the learning or the process of attaining complex reasoning gets thrown away. As a society, we made "good grades" important, at least to a large extent because they're an easy proxy for harder-to-measure learning. LLMs are exposing that for a bad choice.