I’ve been training two small models on a classic long novel (Hongloumeng, 2.6M bytes) to see how they "learn" a story. After a few days of watching the logs, I noticed something interesting about where Transformers struggle.

The "Goldfish" Problem The Transformer is incredibly fast at learning how to finish a sentence. Because it uses "Attention," it’s like a student with perfect short-term memory. But it is "blind" to the long run. It only sees 128 characters at a time. It has no way to remember the beginning of the book while it's reading the end.

The Crossover My "Infinite Brain" model (a recurrent architecture) started out much worse. It was confused and the output was garbage. But around the 5th time reading the book, it "crossed over" and started beating the Transformer.

Because the Brain carries a small "memory state" forward forever, it eventually builds a "vibe" of the whole book that the Transformer just can't see.

What I learned:

Transformers are like great eyes: They see the immediate details perfectly.

Recurrent models are like a stomach: They digest the whole thing slowly, but they keep the "nutrients" of the story for much longer.

It’s a small toy experiment, but it reminded me that while Attention is powerful, having a persistent "soul" or memory state still matters for long-form data.

......

https://github.com/MrPan2048/GeometricTransformer