> The big, old ideas about intelligence that ended up basically vindicated were the ideas about how intelligence is about prediction, and prediction is about compression, and compression is about finding better and better upper bounds on Kolmogorov complexity. Not the self-reference stuff.

> What’s left? Consciousness and subjective experience of course remain extremely mysterious. For all we know, Hofstadter could be right that those have something to do with self-reference. (For all we know, even Penrose could be right that they have something to do with exotic physics accessible to biological brains but not digital computers!)

I'm not convinced that this is the case. Biological entities have a spatial-temporal sense, a memory, and, alongside those, a predictive/simulating function in space and time. A cat who knows where it is in the world at a given time, sees a mouse, and predicts where that mouse is going to go, is essentially performing a physical calculation, much as you do when you attempt to catch a ball. The more [powerful + general] the predictive function, the more intelligent a given biological entity is. Past a certain threshold, the predictive function becomes capable of broad abstraction. This enables man to acquire, process, store, represent, and continually re-acquire (in a Bayesian sense) knowledge that is external to that man's subjective existence. This is pure calculation in the strictest sense, and doesn't require exotic physics or self-reference beyond a sense of being bounded in space and time, which is to say a reference point.

All that aside, Hofstadter and Chomsky have been pretty hard-hit by LLMs!