With that said, maybe he meant inevitable as N approaches infinity... So there's still time!
Aside: I think the biggest interesting prediction he had was that AI would obey a user driven network effect. It seems like a very clean prediction, and yet it has failed to materialize? I wonder if this has more to do with just how scrape-able the whole internet was or with distilation or what? There is certainly an advantage to having good user data, but it seems like it is largely something that can be worked around (by hook or crook)?
Possibly the best photography book I've come across. It's out of print, but I managed to snag a copy on eBay last year
At the most abstract level, you can observe patterns in a model's activations and theorize that particular collections of components contribute to particular kinds of operations. There are many flavors of this intuition. In that sense, you can think of there being "submodels" within a model, insofar as different parts of the network can participate in specialized computations. Some reading that might interest you in this area would include things like mechanistic interpretability.
There are also Mixture of Expert models, which are a dominant architecture for LLMs now, in which you essentially have a bunch of smaller FFNs that get routed over/aggregated together when the LLM performs a forward pass. It would be a mistake to treat any single expert as a better task specific model though. They are trained as part of an ensemble, and they're almost always being combined with other experts in the forward pass, so they're more like (to use a loose analogy) regions of the brain that tend to correspond more towards particular actions than others. You couldn't just extract one and say "This is now our Chess model that performs as well as the larger LLM we took it from". You could potentially use the relevant experts as a basis for training a smaller task-specific model, and there is work around expert pruning and other distillation techniques, but it's not as simple as "This smaller inner model is a complete task-specific model as is". And more importantly, that ensemble of experts *is* the LLM you're comparing it to, at least in major part. It doesn't really make sense to treat the LLM as some foreign model that calls the expert. The experts themselves don't often correlate to easily interpretable task domains, as they've been learned via joint computation with the other experts and broader LLM.
Then there's tool calling. Today LLMs are trained to make tool calls (basically execute functions) for particular tasks. Those tools can be smaller specialized models, or deterministic code, or anything really that can be fired from a host environment.
The most striking thing is that almost all of his missed predictions were missed not because the technology was not available, but because society rejected or didn’t adopt that technology
OK, so imagine that in 1980, I predicted that we'll colonize Mars by the year 2020. Today, I'm arguing that I was right in my prediction - in broad strokes, we have the technology! Nothing is stopping us. It's just that politicians have different priorities.
Of course, that's a load of bull. You can't decouple predictions about the future of mankind from what said mankind actually decides.
I looked up the predictions on Wikipedia and I'd say that most of them are general enough that they'd hard to disagree with back in 1999 (there will be bioengineering, there will be portable computers, there will be robots, there will be AI), but the timing or the specifics are off on almost every single one of them.
For instance, by his prediction, we're already many years past a 100 year average life expectancy (with 120 years expected by the end of the decade).
I think his mistake was tightly coupling exponential compute with exponential progress in biology - which has so far not panned out.
I would argue that society would likely have been glad to adopt, but the profit-seeking motive got in the way (again).
I was focused on augmented reality largely in part because it was such a big enabler of all other AI technologies
However people have almost uniformly rejected AR glasses my entire career for social reasons and that is only increasing
To the point where in 2016 I was debating with Jessica Alba and she was firm that they won’t take off cause most people don’t want to wear glasses