The transformer based approach is seen as a highway to race down in terms of training and building better models with minimal organizational distractions allowed while every company puts its foot down on this highway. Exploratory research into other AI methods is highly discouraged for now and organizations are encouraging transformer based scale out and iteration as a top priority.
As an example Meta literally fired it's previously head of AI who held views that transformers won't scale. He wanted to veer off the highway and heavily fund exploratory new types of models. They got someone willing to scale the current method out. Essentially the messaging it "Foot down, there's a straight highway ahead don't divert".
I have the view this makes sense too. Current top models are mere (mere in big datacenter terms) trillions of parameters right now. We definitely haven't hit the ceiling and everyone's noticing "Hey this Opus 5.5 with the increased parameters seems much better than the last one". Well that's a trillion parameter model. We'll soon start hitting 10's of trillions and 100's of trillions of parameter models. Unless there's a ceiling which nobody expects they should keep scaling just as they have been. I really do agree that it'd be insane to veer off this highway with exploratory research at this particular point in time. Foot down, get to 10 trillion parameters, see if the scaling holds, if so off to 100 trillion parameters. If the scaling continues to hold we'll be asking the AI to do the exploratory research for us soon enough.
Almost all statements Yann LeCun made back then (LLMs won't be able to do this or that) turned out to be wrong. They were right to fire him.
The author states "I consider the scaling-only focus to be both misdirected and not fun.". I get the point but also totally get the focus on scale out for now.
Already happening: https://news.ycombinator.com/item?id=50397994
Obviously we'll see how it pans out, but other approaches feel a bit like trying to ignore the bitter lesson.
Perhaps "The End of Microsoft Research's AI Frontiers Group" would be a less ambiguous title.
To the author: 100% need this kind of thinking. It’s a balancing act. Historically though, dedicated labs haven’t had the best track record at commercializing their tech. The skillset differs vastly from running a product-focused business.
That said, even within this narrow scope - there are areas ripe for research.
Some of my particular interests include:
(1) Latent thinking/reasoning, which could arguably be more efficient but also opaque is one
(2) Separation of “intelligence” from knowledge - ie. keep the ability to reason like a frontier model and make tool calls (calling the ‘oracle’ or searching the web) but remove all parameters associated with niche knowledge/facts.
These are a couple off the top of my head that are very much aligned to scaling.
That statement is broad enough to be unfalsifiable, but if you narrow it down just a bit to scaling LLMs, maybe there is an end in sight. At least financial markets are skeptical of there being no end in sight.
If scaling LLMs really was the road to God in a box, then compute demand would be infinite, and nobody building to meet that demand would have a junk debt rating. Did those money people just not get it?
https://en.wikipedia.org/wiki/Digg#2025%E2%80%93present:_The...
Seems like it quickly pivoted quickly to cash in on "AI" hype.
I'm not sure how that, combined with open models inching up on frontier intelligence, is compatible with these massive AI IPO valuations?
It seems like closed model AI demand has a lot of contingencies that are ultimately fragile and possibly short sighted.
If your research aligns with the incentives of industry, you can get the best of both worlds in industry. This was the case for machine learning researchers in the post-AlexNet, pre-ChatGPT era. The good times can't last forever, though.
Perhaps it is more free, but my understanding has always been that professional academia still restricts freedom of research. You have to apply for funding, convince the board that your direction will be fruitul.
An archaeologist can't just dig wherever. They compete with their peers for a limitted funding pool. If the school's board is more interested in the bronze age than the paleolithic, you're probably not going to get your dino dig approved.
The same is true for CS, I assume. You can't just "i would like spend university resources developing brainfuck 2.0" because the other teams are more likely to get their funding over you getting yours.
Both corporations and university's have finite budgets, both with bottom lines and margins. The uni wants their research done in a popular field that will attract students because students generate income that funds research that attracts students that generate income that funds research that...
They just think it makes sense to explore things, rather than just rinse and repeat with more cash.
The good news for John is that when 99.99% of the competitors are doing all the same, it’s when the rewards for looking around can be the biggest. Those are the real moments for disruption, and companies disregard them at their peril.
The working environment in academia is often precarious :-( , so such people often have no alternatives to going into industry, even though they still deeply appreciate and are influenced by the values of academia.
It was my introduction to feature hashing.
The more interesting idea is that MS has abandoned AI research entirely, presumably because their zillions invested in other companies are a better bet in that area.
This is certainly an interesting timeline in this section of the light cone
As a result, you end up with companies doing basic research under the guise of developing a product, announcing timelines which are never going to be met. Fusion energy and quantum computing startups are some of the most egregious examples of this.
Remember Bell labs, XEROX parc? Ask yourself why that existed and why research came out of them.
Monopoly.
So - many people don’t want to admit this but monopolies can be good from an R&D perspective, but without strong oversight by regulators - money will instead spent on R&D projects that specifically raise entry barriers.
P.S. Getting downvotes (multiple) for basic economic facts is ...something. Someone flunked their economy 101.
They look more like complete organizational shit shows.
That's just patently not true. History is littered with products that were very innovative and useful (and often far superior to competing products), but were canned, bought by competetors, or shelved in other way - often because they threatened the profits being made by some entity (though that's not the only reason).
Some are, some aren't. Some try, but botch it.
Edit: Fara is open weight as well. Seems like a great match for https://www.microsoft.com/en-us/windows-365/agents ?