(1) they are claiming to produce apparently bijective closed-form symbolic representations/approximations of, among other things, LLMs. Is evaluating these closed-form representations more computationally efficient? The implications of that are potentially huge. It would be essentially analytic distillation. Fable on a chip and not a data center would be important — and disruptive - in many ways.
(2) Unsupervised, and even supervised, symbolic approaches to problem solving break down due to combinatorial explosion, among other things. This could potentially allow us to treat LLM training and inference as a search algorithm for novel symbolic approaches to solving new classes of complex problems hitherto unreachable through other approaches. If that works, I suspect it’s a feedback loop, too - the learnings from one representation push advances in the other. This would also increase the economic value of large training runs, since the model itself is now valuable, not just its inference.
(3) Per the above, can this push LLM design to greater capabilities?
The relationship between this and Anthropic’s J-space observation is also interesting. This is much, much deeper and more directly actionable, though.
EDIT: I ran my questions through Sonnet — yes, I appreciate the irony — and it was none too sanguine about questions (1) and (2), but thought (3) was reasonable. In any case, this is quite the paper. On reflection, I do think that the apparent reliance on very simple symbolic representations and tasks is underwhelming. But the approach is impressive. And obviously this is still early days, and the value of building a bridge between the very fuzzy LLM models and the rigorous, mechanically provable models would be enormous.
That's pretty cool. I hope I've got that kinda-right.
Just going from 2D to 3D creates massive new positional potential (e.g. surface of the earth, vs. the atmosphere above earth...).
Now imagine 1,000 dimensions.
Training identifies these dimensions in the training data and links it with each word/token. Then given a stream of such tokens, each with its own set of dimensions (which can be huge), and LLM predicts the dimensions that the next token is most likely to have...
The trivial example that comes to mind is the character customization sliders in many video games.
That is why the scam works, because investors are humans...
Seeing LLMs for what they really are will also make it clear they are fundamentally unfit for a lot of tasks they are currently marketed for...
It's been several years now of LLMs only appeasing those with low expectations and inexperience. Unless the only goal was generating boilerplate or really sloppy proofs of concept, LLMs are a waste time for everyone else. This argument is so over already. We're all just hoping for a soft landing when the hangover really kicks in.
I'm sure someone will point out something like the 4-color theorem as a counterargument. Where is that kind of theorem proving in this generation of AI? We seem to have hit a dead end rather quickly.
their algorithm is even named "DISCOVER" so they set out to discover the connective tissue of why the universe has invariants like math, and lo it was discovered.
i guess good job for having credentials & publishing the math so people 2years behind the curve can learn from your tenure?
yes. large matrices can gradient descend to understand arbitrary symbolic logic.
ENGLISH IS INSUFFICIENT but it is at least a few decades of math proofs & progress :) welcome to the future Slackernews