Some context about this:
- This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs
- Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but used v complex architectures or extremely high amounts of training compute. No one expected a simple AR transformer to perform this well, at this low cost and w these few training samples.
- Sample Efficiency is one of the most important unsolved problems today in AI. That's what I was targetting with this work. We know it is easy to increase SE by increasing compute/params, so it was important to constrain cost as much as possible (also why OpenAI's Parameter Golf had fixed compute and why Modded NanoGPT is considered very sample efficient)
- Can the perf be improved? Yes but the competition is ongoing so can't talk about it
- Personally I think today's frontier models can be beat by training from scratch. Haven't proved this yet tho
- Fun: I was new to ML when I posted this first (dec '25). I basically used ARC as a way to learn ML
- it feels logarithmic (like most perf-compute graphs), and eventually plateaus. 44% @ 67 cents was a good stopping point for me
- more compute would require a lot of effort and dealing with new problems like training stability, cost of iterations/sweeps (didnt have the money to convincingly run larger iterations)
> Training on the eval puzzles is cheating / “training on test”
> No this is false. “Training on test” specifically means training on the labels of test data. The labels were not trained on.
> Also, ARC is a metalearning benchmark, so you’re supposed to learn from the eval puzzles.
> Jargon: ARC has a set of train puzzles and a set of eval puzzles. Each puzzle has example pairs and test pairs. A pair consists of an input grid + output grid.
> The ARC, the label is only the test pair’s output grid in an eval puzzle.
> These labels were not trained on. They are hidden. You can delete it beforehand if you wish
I think what I gather here is that the test comes with one batch of training problems, which everyone agrees you can train on. But maybe the eval problems also come with input/output examples (to help define the problem) and training on those is controversial? I can’t see why it would be controversial but is that the criticism?
The other tension is the fact that this score is on the public eval set. In machine learning, you typically have 3 datasets: training, evaluation, and test. The training set is the dataset that's used to update the weights according to your loss function, you are "encoding" the patterns from the training set directly into your model. The eval set is what you use to track performance while training, it is NOT used to update model weights, but shows how well the model generalizes. The test set is a private holdout set that is only used when you're "done" developing your model. The difference between test and eval is information leakage: you can use performance against the eval set to modify your hyperparameters and model architecture to get better eval scores. So while the eval set doesn't directly update the weights, it can indirectly cause "overfitting" by tailoring your model to do well on the eval set. What you really want to see is the private test set performance, not the eval set. For all we know, this model could be ridiculously overfit on the eval set and perform poorly on the private test set.
In the test dataset's Q,A pairs, it was only trained to next-word predict the question itself, and it was not given the answer at all.
It was then evaluated by seeing if it is able to output A_test given the Q_test as prompt.
What would be cheating is training it to produce A_test (given Q_test as prompt) as well, since then you can always make a model that scores 100% by just memorising Q_test, A_test pairs.
The complaints online mostly stem from not reading that properly and assuming they trained on the test set. This is further because these days large LLMs are inadvertently trained on many benchmark solutions even unintentionally due to the massive scale of data and the infeasibility of auditing it all. But none of that is the case here.
The reason you want to train on Q_test is because in these AR transformer models, they learn useful composable encodings of Q by simply learning to next-word predict Q. So you enable the model to learn composable encodings of the test questions, so that it can hopefully "connect it" to an earlier train problem it had seen, and adapt the solution it had seen for that, much like humans do in school exams.
Without this step, you are making it difficult for the model to "connect" the test question to a train question it had seen earlier, and then it still has to adapt the solution. This way, you precompute that "this test question is like this train question" and then during the exam you only have to do the adapting the solution part after a simpler "retrieval" process.
This practice often used in continual learning or "test time training" is not yet useful in general real world realtime ML tasks due to the differences in memory and compute requirements, and more so the general fragility of training large neural networks, versus inferencing from a static neural network.
While the increased compute and memory is difficult to solve inherently, there are various efforts being made to fix the fragility, especially in reinforcement learning where this is called "streaming RL", there is revival of interest as seen in RLC 2026.
[Note] Arc-AGI-1 doesn't have any actual english words or such, but it's simpler to pretend it was a basic Q&A benchmark to explain the above
I saw this on the community note for the last blog you wrote - anything to do here.
“Doctor.”
I loved the subject and nerded out about the course material - I spent my time designing my own experiments around gene cloning that took several semesters to run. They were sharing last year's tests with their frat buddies and laughing at us nerds.
I've never looked at doctors the same way again after college. I looked up to them as a child, yet after seeing how the sausages were made, I started to doubt everything.
I frequently ask doctors, who spend all of ten minutes with me while the nurses do all the work, about the molecular specifics of what they're talking about. They talk down to me as if they're explaining to a child, yet they're frequently quite wrong. I'm not trying to sound superior to them, but I'm shocked they seem to care so little about the subject. It doesn't give me much hope about what they know and their abilities or competency.
I suspect surgeons and specialists are a different breed and aren't like this at all.
And to be clear, this isn't everyone. But it does seem to be the majority I've interacted with throughout my life.
When they act disgruntled at patient interaction, I detest that their profession tries to cap the number of med students per year. We should be letting in as many med students as we can take. We should let doctors from overseas immigrate and easily become practicing doctors here in the US. We should provide easy paths for nurses to become doctors.
The premed students in my university were chiefly concerned about money and prestige. They drove BMWs gifted to them by their parents and laughed at what I drove and how hard I studied. I had to put up with their bullying for years. I know not everyone who studies to become a doctor is like that, but it permanently skewed my view of their profession.
Lol, what does this has to do with anything regarding aptitude or curiosity!?
I do agree with the sentiment though that the US needs to fund more residency slots as it's an asinine professional barrier, and that we would benefit from more physicians coming from more diverse financial backgrounds.
Maybe self preservation?
In east europe, public hospitals will force doctors to work 36 hours shifts (overnight ER with theoretical sleep). Doctors have a full criminal liability for mall practise.
I have been wondering the same. We are now exposed to so many stimuli, we are tricked into thinking this is the norm - to have a reasonable understanding about everything, unless specialization is called for.
There was this a few weeks ago:
"Schema Harness Achieves ~99% on Arc‑AGI‑3 Public" https://news.ycombinator.com/item?id=48938163
>> Schema, the harness we introduce today, reaches 99% on the ARC_AGI_3 Public set using Claude Opus 4.8 and Fable 5, and 95.35% using GPT‑5.6 Sol
What does that do with 5.6 Luna instead of the expensive models?
What of 'schema' would improve the performance of mdlARC?
mdlARC: https://github.com/mvakde/mdlARC
There's an updated ARC-AGI-1 chart with 5.6 Luna in each thinking level in this video from last week: "A New Architecture [..] | MOONSHOTS " https://youtube.com/watch?v=qQfUbo7Ldc0&t=2m5s
The new arch in that video is kinda misleading. Didn't really compare against proper baselines
Agreed and that's for any benchmark. Private tests are better but you still have to trust the provider to not log and use them for training.
That's why I like when a new set of tests like a new ARC-AGI version is published, that's where you can see which of the models abstracted to more general capabilities instead of being focused on the previous tasks. Most models completely fail new ARC-AGI tests.
The "67 cents" part though is misleading imho. You can't extrapolate from there and think that investing say $100 will get you a lot better results. You hit a ceiling very fast and investing into more compute will give you diminishing results. So yes, you can train a custom model to do somewhat decently on a specific set of tasks but then what?
Putting solutions in terms of cents is a great way to potentially win over some ai boosters imo. There are other ways to solve hard problems.
I think thats unfair. Perf-compute is often logarithmic and will always saturate . Reaching the plateau faster is valuable as it often leads to better peaks (held true here and also look at modded nanogpt)
And more compute increases the perf (after dealing with other scaling problems)
- how about we allot the possibility that so many of presumed ML experts don't have any clue what they be doing, and are eventually API bitches, nothing more.
If you just wanted to pass these specific tasks in this specific benchmark, and wanted to do so cheaply, I'm sure a non-LLM-based approach would yield better results for even cheaper, since what the author's model does, seem to basically be "solve ARC puzzles", not a general LLM or "coding" LLM.
I found it to be a very interesting angle.
Isn't it a LLM he's building though? My very point is that this particular use case could be solved better without building a LLM, now you claim he is not? The description of what he's doing surely makes it sound like it's a (very small) LLM, and personally I'm still on the "if it quacks like a duck" train in life.
> A lot of people have sort of forgot that machine learning is more than just LLMs these days.
Yeah, which I guess if you make my previous comment more concise, is exactly what I state too.
Please describe what in your mind a "LLM" is exactly, then describe what this person is building. To me this sounds like "He's not building a calculator, he's just building a program that can do addition, minus, multiplication and division and display the results".
Obviously it's not a Large Language Model, but to me this looks more like a LLM than not, given the architecture he's chosen. But again, maybe I misunderstand?
When I made this, the point was to show that you dont need pretraining (which is what makes an LLM) to perform well on complex tasks
And yes it is not a language model either. I did not train it on any language data. Only ARC puzzles
Use of a transformer is not necessary or sufficient to qualify as an LLM.
BUT, he does not use labels when training, so the model does not know the answers.
But benchmaxxing is what we generally try to avoid for training, as there is no point really for it. We used to call it "overfitting", now you're saying this person does it intentionally? Why?
Also, the complexity of the task he is using occupies an interesting middle ground of ultra high dimensionality (for a “simple” problem) while being limited in width to a narrow set of solves- a space where one would be tempted to imagine you would need a much more capable system.
I would not call this overfitting, it's finetuning for specific task where you have a benchmark.