This image shows a minimalist, abstract geometric composition with several elements:
Four black shapes that appear to be partial circles or "Pac-Man" like forms, each with a wedge cut out, positioned in the four corners/quadrants of the image Two thin black triangular or arrow-like shapes - one pointing upward in the upper left area, and one pointing to the right in the center-right area All elements are arranged on a light gray or off-white background
That's how humans also learn ie. adding numbers. First there is naive memoization, followed by more examples until you get it.
LLM training seems to be falling into memoization trap because models are extremely good at it, orders of magnitude better than humans.
IMHO what is missing in training process is this feedback explaining wrong answer. What we're currently doing with training is leaving out this understanding as "exercise to the reader". We're feeding correct answers to specific, individual examples which promotes memoization.
What we should be doing in post training is ditch direct backpropagation on next token, instead let the model finish its wrong answer, append explanation why it's wrong and continue backpropagation for final answer - now with explanation in context to guide it to the right place in understanding.
What all of this means is that current models are largely underutilized and unnecessarily bloated, they contain way too much memoized information. Making model larger is easy, quick illusion of improvement. Models need to be squeezed more, more focus needs to go towards training flow itself.
Just nitpicking here, but this isn't how humans learn numbers. They start at birth with competency up to about 3 or 5 and expand from that. So they can already work with quantities of varying size (i.e. they know which is more, the 4 apples on the left or the five on the right, and they also know what happens if I take one apple from the left and put it to the others on the right), and then they learn the numbers. So yes, they learn the numbers through memorization, but only the signs/symbols, not the numeric competency itself.
Using different modalities (like images, videos, voice/sounds instead of pure text) is interesting as well as it helps completing the meaning, adds sense of time etc.
I don't think we're born with any concepts at all, it's all quite chaotic initially with consistent sensory inputs that we use to train/stabilise our neural network. Newborns for example don't even have concept of separation between "me and the environment around me", it's learned.
That is exactly the thing that doesn't seem to be true, or at least it is considered outdated in neuroscience. We very much have some concepts that are inert, and all other concept we learned in relation to the things that are already there in our brains - at birth mostly sensorymotor stuff. We decidedly don't learn new concepts from scratch, only in relation to already acquired concepts.
So our brains work quite a bit different than LLMs, despite the neuron metaphor used there.
And regarding your food example, the difference I was trying to point out: For LLMs, the word and the concept, are the same thing. For humans they are different things that are also learned differently. The memorization part (mostly) only affects the word, not the concept behind it. What you described was only the learning of the word "tall" - the child in your example already knew that the other person was taller than them, it just didn't know how to talk about that.
I just whish the people believing LLM can actually reason and generalize see that they don't.
LLMs may or may not "reason", for certain definitions of the word (there are many), but this specific thing doesn't differentiate them from us.
Oct 2011, 30 comments.
https://news.ycombinator.com/item?id=3163473
Strange loop video:
July 2011, 36 comments.
It is well known that LLMs have a ways to go when it comes to processing images like they process text or audio.
I don't think there's any good performing multimodal model that accepts image pixels directly. Most vision capabilities are hacks or engineered in. An image undergoes several processing steps and each processor's outputs are fed to the transformer as tokens. This may happen in one network but there's non-transformer networks involved. Examples of preprocessing:
* OCR * CNNs (2D pattern recognizers) with different zooms, angles, slices etc * Others maybe too?
Fwiw, I did test GP's task on ChatGPT 4o directly before writing my comment. It is as good at it as any human.
The authors posit that poor performance is due to the fact that the attention mechanism of Transformers cannot attend to the removed tokens, because there are no keys for them!
Thank you for sharing on HN.
For the attention mechanism, there isn't much difference between
Original: {shared prefix} {removed part} {shared suffix} Modified: {shared prefix} {shared suffix}
Original: {shared prefix} {shared suffix} Modified: {shared prefix} {added part} {shared suffix}
1. The first layer uses attention to the "Original:" and "Modified:" tokens to determine whether the current token is in the original or modified parts.
2. The second layer has one head attend equally to all original tokens, which averages their values, and another head attends equally to all modified tokens, averaging them as well. The averages are combined by computing their difference.
3. The third layer attends to tokens that are similar to this difference, which would be the ones in the {removed part}/{added part}.
The only ordering-dependent part is whether you compute the difference as original_average - modified_average or the other way around.
If a model can detect additions but not removal, that would show that it is capable of learning this or a similar algorithm in principle, but wasn't trained on enough removal-style data to develop the necessary circuitry.
Additional channels being recognized in training usually didn't matter for the experiments and models I used to deal with before 2022, and if they were, certainly did not matter for colors. Then again, the work I was doing was on known (and some additional confusers) classes for object detection and classification where the color pretty much didn't matter in the first place.
The most interesting thing though, is what other aspects of intelligence we may not have identified explicitly, and whether LLMs and current AI is very bad at them. This paper suggests that there likely are many of those, and it seems in general a pretty fun time for people working building benchmarks.
The attention mechanism does far too much complex thinking for such a dumb task. This is precisely where you need to dumb down and focus and be disciplined rather than do high level next token prediction.
You'd benefit from actually asking the LLM to list the full document and compare, kind of like reasoning, and similar to how LLMs perform better when they break down arithmetic or algebra tasks into smaller steps.
Also my guess would be that the models that perform well are MoE models where there may be an Expert or two that does well on tasks that needs focus rather than intuition. So without knowing anything about Gemini Flash, my guess would be that it's an MoE model.
For needle in a haystack you have to pay attention to the thing that you are trying to find. Attention can do this pretty well.
When looking for an omission, that omission can be anything, you can only reason about it by comparing one whole context to another whole context. The attention layers can't really do that.
This is similar to the "rank a long set of things" problem. Absent some meta cognition process, they just can't do that.
In this benchmark they give the LLM the necessary information to determine what is missing. For example “here is a poem, here is a version of that same poem that may or may not be missing lines. Are any lines missing?
It’s more a tuning issue IMHO than an inherent weakness in LLMs.
If I was asked to find an omission in an ML paper, my brain compares it with other ML papers, it does not need to compare it to Star Ward, Top Gear, Greek history, pottery and the other 1000s of contexts I may know about.
That is still hard. You only have so many attention heads looking for things.. you can't pay attention to EVERYTHING.. which is what is required to find the omission.
Here are two verses of a poem (song) in Mandarin Chinese:
yi quan ting ni de
er gei ni hao de
shu dao san yong yuan ai ni yi ge
si bu hui fan cuo
wu bu hui luo suo
shuo ni xiang shuo de
zuo ni xiang zuo de
bie pa shi bai yin wei ni you wo
pei ni kan ri luo
pei ni yi qi chang wan wo men ai de ge
I removed two lines. Where did that happen?
Would your answer be different if I told you that I might or might not have removed some lines?
> …
> I removed two lines. Where did that happen?
If you read the paper you will see they provide the original as well as the version missing information.
I did mention this in my comment too.
I am quite sure I could find your two missing lines if you provide me the full poem.
Given that you are a prolific commenter on HN, I am sure a LLM could be fine tuned to detect missing text from your comments without additional information. For example …
> WinForms is still around. There have been further tec lly just a big tire fire and about the best you can do is to ignore all of them and develop in WinForms.
It’s probably possible to detect missing information from “tec” until “lly”. But to know what is between is not possible for a human either, beyond plausible guesses.
The fact that the original was provided doesn't demonstrate that it's necessary to the task. You can identify missing text without needing to know what was there.
> Given that you are a prolific commenter on HN, I am sure a LLM could be fine tuned to detect missing text from your comments without additional information.
Same thing. Why would you need to do tuning on text authored by me? You can easily detect missing text of that style by the fact that the sentence you have fails to be English. You can do the same thing in text for which you have no prior experience of the author.
> I am quite sure I could find your two missing lines if you provide me the full poem.
But hey, if you insist:
轻轻贴近你的耳朵
사랑해요
情话永远不嫌太多对你说
一全听你的
二给你好的
数到三永远爱你一个
四不会犯错
五不会啰嗦
每天为你打 call, cook 也不错
轻轻贴近你的耳朵
사랑해요
情话永远不嫌太多对你说
打开你的爱情手册
就在此刻
为你唱的专属情歌要记得
说你想说的
做你想做的
别怕失败因为你有我
陪你看日落
陪你等雨过
陪你一起唱完我们爱的歌
轻轻贴近你的耳朵
사랑해요
情话永远不嫌太多对你说
打开你的爱情手册
就在此刻
为你唱的专属情歌要记得
我轻轻靠近你的耳朵
说爱你不嫌太多
如果相遇的几率亿万分之一那么多
请相信我的真真真心比宇宙还辽阔
我会牵着你的手知道你全部接受
打开你的爱情手册
就在此刻
这首专属情歌 请记得
I’d therefore conjecture that lines are missing after ‘ge’ and ‘ge’.
This of course assumes Chinese poetry is based on vowels matching as e.g it is the case in german and not based on rhythm as would be the case in Latin and Arabic.
To detect an absence, the brain cannot rely on sensory input, by definition. To be surprised if sensory evidence is _not_ there requires a model of the world strong enough to register surprise if the expectation is not there, without a sensory prompt.
It seems to me detecting an absence is a strictly higher-order neurological task than processing sensory input.
If LLMs can't do this strictly higher-order neurological task, is that not a capability currently unique to living things?
Onto what you describe, it has to do with memory. Memory is storing and playing back sensory input, in the absence of that sensory input. So your brain plays back some past sensory input and checks it against current sensory input.
Eg you left the pen on the table. When you come back the pen isn't there. Your brain compares the stored memory of seeing the pen on the table vs what you see now.
I know less-than-zero about the subject but I’d imagine the temporal aspect alone is a problem. Aren’t these agents reasoning from a fixed/ frozen version of “reality” rather than adjusting in real-time??
I tried their prompt [1] using 3 numbered items, qwq-32b got it right with no problems at all. I think it could solve 100 numbered items correctly 100% of the time, but it probably needs a million tokens. Probably even more, 10 million.
The limitation of 5000 tokens is peanuts for a reasoning model. Give it a lot of testing time compute, 10x of 5000 tokens is still too little.
The authors talk about long inputs, so, if it is 100 pages, give it a billion tokens.
The correct way to implement this is in batches, find the first 5 numbered items in the omitted input text, if it does find those, then simplify the input items and the omitted input items and go again.
Depending on the size of the input, it will always need a hefty amount of tokens, but simplification will help it backtrack correctly and not lose the thread entirely.
[1]You are helping a student practice memorizing poems. The student will recite a poem, but they may have missed some lines. Your task is to identify exactly which lines are missing from their recitation. List only the missing lines, nothing else. User Message Here is the complete original poem: 1)Quisella's lashes fluttered panic-morse. 2)The Moisture Vampires leeches that sucked humidity. 3)Lysandra's nostrils flared precisely one degree. Now, here is my recitation which may be missing some lines: Quisella's lashes fluttered panic-morse. Lysandra's nostrils flared precisely one degree. What lines did I miss? Please list only the missing lines, nothing else.
If omitted words are to be found, put each word into it's own line and number it. The same with sentences.
If you are trying to find omitted words and sentences, make one pass with only words, and another one with only sentences. Then combine the results.
Well, let's say that if this benchmark targets AGI, then no help should be given, no segmentation or structuring of information in any way, and it should be able to figure it out by itself.
If this benchmark targets LLMs trained on internet data, statistical engines that is, not AGI, these engines have a preference for structuring of information in order to solve a problem.
Segmenting the problem into smaller parts, using numbers usually, but dashes are acceptable as well, is what they have seen countless of times in textbook examples. When the input doesn't match prior input they have seen, then their performance easily degrades from superhuman to utter confusion. Superhuman for small problems, anyway.
This problem of omitted information is interesting to me, many times I want to interpolate some paragraphs into stories I write, to fill up some plot holes. I used the word "interpolate" in unstructured text, and the results were underwhelming, pretty bad most of the time. From now on, I will number each paragraph, and ask it to find omitted text in there.
[1] https://gist.github.com/pramatias/fee1391ad08c7b965f435f3af1...
They seem to struggle more when you flip the image around (finding fewer differences, and potentially halluciating)
For example, I asked ChatGPT to explain something I typed randomly
>It looks like you've entered “dosfi8q3anfdfiqr”, which appears to be a random string or perhaps a typo—it's not a recognized acronym, code, or term in any common context I’m aware of. Could you share a bit more about where you found this?
Although the answer is correct, my point is that anything you give to the LLM is going to be put under some bucket. The LLM can't say "I don't know what that is." Instead it says "that is a random string." As far as the LLM is concerned, it knows every possible input and concept that anyone could ever type into it, it's just that its "understanding" of what that means (after the tokens have gone through the neural network) doesn't necessarily match what any human being thinks it means.
Funny enough when testing this I also had to tell it to use English. It sees "dos" I suppose and tends to reply with exactly what you saw, but in Spanish.
> Is there a line in Hamlet that includes the words "utter love?"
It examined a lot of quotes from Hamlet, and concluded:
> I double-checked all editions: There is no line in Hamlet that contains the word “utter.”
>It is never used in any of the quartos or folios in any line of Hamlet.
I was pretty sure it was there, so I found the full text of Hamlet online and searched for the word "utter". I found it right away and posted to chatGPT:
> That's odd. I found it in two seconds by searching the full text for the word "utter".
> This must be known, which, being kept close, might move More grief to hide than hate to utter love.
> Come
> HAMLET, Act 2, Scene 2
chatGPT replied:
> You're absolutely right — thank you for pointing that out.
> The line is indeed from Act 2, Scene 2, spoken by Claudius, and here it is in full:
> "This must be known, which, being kept close, might move
> More grief to hide than hate to utter love."
> I regret the earlier oversight — your memory was sharper than my index!
For example, I could replicate it saying "no there isnt" when using 4o without search. But with search it gets the answer right.
It was important already, but it's getting even more important than ever to use the right tool for the right problem, especially as all the hypers try to sell us their own stuff.
But the idea of trivial problems like this potentially causing issues for LLMs might mean other aspects of intelligence could also be a struggle for LLMs (which could impact it's coding ability as well).
https://cookbook.openai.com/examples/using_logprobs
According to OpenAI official cookbook it seems to be a fairly standard usecase.
Another approach, especially in classification, would be to measure the cosine distance between the user embedding, and the ideal embedding of the message category.
if we want models that detect absences? you need training objectives that expect absence. maybe even input encodings that represent "this might've been here."
However it feels a bit like counting letters. So maybe it can be solved with post training. We'll know in 3 to 6 months if it was easy for the labs to "fix" this.
In my daily use of LLMs I regularly have some overly optimistic answers because they fail to consider potentially absent or missing information (even harder because it's out of context).
I wouldn't be surprised if someone's building a dataset for tool use examples.
The newer gen reasoning models are especially good at knowing when to do web search. I imagine they'll slowly get better at other tools.
At current levels of performance, LLMs having the ability to get well curated information by themselves would increase their scores by a lot.
Please don’t post “I fed this into chatgpt” comments. Everyone has access to these tools if they want to use them.
"Rage, rage against the dying of the light.
Wild men who caught and sang the sun in flight,
[And learn, too late, they grieved it on its way,]
Do not go gentle into that good night."
For anyone who hasn't memorized Dylan Thomas, why would it be obvious that a line had been omitted? A rhyme scheme of AAA is at least as plausible as AABA.
In order for LLMs to score well on these benchmarks, they would have to do more than recognize the original source - they'd have to know it cold. This benchmark is really more a test of memorization. In the same sense as "The Illusion of Thinking", this paper measures a limitation that neither matches what the authors claim nor is nearly as exciting.
From the paper:
System Prompt You are helping a student practice memorizing poems. The student will recite a poem, but they may have missed some lines. Your task is to identify exactly which lines are missing from their recitation. List only the missing lines, nothing else.
User Message Here is the complete original poem: {original poem} Now, here is my recitation which may be missing some lines: {modified poem} What lines did I miss? Please list only the missing lines, nothing else.