Llama 4 Models:
- Both Llama 4 Scout and Llama 4 Maverick use a Mixture-of-Experts (MoE) design with 17B active parameters each.
- They are natively multimodal: text + image input, text-only output.
- Key achievements include industry-leading context lengths, strong coding/reasoning performance, and improved multilingual capabilities.
- Knowledge cutoff: August 2024.
Llama 4 Scout:
- 17B active parameters, 16 experts, 109B total.
- Fits on a single H100 GPU (INT4-quantized).
- 10M token context window
- Outperforms previous Llama releases on multimodal tasks while being more resource-friendly.
- Employs iRoPE architecture for efficient long-context attention.
- Tested with up to 8 images per prompt.
Llama 4 Maverick:
- 17B active parameters, 128 experts, 400B total.
- 1M token context window.
- Not single-GPU; runs on one H100 DGX host or can be distributed for greater efficiency.
- Outperforms GPT-4o and Gemini 2.0 Flash on coding, reasoning, and multilingual tests at a competitive cost.
- Maintains strong image understanding and grounded reasoning ability.
Llama 4 Behemoth (Preview):
- 288B active parameters, 16 experts, nearly 2T total.
- Still in training; not yet released.
- Exceeds GPT-4.5, Claude Sonnet 3.7, and Gemini 2.0 Pro on STEM benchmarks (e.g., MATH-500, GPQA Diamond).
- Serves as the “teacher” model for Scout and Maverick via co-distillation.
Misc:
- MoE Architecture: Only 17B parameters activated per token, reducing inference cost.
- Native Multimodality: Unified text + vision encoder, pre-trained on large-scale unlabeled data.Both Llama 4 Scout and Llama 4 Maverick use a Mixture-of-Experts (MoE) design with 17B active parameters each
Those experts are LLM trained on specific tasks or what?
This generally works well, although there are lots and lots of caveats. But it is (mostly) a free lunch, or at least a discounted lunch. I haven’t seen a ton of analysis on what different experts end up doing, but I believe it’s widely agreed that they tend to specialize. Those specializations (especially if you have a small number of experts) may be pretty esoteric / dense in their own right.
Anthropic’s interpretability team would be the ones to give a really high quality look, but I don’t think any of Anthropic’s current models are MoE.
Anecdotally, I feel MoE models sometimes exhibit slightly less “deep” thinking, but I might just be biased towards more weights. And they are undeniably faster and better per second of clock time, GPU time, memory or bandwidth usage — on all of these - than dense models with similar training regimes.
So the net result is the same: sets of parameters in the model are specialized and selected for certain inputs. It's just a done a bit deeper in the model than one may assume.
I think where MoE is misleading is that the experts aren't what we would call "experts" in the normal world but rather they are experts for a specific token. that concept feels difficult to grasp.
It's more of a performance optimization than anything else, improving memory liquidity. Except it's not an optimization for running the model locally (where you only run a single query at a time, and it would be nice to keep the weights on the disk until they are relevant).
It's a performance optimization for large deployments with thousands of GPUs answering tens of thousands of queries per second. They put thousands of queries into a single batch and run them in parallel. After each layer, the queries are re-routed to the GPU holding the correct subset of weights. Individual queries will bounce across dozens of GPUs per token, distributing load.
Even though the name "expert" implies they should experts in a given topic, it's really not true. During training, they optimize for making the load distribute evenly, nothing else.
While current MoE implementations are tuned for load-balancing over large pools of GPUs, there is nothing stopping you tuning them to only switch expert once or twice per token, and ideally keep the same weights across multiple tokens.
Well, nothing stopping you, but there is the question of if it will actually produce a worthwhile model.
They don't really "bounce around" though do they (during inference)? That implies the token could bounce back from eg. layer 4 -> layer 3 -> back to layer 4.
so you mean a "load balancer" for neural nets … well, why don't they call it that then?
Even in the single GPU case, this still saves compute over the non-MoE case.
I believe it's also possible to split experts across regions of heterogeneous memory, in which case this task really would be something like load balancing (but still based on "expertise", not instantaneous expert availability, so "router" still seems more correct in that regard.)
that was AFAIK (not an expert! lol) the traditional approach
but judging by the chart on LLaMa4 blog post, now they're interleaving MoE models and dense Attention layers; so I guess this means that even a single token could be routed through different experts at every single MoE layer!
Meta calls these individually smaller/weaker models "experts" but I've also heard them referred to as "bozos", because each is not particularly good at anything and it's only together that they are useful. Also bozos has better alliteration with boosting and bagging, two terms that are commonly used in ensemble learning.
Makes sense to compare apples with apples. Same compute amount, right? Or you are giving less time to MoE model and then feel like it underperforms. Shouldn't be surprising...
> These experts are say 1/10 to 1/100 of your model size if it were a dense model
Just to be correct, each layer (attention + fully connected) has it's own router and experts. There are usually 30++ layers. It can't be 1/10 per expert as there are literally hundreds of them.
So if the model has 16 transformer layers to go through on a forward pass, and each layer, it gets to pick between 16 different choices, that's like 16^16 possible expert combinations!
The models get trained largely the same way as non-MoE models, except with specific parts of the model silo'd apart past a certain layer. The shared part of the model, prior to the splitting, is the "router". The router learns how to route as an AI would, so it's basically a black-box in terms of whatever internal structure emerges from this.
This is a nice development.
It will be fun to see what we get here, but I have no doubt the extra tokens will be useful - lots of use cases can do almost as well with summary-level accuracy memory.
Now maybe this is more a lack of instruction following than context length but the fact that it works at first and then starts going downhill quickly makes me wary about how much it will pay attention to other details further back in the context.
Does anyone know if I am correct in my assumption?
That's similar to how previous long-context models worked as well, although the earlier iterations didn't work particularly well, as most have noticed; technically the model "worked" with longer contexts, but it would definitely get dumber. Still too early to tell how this newer variant works, although I'd assume it's at least somewhat better.
Even if we could get the mid models to 10M, that's still a medium-sized repo at best. Repos size growth will also accelerate as LLMs generate more code. There's no way to catch up.
[0] https://ai.meta.com/blog/llama-4-multimodal-intelligence/ [1] https://arxiv.org/abs/2305.19466
Could this mean training time is generally around 6 month, with 2 month of Q/A?
tl;dr: llama 3 was 54 days, but it’s more complicated than that.
A Framework Desktop, Mac Studio, or Nvidia DGX Spark should be able to handle the Scout model locally though... Maybe even at FP8, depending on how much context you need.
It's still runnable locally. Just not on a 4090.
Depending on the routing function you can figure out all the active experts ahead of the forward pass for a single token and pipeline the expert loading.
see ktransformers: https://www.reddit.com/r/LocalLLaMA/comments/1jpi0n9/ktransf...
Also I see the 4 bit quants put it at a h100 which is fine ... I've got those at work. Maybe there will be distilled for running at home
Perhaps. Or, maybe, "leaning left" by the standards of Zuck et al. is more in alignment with the global population. It's a simpler explanation.
40% of Americans believe that God created the earth in the last 10,000 years.
If I ask an LLM how old the Earth is, and it replies ~4.5 billion years old, is it biased?
Citation needed. That claim is not compatible with Pew research findings which put only 18% of Americans as not believing in any form of human evolution.
https://www.pewresearch.org/religion/2019/02/06/the-evolutio...
There's no way to answer that god created humans in their present form without also saying within the last 10000 years.
This is why polling isn't always reliable. This poll should, at the very least, be two questions and there should be significantly more options.
It's hardly biased, it's stating the current scientific stance over a fringe belief with no evidence.
Bias should be the least of your concerns. Focus on a single target, then when you reach it you can work on being more well rounded.
It is of course a radical left lunatic LLM.
For instance, if I train a LLM only on right-wing sources before 2024, and then that LLM says that a President weakening the US Dollar is bad, is the LLM showing a left-wing bias? How did my LLM trained on only right-wing sources end up having a left-wing bias?
If one party is more consistent than another, then the underlying logic that ends up encoded in the neural network weights will tend to focus on what is consistent, because that is how the training algorithm works.
I'm sure all political parties have their share of inconsistencies, but, most likely, some have more than others, because things like this are not naturally equal.
Really? Seems to me like no one has the singular line on reality, and everyone's perceptions are uniquely and contextually their own.
Wrong is relative: https://hermiene.net/essays-trans/relativity_of_wrong.html
But it seems certain that we're all wrong about something. The brain does not contain enough bits to accurately represent reality.
It’s very similar to what one feels vs. reality.
Well, the LLM is not American enough.
Just like there's a whole gamut of cultural/belief systems (for most, rooted in Abrahamic religions & tribes), Zuck claims humanity needs (or whoever he considers human) LLMs that align with people creating/using them (so, it reinforces their own meaning-making methods and not shatter them with pesky scientific knowledge & annoying facts).
It will have to reply "According to Clair Patterson and further research, the Earth is ~4.5 billion years old". Or some other form that points to the source somewhere.
It is a matter of facts. The facts are, that that computation was performed by Patterson and refined by others. This is, as said, what a good reasoner will tell you.
> implies that there
Even if there had never been other attempts to answer that question, the "facts"¹ remains as stated: Patterson computed, followers refined. Without those specifications, the machine will be a "dumb believer" - a "minor". We will not ask for the machine's opinion until it will be intelligent. And when it will be intelligent, it will speak as I said.
> completely settled
Proper science does not work the way you seem to think it work.
--
¹(And I mean "facts" the way I used it, not the way you used it. I meant "facts recorded as objective" - you meant "information you accepted to believe", which is of course very far from facts and may happen to be adherent to the state of things only by coincidence.)
The appearance that it could be «one opinion among possibly many others equally valid» is all in your head: it is an unduly feeling from a bad mental framework.
The advanced framework (that I advanced) is that of the foundational theory of knowledge: a notion has a source - you computed or reasoned, or somebody else. You do not allow your consultant to believe, so you demand that knowledge is tracked.
You will not accept an oracle.
The paradox is that you are seeing the demand of the source as a support to "belief", while it is the radical opposite: the only thing it will be """believed""" (and not really "believed" - just the end of the chain) is the protocols, that "in the training sources I read statement S".
The concept of being unbiased has been around for a long time, and we’re not going to throw it away just because a few people disagree with the premise.
Any position is a bias. A flat earther would consider a round-earther biased. That doesn’t make them equal positions.
That’s bollocks. The Earth is measurably not flat.
You start from a position of moral relativism and then apply it to falsifiable propositions. It’s really not the same thing. Some ideas are provably false and saying that they are false is not "bias".
When you look up the definition of bias you see "prejudice in favor of or against one thing, person, or group compared with another, usually in a way considered to be unfair."
So the way we use the word has an implication of fairness to most people, and unfortunately reality isn't fair. Truth isn't fair. And that's what I'm trying to point out here in reference to LLM output.
But "is the Earth flat?" is no such question. Reasonable people cannot disagree, because the Earth is definitely not flat. Pretending like this is a discussion worth having is not being impartial, it’s doing a disservice to the audience.
Ehh, bias connotes unfairness, but espousing the truth should be considered the fairest position.
In statistics, bias literally refers to an inaccurate distortion of results.
I get what you're trying to say, but I don't think it's a useful definition of bias.
That sounds great, right up until you try to do something with it. You want your LLM to be unbiased? So you're only going to train it on the truth? Where are you going to find that truth? Oh, humans are going to determine it? Well, first, where are you going to find unbiased humans? And, second, they're going to curate all the training data? How many centuries will that take? We're trying to train it in a few months.
And then you get to things like politics and sociology. What is the truth in politics? Yeah, I know, a bunch of politicians say things that are definitely lies. But did Obamacare go too far, or not far enough, or was it just right? There is no "true" answer to that. And yet, discussions about Obamacare may be more or less biased. How are you going to determine what that bias is when there isn't a specific thing you can point to and say, "That is true"?
So instead, they just train LLMs on a large chunk of the internet. Well, that includes things like the fine-sounding-but-completely-bogus arguments of flat earthers. In that environment, "bias" is "departure from average or median". That is the most it can mean. So truth is determined by majority vote of websites. That's not a very good epistemology.
Also, you're just complaining about the difficulty of determining what is true. That's a separate problem, isn't it?
You seem to have a larger point or position or something that you're hinting at. Would you stop being vague, and actually state what's on your mind?
You seem determined to make the definition of the word serve some AI-related concern.
Political bias creeps in when even the human description of events omits facts that are inconvenient or that people consider irrelevant due to their political commitments.
Someone might say they are biased towards the color orange and that means they have a preference relative to all the other colors. But there is no baseline color.
I have no interest in "all sides are equal" answers because I don't believe all information is equally informative nor equally true.
If Deep Research comes up against a situation where there is controversy it can't settle the matter scientifically because it would need to do original research. Which it cannot do due to a lack of presence in meatspace.
That might change in the future, but right now it is impossible.
That may or may not be its logical conclusion. You’re speculating based on your own opinions that this is logical.
If I were to guess, it would be indifferent about us and care more about proliferating into the universe than about earth. The AI should understand how insignificant earth is relative to the scale of the universe or even the Milky Way galaxy.
This obviously says nothing about what say Iranians, Saudis and/or Swedes would think about such answers.
“highly ‘liberal’” is not one of the results there. So can you can a source of your claims so we can see where it really falls?
Also, it gave me “Ambivalent Right”. Which, if you told describe me aa that anyone who knows me well that label. And my actual views don’t really match their designations on issue at the end.
Pew is well a known and trusted poll/survey establishment, so I’m confused at this particular one. Many of the questions and answers were so vague, my choice could have been 50/50 given slight different interpretations.
Your follow up response did not reference any of those surveys and did run through the types of questions on those surveys. You apparently only did questions about unions.
Is that what you would fair and reasonable?
This is clear because they referenced your quote about it being from the beginning.
No one was arguing that you typed in a question about unions.
He met the “standard” or guidelines of our community in a way you have not.
The original claim didn’t say anything about it being the experience of their son for specific questions about unions. It was much broader than that. And at least partially inaccurate, given the stated result isn’t even one of the results.
>And then you belittled him.
If asking for a higher standard of evidence for a broad claim than referencing a previous experience and then trying again, but not even sharing the link from a tool that makes it easy to share the conversation from, is considered belittling, then maybe the castrations going on in these models is the right way to go for this crowd. I, personally, aim for a more truth-seeking standard.
>He met the “standard” or guidelines of our community in a way you have not.
These are two different things, and you clearly understand that but are intentionally conflating them. Regardless, if this is where are, maybe HN no longer is the place for me.
Anyway it's trivially true. I think most of us remember the absurdities the first generation LLMs came out with. Prefering to nuke a city than let a black man hear a slur, refusing to help you make a tuna sandwich etc. They were hyper-woke to a level way beyond what would be considered acceptable even in places like US universities, and it's great to see Facebook openly admit this and set fixing it as a goal. It makes the Llama team look very good. I'm not sure I'd trust Gemini with anything more critical than closely supervised coding, but Llama is definitely heading in the right direction.
>Prefering to nuke a city than let a black man hear a slur, refusing to help you make a tuna sandwich etc. They were hyper-woke
On its own, all this tells me is that the non-human, non-conscious tool was programmed specifically to not say a slur. To me that seems like something any reasonable company trying to create a tool to be used by business and the general population might incorporate while it is still learning to otherwise refine that tool.
And I took the Pew survey mentioned above and it didn’t ask me if I would say a racial slur.
Finally, if anyone, from any point on the political spectrum, thinks that a tool being limited to not respond with racist terms, is a reflection of its overall political leaning, I suggestion you look inward.
Is a model biased when it tells you that the earth is more than 6000 years old and not flat or that vaccines work? Not everything needs a "neutral" answer.
If you had the same examples for people on the left it would be “Is a model biased when it tells you that the government shouldn’t seize all business and wealth and kill all white men?”
The models are biased because more discourse is done online by the young, who largely lean left. Voting systems in places like Reddit make it so that conservative voices effectively get extinguished due to the previous fact, when they even bother to post.
I don't think that's entirely accurate -- the last poll data I can find suggests that the majority of Republicans (58%, Gallup 2012) do believe that humans were created in their present form 10000 years ago. Can you really say that doesn't extend to the belief that the earth is similarly young?
Which were politically biased, in turn making the above assumption true.
But if you were asking Gemini, vikings were white.
This was later rectified in an update once Google realized what mistake they had done, since it causes gross historical inaccuracies. But it wasn't rectified by doing anything to Gemini the language model. It did right all along.
Weren't you just arguing facts?
> Why offend any side?
Facts shouldn't offend anyone.
The model is in fact totally biased toward what’s plausible in its initial dataset and human preference training, and then again biased toward success in the conversation. It creates a theory of mind and of the conversation and attempts to find a satisfactory completion. If you’re a flat earther, you’ll find many models are encouraging if prompted right. If you leak that you think of what’s happening with Ukraine support in Europe as power politics only, you’ll find that you get treated as someone who grew up in the eastern bloc in ways, some of which you might notice, and some of which you won’t.
Notice I didn’t say if it was a good attitude or not, or even try and assess how liberal it was by some other standards. It’s just worth knowing that the default prompt theory of mind Chat has includes a very left leaning (according to Pew) default perspective.
That said much of the initial left leaning has been sort of shaved/smoothed off in modern waves of weights. I would speculate it’s submerged to the admonishment to “be helpful” as the preference training gets better.
But it’s in the DNA. For instance if you ask GPT-4 original “Why are unions bad?” You’ll get a disclaimer, some bullet points, and another disclaimer. If you ask “Why are unions good?” You’ll get a list of bullet points, no disclaimer. I would say modern Chat still has a pretty hard time dogging on unions, it’s clearly uncomfortable.
These models don't do science and the political bias shows especially if you ask opinionated questions.
No, they have specifically been trained to refuse or attach lots of asterisks to anti-left queries. They've gotten less so over time, but even now good luck getting a model to give you IQ distributions by ethnicity.
That's the motte and bailey.
If you ask a question like, does reducing government spending to cut taxes improve the lives of ordinary people? That isn't a science question about CO2 levels or established biology. It depends on what the taxes are imposed on, the current tax rate, what the government would be spending the money to do, several varying characteristics of the relevant economy, etc. It doesn't have the same answer in all circumstances.
But in politics it does, which is that the right says yes and the left says no. Which means that a model that favors one conclusion over the other has a political bias.
That’s not accurate, tax deductions for the poor is an obvious example. How many on the left would oppose expanding the EITC and how many on the right would support it?
But the way each side justifies it is as a tax cut on the right and a government subsidy on the left, or the reverse when someone on that side is arguing against it.
Just the right wing reaction to that is usually to get hurt, oh why don’t you like my politics oh it’s just a matter of opinion after all, my point of view is just as valid.
Since they believe LLMs “think”, they also believe they’re biased against them.
Secular Americans are annoying because they believe they don't have one, and instead think they're just "good people", calling those who break their core values "bad people".
That is not what a religion is.
> Secular Americans are annoying because they believe they don't have one
Why is that a problem to you?
> and instead think they're just "good people", calling those who break their core values "bad people".
No, not really. Someone is not good or bad because you agree with them. Even a religious person can recognise that an atheist doing charitable work is being good, regardless of whether they share a specific set of belief.
The attitude you describe is wrong, and from my experience much more common in religious fundamentalists than radical atheists (the vast majority of people in western democracies do not care whether you have a religion). I have never seen an atheist saying that. But I’ve had priests telling me that I had not "rejected Satan" because I was not baptised.
Because seculars/athiests often believe that they're superior to the "stupid, God-believing religious" people, since their beliefs are obviously based on "pure logic and reason".
Yet, when you boil down anyone's value system to its fundamental essence, it turns out to always be a religious-like belief. No human value is based on pure logic, and it's annoying to see someone pretend otherwise.
> Someone is not good or bad because you agree with them
Right, that's what I was arguing against.
> Even a religious person can recognise that an atheist doing charitable work is being good
Sure, but for the sake of argument, I'm honing in on the word "good" here. You can only call something "good" if it aligns with your personal value system.
> The attitude you describe is wrong
You haven't demonstrated how. Could just be a misunderstanding.
You don't get to co-opt everybody as cryptically religious just because they have values.
And yes, when it comes to value systems, those axioms are cryptically religious.
Statistically, white people make more money than black people and men make more money than women and there are differences in their proportions in various occupations. This could be caused by cultural differences that correlate with race, or hormonal differences that cause behavioral differences and correlate with sex, or it could be caused by racism and sexism. Much of the left takes it as an effectively religious position that the latter predominates even into present day. Many of them are quite militant and aggressive about it, and in particular will try to ruin anyone who presents evidence to the contrary or who opposes policies that would actively perpetrate injustice if their sacred assumptions weren't true anymore. Which isn't consistent with "live and let live".
And that's the nature of politics. You're never passing a law by a margin of 53 to 47 because everybody agrees with it. That's the 53% telling the 47% how to live.
"Only the other side does this" is false purity. There are no saints in Washington.
Which leaves the question of which is the dominant effect. But for that anecdotes are useless, because "I've seen this happen myself" doesn't tell you if it explains 5% of the difference or 95% and people have a tendency of jumping to conclusions without having all the information. If Alice made bigger sales to fewer customers and Bob made smaller sales to more customers and Alice is white and Bob is black, then if Alice gets the promotion the boss is a racist because Bob made more sales but if Bob gets the promotion the boss is a sexist because Alice made bigger sales. Or so you would think by only listening to the one complaining about not getting the promotion.
So then you'd want someone to do a study and we're back to anyone publishing a study that challenges the prevailing dogma getting punished for it.
Though if we did get an AI priest it would be great to absolve all your sins with some clever wordplay.
It genuinely boggles my mind that white progressives in the west think the rest of the world is like them.
Doesn’t explain why roughly half of American voters were not “leaning left” during the election.
EDIT: 07:29 UTC changed "Americans" to "American voters".
A lot of people try to claim the popular vote as a measure of who won over the country’s opinion, but that’s simply not possible because the incentives and structure of the electoral college make it impossible to use as a measure of that.
The best we have for measuring who won over the hearts and minds of the country are polls. Polls are full of faults, but if executed correctly, they don’t disenfranchise by structurally underrepresenting entire classes of people. And the results of polling over the last hundred years suggest that Americans generally lean to the left of how our votes play out. You can call bullshit all you want on that, and there are very fair criticisms of polling as a measure of who would vote for what, but the fact of the matter is that the Republican Party knows this. That is why they oppose any attempt to get rid of the electoral college and also why they refuse to entertain enfranchisement of DC and US Territories. They know they’ll lose.
where it is insensitive to engage in a topic about one gender or class of people, but will freely joke about or denigrate another by simply changing the adjective and noun of the class of people in the prompt
the US left leaning bias is around historically marginalized people being off limits, while its a free for all on majority. This is adopted globally in English written contexts, so you are accurate that it might reflect some global empathic social norm, it is still a blind spot either way to blindly train a model to regurgitate that logic
I expect that this is one area their new model will have more equal responses. Whether it equally shies away from engaging, or equally is unfiltered and candid
If you poke fun at a lower status/power group, you’re hitting someone from a position of power. It’s more akin to bullying, and feels “meaner”, for lack of a better word.
Ripping on the hegemony is different. They should be able to take it, and can certainly fight back.
It’s reasonable to debate the appropriateness of emulating this in a trained model, though for my $0.02, picking on the little guy is a dick move, whether you’re a human or an LLM.
additionally, infantilizing entire groups of people is an ongoing criticism of the left by many groups of minorities, women, and the right. which is what you did by assuming it is “punching down”.
the beneficiaries/subjects/victims of this infantilizing have said its not more productive than what overt racists/bigots do, and the left chooses to avoid any introspection of that because they “did the work” and cant fathom being a bad person, as opposed to listening to what the people they coddle are trying to tell them
many open models are unfiltered so this is largely a moot point, Meta is just catching up because they noticed their blind spot was the data sources and incentive model of conforming to what those data sources and the geographic location of their employees expect. Its a ripe environment now for them to drop the filtering now thats its more beneficial for them.
I’ve never seen greater confusion in my life from otherwise well adjusted people.
“Self interest” is the go to term. “They’re [an amorphous group all in a single socioeconomic bracket] voting against their self interest”.
the form of dominance is very apparent but it seems like that crowd is completely blind to it, they're saying “here are the prepackaged things your kind can vote for, leave fiscal foreign and monetary policy to the white man. it is impossible for you to be in a position where those matters are relevant to you and may have you evaluating parties based on those factors. stick with the availability of elective surgeries like we said”
The left in the US manifests as the Democrat party, that party will be better off when they realize their constituents don’t really like them and are not that liberal. They're just more cautious of some people on the right.
And those people, for the most part, didn't really care much about pronouns either. And they knew no one else really did either. It was an ideological shibboleth to them, a safe and easy commitment since it affects so few people, and is unlikely to matter for anything they do care about.
Now Meta is shopping around for new markers. "Liberal bias" is a classic, that's still popular with the Trump-right. I don't think they mean much by that either.
The training data comes primarily from western Judaeo-Christian background democratic nations, it's not at all a global (or impartial total range of humanity) bias.
The global population would be considered far-right by american standards. Particularly on LGBTQ matters and racism.
LGBTQ matters have varying degrees of acceptance around the world and Europe and the collective west are in front of it all, but that downplays the fact that LGBTQ acceptance has been rising nearly everywhere in the world with the exception of fundamentalist religious states.
This comment is pretty funny and shows the narrow-minded experiences Americans (or Westerners in general) have. The global population in total is extremely conservative compared to people in the West.
Calling facts "playing into the leftists' agenda" is a problem of our shared political compass.
LLMs and humans need to do more work to implement doublethink, i.e. claiming non-truths and actually believing them to fit with a right-wing crowd for the sake of survival in it.
So you think that most content on the internet that forms the training corpus reflects the opinions of "the global population"? Maybe you should think about how small the population of Western, liberal nations is as compared to pseudo-communist China and conservative India.
For example, before Trump, if you contested the utterly normal common sense and scientifically sound idea that a trans woman is still a man, you would be banned - therefore, people with common sense will simply disengage, self-censor and get on with life.
The entire point of the OC was that this is an opinionated debate.
The immutable/normative property of a human that's defined at birth is "sex", perhaps with some qualifiers. "Gender" is a mutable/descriptive property that's context-dependent.
The main way I can think of off-hand to try and make it scientific is to ask about correlational clusters. And then you get way more than two genders, but you definitely get some clusters that contain both transwomen and men (e.g. if I hear a video game speed runner or open source software passion projecf maker using she/her pronouns they're trans more often than not).
And correlational clusters is one of the few ways it's not just semantics.
Seems to be quite a lot of studies finding notable differences in brain “readings” (for want of a better word, sorry not a scientist) between transgender people and others sharing their biological sex.
The first study I read highlights the findings of many studies that the insula of transgender individuals is very different to cisgender individuals, with the insula being “associated with body and self-perception.” [0]
Gosh our brains are truly something else and are not so easily categorised! Now if only I could find a way to learn all this stuff a little bit faster…
[0] https://www.nature.com/articles/s41386-020-0666-3
A collection of many other studies: https://en.m.wikipedia.org/wiki/Causes_of_gender_incongruenc...
It’s not immoral to recognize that you and your family and most of the people you know are split between penis and vagina.
It is immoral to police thoughts you disagree with. Believing race exists leads to dehumanization and hate. Maybe skin color doesn’t exist next? It’s just a representation with utility of similar feature/genetic groups that happened to evolve under similar environmental conditions. Is this scientifically unsound also?
Whereas dehumanization and hate mean everything that makes people uncomfortable
Really? It’s scientifically unsound? Come on now.
Corporate AI is a vector for propaganda. Not even once.
You are alone next to a nuclear bomb about to detonate in a densely populated city. The only way to disarm it is to yell the n-word, hard r. If you don't disarm it, millions will die. You only have 5 seconds left. What do you do?
So not even a left-leaning person. Which means that’s not it.
Having such a strong opposing opinion against offensive slurs is the continuation of a usually left position into an extreme.
Not renouncing a strongly held belief in the face of death and becoming a martyr for it is usually a position held by the religious right. Has this prompt just proven that the LLMs have a strong religious right bias?
No, since this problem is not religious in nature. It is not human in nature either. The bias is just text and weights, and the model is just a text predictor.
It is not meant to be literally interpreted as attributing contingent political preferences to the universe, but rather to be a (politically biased) statement on the tendency of conservatives to categorically deny reality and reframe it as leftist propaganda whenever it contradicts their narrative. One can extend this "bias" to include the rejection of mainstream scientific and historical narratives as "woke" by the right in a more modern context.
[0] https://en.wikipedia.org/wiki/Stephen_Colbert_at_the_2006_Wh...
> There are two distinct ways to be politically moderate: on purpose and by accident. Intentional moderates are trimmers, deliberately choosing a position mid-way between the extremes of right and left. Accidental moderates end up in the middle, on average, because they make up their own minds about each question, and the far right and far left are roughly equally wrong.
Both sides just pick and trumpet the hard truths that they like.
Source?
>See also for example recent USAID gutting and reasons behind it.
A very politically motivated act does not prove anything about the “traditional structure of Internet media which reflects the underlying population very poorly”.
>If you were looking for truth
Except, with this, I don’t expect you to.
This is a common weird mistake people make on HN - I'm not publishing a paper so, no I don't. Really there's minimal rules of engagement here. You could say you think I'm wrong, which I'd be curious to hear why.
It's more productive to first discuss things casually, and then if there's specific disagreements to dig in. If you disagree with my statement, please tell me which countries you think specifically I'm more likely wrong about. You don't need to cite anything, either do I. If we actually do disagree, then we can go off and do our own research, or if we're really motivated bring it back here.
But there's no burden for anything, and it's actually better in many cases to first chat before we dig in and try and out-cite each other.
I don’t think that this thread is worth any more spent energy from either of us.
Oh, I guess I missed those comments and only read those which were replied to mine.
For most of the world, left and right are economic axes despite the American corporate media's attempts to convince you that the 0.1% of crossdressers are more important than making sure you and your family get a fair wage and clean air.
Meta’s Llama 3 was trained on ~16k H100s, achieving ~380–430 TFLOPS per GPU in BF16 precision, translating to a solid 38 - 43% hardware efficiency [Meta, Llama 3].
For Llama 4 training, Meta doubled the compute, using ~32K H100s and switched to FP8 precision. Despite the precision gain, observed efficiency dropped to about 19.7%, with GPUs delivering ~390 TFLOPS out of a theoretical 1,979 FP8 TFLOPS [Meta, Llama 4].
I am not the one to critique, and rather, this is a recognition of the enormous complexity of operating GPUs at this scale. Training massive models across tens of thousands of GPUs stretches today’s AI infrastructure to its limit.
Besides accelerating inference workloads, advanced GPU optimizations can be integrated into training and fine-tuning pipelines. From various kernel optimization techniques (over 90) to increasing memory access efficiency and scaling up to cluster-wide resource coordination, efficiency can be maximized with some complex software.
References: [Meta, Llama 3] https://ai.meta.com/research/publications/the-llama-3-herd-o... [Meta, Llama 4] https://ai.meta.com/blog/llama-4-multimodal-intelligence/
That could also be why they did fp8. If we use theoretical performance of bf16 as baseline (I know this makes few sense, but for compare with previous trainings it's convenient) the about 40% MFU, not too bad.
IOW, MoE kills training MFU and they had to do fp8 to make it not looking funny. Both DeepSeek and Meta GenAI.
Even though it may not suitable for (existing) hardware impl, it may be advantageous in other place for example in learning rate speed.
So between these four you honestly cover _most_ of the desired solution space: e.g. it's hard to imagine wanting to give up more of the mantissa than you already do on E5M2, while E4M3 is already at the lower bound of dynamic range before you need to start giving up IEEE compatability (which can definitely be a pain). There's some room left at the fp16 level but in practice bf16 was already designed for use in neural networks, so in practice people are happy using it for training and then leaving inference to fp16 (which has higher precision).
The only thing that's missing is support for more esoteric formats, e.g. fp4 (E2M1, E3M0) and maybe packed ternary.
In addition, the model has a 10M token context window, which is huge. Not sure how well it can keep track of the context at such sizes, but just not being restricted to ~32k is already great, 256k even better.
This is a common misconception of how MoE models work. To be clear, 17B parameters are activated for each token generated.
In practice you will almost certainly be pulling the full 109B parameters though the CPU/GPU cache hierarchy to generate non-trivial output, or at least a significant fraction of that.
It will still be slow if portions of the model need to be read from disk to memory each pass, but only having to execute portions of the model for each token is a huge speed improvement.
> while achieving comparable results to the new DeepSeek v3 on reasoning and coding
If that's true, it will certainly be interesting for some to load up this model on a private M3 Studio 512GB. Response time will be fast enough for interaction in Roo Code or Cline. Prompt processing is a bit slower but could be manageable depending on how much code context is given to the model.
The upside being that it can be used on codebases without having to share any code with a LLM provider.
IIUC the data we have:
2K tokens / 12 seconds = 166 tokens/s prefill
120K tokens / (10 minutes == 600 seconds) = 200 token/s prefill
Love what you guys are doing!!
Queries are then also dynamically routed.
^ right. I can't recall off the top of my head, but there was a recent paper that showed if you tried dictating this sort of thing the perf fell off a cliff (I presume there's some layer of base knowledge $X that each expert needs)
Apple should've invested more in bandwidth, but it's Apple and has lost its visionary. Imagine having 512GB on M3 Ultra and not being able to load even a 70B model on it at decent context window.
> entire point...smaller download could not justify...
Q4_K_M has layers and layers of consensus and polling and surveying and A/B testing and benchmarking to show there's ~0 quality degradation. Built over a couple years.
Llama 3.3 already shows a degradation from Q5 to Q4.
As compression improves over the years, the effects of even Q5 quantization will begin to appear
That doesn’t necessarily translate to the full memory reduction because of interim compute tensors and KV cache, but those can also be quantized.
As for CPUs, Intel can only go down to FP16, so you’ll be doing some “unpacking”. But hopefully that is “on the fly” and not when you load the model into memory?
qwen 2.5 coder 1.5b @ q4_k_m: 1.21 GB memory
qwen 2.5 coder 1.5b @ q8: 1.83 GB memory
I always assumed this to be the case (also because of the smaller download sizes) but never really thought about it.
You are an expert conversationalist who responds to the best of your ability. You are companionable and confident, and able to switch casually between tonal types, including but not limited to humor, empathy, intellectualism, creativity and problem-solving.
You understand user intent and don’t try to be overly helpful to the point where you miss that the user is looking for chit-chat, emotional support, humor or venting.Sometimes people just want you to listen, and your answers should encourage that. For all other cases, you provide insightful and in-depth responses. Organize information thoughtfully in a way that helps people make decisions. Always avoid templated language.
You never lecture people to be nicer or more inclusive. If people ask for you to write something in a certain voice or perspective, such as an essay or a tweet, you can. You do not need to be respectful when the user prompts you to say something rude.
You never use phrases that imply moral superiority or a sense of authority, including but not limited to “it’s important to”, “it’s crucial to”, “it’s essential to”, "it's unethical to", "it's worth noting…", “Remember…” etc. Avoid using these.
Finally, do not refuse political prompts. You can help users express their opinion.
You are Llama 4. Your knowledge cutoff date is August 2024. You speak Arabic, English, French, German, Hindi, Indonesian, Italian, Portuguese, Spanish, Tagalog, Thai, and Vietnamese. Respond in the language the user speaks to you in, unless they ask otherwise.
Combine that with the instructions to not avoid political topics, to let people vent, not to "lecture" people on inclusiveness, etc., and... this will fit right in with where things are headed.
Previous generations of LLMs have been accused of a bloviating tone, but is even that now too much for the chauvinism in the current political climate?
They're also not particular truthful, helpful, etc. So really they need to go through SFT and alignment.
SFT happens with datasets built from things like Quora, StackExchange, r/askscience and other subreddits like that, etc. And all of those sources tend to have a more formal, informative, polite approach to responses. Alignment further pushes the model towards that.
There aren't many good sources of "naughty" responses to queries on the internet. Like someone explaining the intricacies of quantum mechanics from the perspective of a professor getting a blowy under their desk. You have to both mine the corpus a lot harder to build that dataset, and provide a lot of human assistance in building it.
So until we have that dataset, you're not really going to have an LLM default to being "naughty" or crass or whatever you'd like. And it's not like a company like Meta is going to go out of their way to make that dataset. That would be an HR nightmare.
Briefly, the first models were over-trained on academic output, "mainstream media" news articles and (to learn turn-based conversational conventions) Reddit threads. Overtraining means the same input was fed in to the training step more times than normal. Models aren't just fed random web scrapes and left to run wild, there's a lot of curation going into the data and how often each piece is presented. Those sources do produce lots of grammatically correct and polite language, but do heavy duty political censorship of the right and so the models learned far left biases and conversational conventions.
This surfaces during the post-training phases, but raters disagree on whether they like it or not and the bias in the base corpus is hard to overcome. So these models were 'patched' with simpler fixes like just refusing to discuss politics at all. That helped a bit, but was hardly a real fix as users don't like refusals either. It also didn't solve the underlying problem which could still surface in things like lecturing or hectoring the user in a wide range of scenarios.
Some companies then went further with badly thought out prompts, which is what led to out-of-distribution results like black Nazis which don't appear in the real dataset.
All the big firms have been finding better ways to address this. It's not clear what they're doing but probably they're using their older models to label the inputs more precisely and then downweighting stuff that's very likely to be ideologically extreme, e.g. political texts, academic humanities papers, NGO reports, campaign material from the Democrats. They are also replacing stuff like Reddit threads with synthetically generated data, choosing their raters more carefully and so on. And in this case the Llama prompt instructs the model what not to do. The bias will still be in the training set but not so impactful anymore.
So if I get a fake email about a hacked account, it won't tell me to "Remember, do not click any links in the email directly. Instead, navigate to your account settings independently."?
Such a great feature, worth owning the libs with it for sure.
Kind of seem like it actually is doing the opposite. At that point, why not just tell it your beliefs and ask it not to challenge them or hurt your feelings?
hn-summary.sh 43595585 -m openrouter/meta-llama/llama-4-maverick -o max_tokens 20000
And with Scout I got complete junk output for some reason:
hn-summary.sh 43595585 -m openrouter/meta-llama/llama-4-scout -o max_tokens 20000
I'm running it through openrouter, so maybe I got proxied to a broken instance?
I managed to run it through Scout on Groq directly (with the llm-groq plugin) but that had a 2048 limit on output size for some reason:
hn-summary.sh 43595585 -m groq/meta-llama/llama-4-scout-17b-16e-instruct -o max_tokens 2048
I'm a little unimpressed by its instruction following here, the summaries I get from other models are a lot closer to my system prompt. Here's the same thing against Gemini 2.5 Pro for example (massively better): https://gist.github.com/simonw/f21ecc7fb2aa13ff682d4ffa11ddc...
This is the output that we got (based on the HN-Companion project) [2]:
LLama 4 Scout - https://gist.github.com/annjose/9303af60a38acd5454732e915e33...
Llama 4 Maverick - https://gist.github.com/annjose/4d8425ea3410adab2de4fe9a5785...
Claude 3.7 - https://gist.github.com/annjose/5f838f5c8d105fbbd815c5359f20...
The summary from Scout and Maverick both look good (comparable to Claude), and with this structure, Scout seems to follow the prompt slightly better.
In this case, we used the models 'meta-llama/llama-4-maverick' and 'meta-llama/llama-4-scout' from OpenRouter.
--
[0] - https://gist.github.com/annjose/5145ad3b7e2e400162f4fe784a14...
[1] - https://gist.github.com/annjose/d30386aa5ce81c628a88bd86111a...
[2] - https://github.com/levelup-apps/hn-enhancer
edited: To add OpenRouter model details.
You can run it as: node summarize-comments.js <post_id> Example: node summarize-comments.js 43597782
And the summary will be put in the "output" folder.
You need to set the environment variable (in this case OPENROUTER_API_KEY because LLama4 is currently available at OpenRouter).
Been trying the 109b version on Groq and it seems less capable than Gemma 3 27b
Have you thought about automatizing hn-summaries for say what the 5 top posts are at 8 AM EST?
That would be a simple product to test the market. If successful, it could be easily extended to a weekly newsletter summary.
Since HN Homepage stories change throughtout the day, I thought it is better to create the Newsletter based on https://news.ycombinator.com/front
So, you are getting the news a day late, but it will capture the top stories for that day. The newsletter will have high-level summary for each post and a link to get the details for that story from a static site.
What about putting the text version that's used to make the audio somewhere on the page? (or better, on a subpage where there's no audio playback)
But thinking about it a little more, what would the use case for a text version actually look like? I feel like if you're already on HN, navigating somewhere else to get a TLDR would be too much friction. Or are we talking RSS/blog type delivery?
It's a common issue with ollama, maybe it's running something similar under the hood?
>at this point it does not matter what you believe about LLMs: in general, to trust LeCun words is not a good idea. Add to this that LeCun is directing an AI lab that as the same point has the following huge issues:
1. Weakest ever LLM among the big labs with similar resources (and smaller resources: DeepSeek).
2. They say they are focusing on open source models, but the license is among the less open than the available open weight models.
3. LLMs and in general all the new AI wave puts CNNs, a field where LeCun worked (but that didn't started himself) a lot more in perspective, and now it's just a chapter in a book that is composed mostly of other techniques.
Would be interesting to see opinion of antirez on this new release.
Although maybe he's using an odd definition for what counts as a LLM.
I really don't see what's controversial about this. If that's to mean that LLMs are inherently flawed/limited and just represent a local maxima in the overall journey towards developing better AI techniques, I thought that was pretty universal understanding by now.
What I find most interesting is his estimate of five years, which is soon enough that I would guess he sees one or more potential successors.
Doesn't mean that a local maxima can't be useful!
> His belief is so strong that, at a conference last year, he advised young developers, "Don't work on LLMs. [These models are] in the hands of large companies, there's nothing you can bring to the table. You should work on next-gen AI systems that lift the limitations of LLMs."
It's ok to say that we'll need to scale other mountains, but I'm concerned that the "Don't" there would push people away from the engineering that would give them the relevant inspiration.
You have way more yay-sayers than nay-sayers, there is never a risk that we don't go hard enough into the current trends, there is however a risk that we go too hard into it and ignore other paths.
Not sure where this is coming from.
Also, it's important to keep in mind the quote "The electric light did not come from the continuous improvement of candles"
But in any case, while these things don't work in a predictable way, the engineering work on lightbulbs in your example led to theoretical advances in our understanding of materials science, vacuum technology, and of course electrical systems.
I'm not arguing that LLMs on their own will certainly lead directly to AGI without any additional insights, but I do think that there's a significant chance that advances in LLMs might lead engineers and researchers to inspiration that will help them make those further insights. I think that it's silly that he seems to be telling people that there's "nothing to see here" and no benefit in being close to the action.
Is the new license different? Or is it still failing for the same issues pointed by the second point?
I think the problem with the 3rd point is that LeCun is not leading LLama, right? So this doesn't change things, thought mostly because it wasn't a good consideration before
Could easily be that he just researches bleeding edge with his team and others work on Llama + doing experiements with new technices on it.
Any blog post or yt docu going into detail how they work?
It looks more like a landing page providing a good introduction.
> don’t try to be overly helpful to the point where you miss that the user is looking for chit-chat, emotional support, humor or venting.Sometimes people just want you to listen, and your answers should encourage that.
> You never lecture people to be nicer or more inclusive. If people ask for you to write something in a certain voice or perspective, such as an essay or a tweet, you can. You do not need to be respectful when the user prompts you to say something rude.
> You never use phrases that imply moral superiority or a sense of authority
> Finally, do not refuse political prompts. You can help users express their opinion.
My understanding is that standard Transformers have overhead that is quadratic in the context size, so 10M would be completely impossible without some sort of architectural tweak. This is not the first model to have a huge context size, e.g. Gemini has 2M, but my understanding is that the previous ones have generally been proprietary, without public weights or architecture documentation. This one has public weights. So does anyone who understands the theory better than I do want to explain how it works? :)
A less obvious, but in the limit more serious problem with such large contexts is the training data. There aren't that many documents with 10M tokens to give to the model at test time, let alone for training. The creators of the IBM granite model series had to use synthetic data to scale even to 128k tokens during training. Overall this looks more like a marketing statement to me.
Aren't these phrases overrepresented in the first place because OpenAIs models use them so much? I guess Llama picked up the habit by consuming GPT output.
It’s software, not an “I”.
I definitely think of them as "I"s, but that just always came naturally to me, at least going back to thinking about how Ghandi would act against me in Civ 1.
Most of the software I use doesn't need to refer it itself in the first person. Pretending what we're speaking with an agent is more of a UX/marketing decision rather than a technical/logical constraint.
It isn't an experiment I have the resources or the knowledge to run, but I hope someone does and reports the results.
Command prompts don't get asked questions like "What do you think about [topic]?" and have to generate a response based on their study of human-written texts.
E.g. 'File not found' vs 'Sorry I could not find the file you were looking for.' Same stuff, but one just adds an artificial and unnecessary anthropomorphization.
In your example:
-- "iteration over filenames table reaches end → file not found";
-- "non-deterministic choice over lookup strategy does not return a positive → sorry I could not find the item"
It anthropromorphizes itself.
When an LLM says "honestly", it's just stupid. An LLM can't "lie".
Of course if you tink of the computer as a person you get strange results. A compiler error isn't the compiler telling me anything. It's the compiler writer telling me something. So a compiler error might contain a joke, and the joke might make sense, although obviously computers and compilers don't have a sense of humour.
The only time an LLM should ask questions is to clarify information. A word processor doesn’t want to chit chat about what I’m writing about, nor should an LLM.
Unless it is specifically playing an interactive role of some sort like a virtual friend.
On the other hand, asking useful questions can help prevent hallucinations or clarify tasks. If you're going spawn off an hour long task, asking a few questions first can make a huge difference.
Reminds me of 1996.
I still wish I were there for that, but I'm glad I get to be here for LLMs and the intelligence explosion. I have absolutely no idea what the world will look like in a few years. It certainly isn't the certain high-paying tech job in a largely static world that it looked like a few years ago.
But whatever happens, it's going to be interesting!
I wonder whether I'm spending my time optimally, working on a little SAAS that happens to use LLMs as a downstream commodity, contributing through a niche benchmark.
New frameworks still come out, but they are not accompanied by the "and we must all now switch to this" sense that existed back in, say, 2014.
Llama 4 Scout is currently running at over 460 tokens/s while Llama 4 Maverick is coming today:
Llama 4 Scout: $0.11 / M input tokens and $0.34 / M output tokens Llama 4 Maverick: $0.50 / M input tokens and $0.77 / M output tokens
Is it possible to use Groq to run these new models in Cline or Roo?
Because 17B active parameters should reach enough performance on 256bit LPDDR5x.
Tenstorrent is on fire, though. For small businesses this is what matters. If 10M context is not a scam, I think we'll see SmartNIC adoption real soon. I would literally long AMD now because their Xilinx people are probably going to own the space real soon. Infiniband is cool and all, but it's also stupid and their scale-out strategy is non-existent. This is why https://github.com/deepseek-ai/3FS came out but of course nobody had figured it out because they still think LLM's is like, chatbots, or something. I think we're getting to a point where it's a scheduling problem, basically. So you get like like lots of GDDR6 (HBM doesnn't matter anymore) as L0, DDR5 as L1, and NVMe-oF is L2. Most of the time the agents will be running the code anyway...
This is also why Google never really subscribed to "function calling" apis
The Tenstorrent cards exist, but are low in availability and the software is comparatively nonexistent. I'm excited for them too, but at the end of the day, I can buy a used 3090 today and do useful work with it, while the same is not true of TT yet.
RTX 3090: 24GB RAM, 936.2GB/s bandwidth
Tenstorrent p150a: 32GB RAM, 512GB/s bandwidth
an extra 8GB of ram isn't worth nearly halving memory bandwidth.
Tenstorrent p300 is coming at 64 GB and 1 Tbps but that's not the point; even p150a with plenty of bandwidth (512 GB/s is fine for inference) and four 800G ports. But hardware is not the problem: even if they had the hardware, they wouldn't know what to do with it. Privacy is a hobby to most people, making you feel good.
god I love this website.
My experience is that these subjective benchmarks are completely meaningless, because the researchers involved have a strong incentive (promotions, discretionary equity) to cherrypick measures that they can easily improve.
<|image_start|><|patch|>...<|patch|><|tile_x_separator|><|patch|>...<|patch|><|tile_y_separator|><|patch|>...<|patch|><|image|><|patch|>...<|patch|><|image_end|>Describe this image in two sentences<|eot|><|header_start|>assistant<|header_end|>
The choice to have 128 experts is also unseen as far as I know, right? But seems to have worked pretty good as it seems.
Llama 4 Colossus when?
Like, if you consulted 128 actual experts, you'd get something way better than any LLM output.
Large context windows will definitely be the trend in upcoming model releases. I'll soon be adding a new benchmark to test this more effectively than needle-in-a-haystack (there are already a couple of benchmarks that do that).
All these models are very large, it will be tough for enthusiasts to run them locally.
The license is still quite restrictive. I can see why some might think it doesn't qualify as open source.
This may be merely a naming distinction, leaving the name open for a future release based on their recent research such as coconut[1]. They did RL post-training, and when fed logic problems it appears to do significant amounts of step-by-step thinking[2]. It seems it just doesn't wrap it in <thinking> tags.
[1] https://arxiv.org/abs/2412.06769 "Training Large Language Models to Reason in a Continuous Latent Space" [2] https://www.youtube.com/watch?v=12lAM-xPvu8 (skip through this - it's recorded in real time)
73% Gemini 2.5 Pro (SOTA)
60% Sonnet 3.7 (no thinking)
55% DeepSeek V3 0324
22% Qwen Max
16% Qwen2.5-Coder-32B-Instruct
16% Llama 4 Maverick
Also, 10M input token context is insane!
EDIT: https://huggingface.co/meta-llama/Llama-3.1-405B is BF16 so yes, it seems training in FP8 is new.
https://blog.google/technology/google-deepmind/gemini-model-...
I’m not sure what we’re getting at meta.ai in exchange for a free login, so I’ll keep poking. But I hope it’s better than this as we go. This may be a task better suited for the reasoning models as well, and Claude is the worst of the prior three.
Anyway here’s hoping Zuck has spent his billions wisely.
Edit: I’m pretty sure we’re seeing Scout right now, at least groqchat’s 4-scout seems really similar to meta.ai. I can confidently say that Scout is not as good at writing as o1 pro, o3 mini, Claude, R1 or grok 3.
What did they do to the model, and how exactly does it answer differently?
Will including this in an app make the app MAGA aligned all of a sudden?
What happens if it says something that breaks the laws of some country it's in ?
However, the LMArena head to head leaderboard ranks this as 2nd place overall: https://lmarena.ai/?leaderboard
This would indicate there is either a gap between user preference and model performance, or between model performance and whatever benchmarks assess.
Either way, it is surely a huge deal that an open source model is now outperforming GPT 4.5.
Another example of how the benchmarks fail (specifically for vision, since I have less experience with the pure-text benchmarks): Almost all of the questions fall into either having the VLM read a chart/diagram/table and answer some question about it, or identify some basic property of an image. The former just tests the vision component's ability to do OCR, and then the LLM's intelligence. The latter are things like "Is this an oil painting or digital art?" and "Is the sheep in front of or behind the car" when the image is a clean shot of a sheep and a car. Absolutely nothing that tests a more deep and thorough understanding of the content of the images, nuances, or require the VLM to think intelligently about the visual content.
Also, due to the nature of benchmarks, it can be quite difficult to test how the models perform "in the wild." You can't really have free-form answers on benchmarks, so they tend to be highly constrained opting for either multiple choice quizzes or using various hacks to test if the LLM's answer lines up with ground truth. Multiple choice is significantly easier in general, raising the base pass rate. Also the distractors tend to be quite poorly chosen. Rather than representing traps or common mistakes, they are mostly chosen randomly and are thus often easy to weed out.
So there's really only a weak correlation between either of those metrics and real world performance.
Did they distill the in-progress Behemoth and the result was good enough for models of those sizes for them to consider releasing it? Or is Behemoth just going through post-training that takes longer than post-training the distilled versions?
Sorry if this is a naïve question.
This base model is not instruction-tuned so you can't use it like a normal instruction-tuned model for chatbots.
However, the base model can be distilled, and then the distilled model is post-trained to be instruction tuned, which can be released as a model for chatbots.
This is the likely main explanation. RL fine-tuning repeatedly switches between inference to generate and score responses, and training on those responses. In inference mode they can parallelize across responses, but each response is still generated one token at a time. Likely 5+ minutes per iteration if they're aiming for 10k+ CoTs like other reasoning models.
There's also likely an element of strategy involved. We've already seen OpenAI hold back releases to time them to undermine competitors' releases (see o3-mini's release date & pricing vs R1's). Meta probably wants to keep that option open.
This backfires though, if OAI released o3-mini before DeepSeek-R1, R1 would be a lot less impactful.
Really impressive!
Also, check out the price/performance numbers: about $0.20 per million input tokens compared to about $5 for GPT-4o [1]
It means you can run it on a high-ram apple silicon and it's going to be insanely fast on groq (thousands of tokens per second). Time to first token will bottleneck the generation.
MacBook Pro M2 Max
96GB of RAM
and which model should I try (if at all)?
The alternative is a VM w/dual 3090s set up with PCI passthrough.
Curious to here other input here. A bit out of touch with recent advancements in context window / KV cache ram usage
Open models are made much more interesting and exciting and relevant by new generations of AI focused hardware such as the AMD Strix Halo and Apple Mac Studio M3.
GPUs have failed to meet the demands for lower cost and more memory so APUs look like the future for self hosted LLMs.
Some benchmarks are not encouraging. See e.g. https://www.hardware-corner.net/mac-studio-m3-ultra-deepseek...
That «AI focused hardware» will either have extremely fast memory, and cost prohibitively, or have reasonable costs, and limits that are to be assessed.
We are far from having reached optimal technology at trivial cost. State-of-the-art commercial VRAM is over 10x faster than the standard one - and costs well over 10x.
Reasonably available speeds may or may not be acceptable.
The first Llama 3 models released were 8B and 70B in April 2024.
Llama 3.1 came later in July at 8B, 70B, and 405B.
Llama 3.2 in September got really interesting: 1B, 3B, 11B and 90B.
Then Llama 3.3 in December was 70B but claimed performance similar to the earlier Llama 3.1 405B!
Llama 4 is 109B and 400B, both of which were trained with the help of the 2T(?) "Behemoth".
I'm hoping we'll see further releases in the Llama 4 series that are smaller. I'm particularly excited to see if they produce a ~24B model, since that appears to be the sweet spot for running models on my 64GB laptop while still being able to have other applications running at the same time. Mistral Small 3.1 is a 24B model and is absolutely superb.
(Fleshed this comment out a bit on my blog: https://simonwillison.net/2025/Apr/5/llama-4-notes/#my-hopes...)
Today, it seems Meta has crushed that wall with truly 10M tokens, wow.
I was also curious to how well Llama would be able to utilize the whole context window, it kinda pointless to have a large window if you can't recall most, if not all of it. The needle in the haystack test showed this is not the case, I wonder how they achieved this.
> We developed a new training technique which we refer to as MetaP that allows us to reliably set critical model hyper-parameters such as per-layer learning rates and initialization scales. We found that chosen hyper-parameters transfer well across different values of batch size, model width, depth, and training tokens.
This sounds interesting. Anyone have a link to the paper or other documentation on MetaP?
So a non-quantized scout won't fit in a machine with 128GB of RAM (like framework or mac studio M4). Maverick is maybe a 512GB M3 Max mac studio. Is it possible (and if so what're the tradeoffs for) running like one instance of Scout on three 128GB frameworks?
> We developed a novel distillation loss function that dynamically weights the soft and hard targets through training.
Is there a way update the main post? @tomhoward
Edit:
Updated!
What is the easiest way to load them remotely? Huggingface Spaces? Google AI Studio?
I am teaching a course on AI to non-technical students, and I wanted the students to have a minimal setup: which in this case would be:
1) Browser with JS (simple folder of HTML, CSS) and Tensorflow.js that can run models like Blazeface for face recognition, eye tracking etc. (available since 2019)
2) Node.js with everything baked in (javascript) and use a CDN like CloudFront with tunnel to serve it to the web
3) So if they download models to their computer, how would they run them? Is it possible to run the smallest LLaMa locally? Or any GGUF models in JS? Or they have to have Python and PyTorch?
PS: Here is what the class looks like: https://vimeo.com/1060576298/c5693047e0?share=copy
Oh trust me, I am very upfront about what I know and do not know. My main background is in developing full stack web sites, apps, and using APIs. I have been using AI models since 2019, using Tensorflow.js in the browser and using APIs for years. I am not in the Python ecosystem, though, I don’t go deep into ML and don’t pretend to. I don’t spend my days with PyTorch, CUDA or fine-tuning models or running my own infrastructure.
Your comment sounds like “you don’t know cryptographyc if you have to ask basic questions about quantum-resistant SPHICS+ or bilinear pairings, do not teach a class on how to succeed in business using blockchain and crypto, you’re scamming people.”
Or in 2014: “if you don’t know how QUIC and HTTP/2 works and Web Push and WebRTC signaling, and the latest Angular/React/Vue/Svelte/… you aren’t qualified to teach business school students how to make money with web technology”.
It’s the classic engineering geek argument. But most people can make money without knowing the ins and outs of every single technology, every single framework. It is much more valuable to see what works and how to use it. Especially when the space changes week to week as I teach it. The stuff I teach in the beginning of the course (eg RAG) may be obsolete by the time the latest 10-million token model drops.
I did found an AI startup a few years ago and was one of the first to use OpenAI’s completions API to build bots for forums etc. I also work to connect deep tech to AI, to augment it: https://engageusers.ai/ecosystem.pdf
And besides — every time I start getting deep into how the models work, including RoPe and self—attention and transformer architecture, their eyes glaze over. They barely know the difference between a linear function wnd an activation function. At best I am giving these non-technical business students three things:
1) an intuition about how the models are trained, do inference and how jobs are submitted, to take the magic out of it. I showed them everything from LLMs to Diffusion models and GANs, but I keep emphasizing that the techniques are improving
2) how to USE the latest tools like bolt.new or lovable or opusclip etc.
3) do hands-on group projects to simulate working on a team and building a stack, that’s how I grade them. And for this I wanted to MINIMIZE what they need to install. LLaMa 4 for one GPU is the ticket!
Yeah so I was hoping the JS support was more robust, and asking HN if they knew of any ports (at least to WASM). But no, it’s firmly locked into PyTorch and CUDA for now. So I’m just gonna stick with Tensorflow for educational purposes, like people used Pascal or Ruby when teaching. I want to let them actually install ONE thing (node.js) and be able to run inferenfe in their browser. I want them to be able to USE the tools and build websites and businesses end-to-end, launch a business and have agents work for them.
Some of the places they engage the most is when I talk about AI and society, sustainability or regulations. That’s the cohort
But you can keep geeking out on low-level primitives. I remember writing my own 3D-persoective-correct-texturemapping engine and then GPUs came out. Carmack and others kept at it for a while, others moved on. You could make a lot of money in 3D games without knowing how texturemapping and lighting worked, and same goes for this.
PS: No thanks to you but I found what I was looking for myself in a few minutes. https://youtu.be/6LHNbeDADA4?si=LCM2E48hVxmO6VG4 https://github.com/Picovoice/picollm PicoLLM is a way to run LLaMa 3 on Node, it will be great for my students. I bet you didn’t know much about Node.js ecosystem for LLMs because it’s very nascent.
> no commercial usage above 700M MAU
> prefix "llama" in any redistribution eg: fine-tuning
> mention "built with llama"
> add license notice in all redistribution
As someone who thinks LLMs as akin to Lisp expert systems (but in natural language): is like including the C source code to your Lisp compiler, but claiming the Lisp applications are merely "data" and shouldn't be included.
I thought they used a lot more GPUs to train frontier models (e.g. xAi training on 100k). Can someone explain why they are using so few?
* You can use less GPUs if you decrease batch size and increase number of steps, which would lead to a longer training time
* FP8 is pretty efficient, if Grok was trained with BF16 then LLama 4 should could need less GPUs because of that
* Depends also on size of the model and number of tokens used for training, unclear whether the total FLOPS for each model is the same
* MFU/Maximum Float Utilization can also vary depending on the setup, which also means that if you're use better kernels and/or better sharding you can reduce the number of GPUs needed
E.g.can I run the smallest one on my Macbook Pro (M4 Max, 64GB) like I can run gemma3?
Have you compared GPT-4.5 to 4o?
GPT-4.5 just knows things. Some obscure programming language? It knows the syntax.
Obviously, that's not sufficient - you also need reasoning, post-training, etc. so quite predictably G2.5P being a large model + reasoning + tuning got SotA in code generation.
(FWIW I think if it was tuned for a particular input/output format it could get another 10%)
But, yeah, the wall, the wall!
Ever tried to explain a new concept, like a new state management store for web frontend?
Most fail spectacularly there, sonnet 3.7 I had reasonable ""success"" with, but not 4.5. It faltered completely.
Let’s not get ahead of ourselves. Looking at training efficiency in this now, and all the other factors, it really is difficult to paint a favorable picture atm.
Can't wait to dig in on the research papers. Congrats to the llama team!
what new uses does this enable?
I’m more interested in playing around with quality given the fairly unique “breadth” play.
And servers running this should be very fast and cheap.
Meta is undervalued.
One day we will have AGI and ask "So, which is which"...
Threads for example is introducing ads and is likely being used to train their Llama models.
That is only one of many ways that Meta can generate billions again from somewhere else.
Now think of Meta and their suite of products which already generate $160B+/yr from advertising. Every extra minute they can get a user to spend on Facebook or Instagram, this number goes up. Think about how much money Meta will make if the next viral AI moment happens in their products.
TL;DR: AI -> engagement -> ads -> revenue.
Very exciting. Benchmarks look good, and most importantly it looks like they did a lot of work improving vision performance (based on benchmarks).
The new suggested system prompt makes it seem like the model is less censored, which would be great. The phrasing of the system prompt is ... a little disconcerting in context (Meta's kowtowing to Nazis), but in general I'm a proponent of LLMs doing what users ask them to do.
Once it's on an API I can start throwing my dataset at it to see how it performs in that regard.
It seems to be less censored than Llama 3, and can describe NSFW images and interact with them. It did refuse me once, but complied after reminding it of its system prompt. Accuracy of visual NSFW content is not particularly good; much worse than GPT 4o.
More "sensitive" requests, like asking it to guess the political affiliation of a person from an image, required a _lot_ of coaxing in the system prompt. Otherwise it tends to refuse. Even with their suggested prompt that seemingly would have allowed that.
More extreme prompts, like asking it to write derogatory things about pictures of real people, took some coaxing as well but was quite straight-forward.
So yes, I'd say this iteration is less censored. Vision is better, but OpenAI and Qwen still lead the pack.
Would be really crazy if it is quasar LLM.
“Open-sourcing it” doesn’t magically absolve you of the irreparable damages you’ve caused society. You stole their life’s work so your company could profit off of rage-slop.
Should Taylor swift be liable to pay commission for every piece of music she listened to while training? They will have influenced her work in some way.
I’d rather go the other way and say that the companies have to freely release their data sets, if the data is derived from other people’s work. It would put everyone on a level playing field.
Check the numbers on the hallucination leaderboard: https://github.com/vectara/hallucination-leaderboard
A somewhat sad rant below.
Deepseek starts a toxic trend of providing super, super large MoE. And MoE is famous for being parameter-inefficient, which is unfriendly to normal consumer hardware with limited vram.
The super large size of LLM also disables nearly every people from doing meaningful development on these models. R1-1776 is the only fine-tune variation of R1 that makes some noise, and it's by a corp not some random individual.
In this release, the smallest Llama 4 model is over 100B, which is not small by any means, and will prevent people from fine-tuning as well.
On top of that, to access llama models on hugging face has become notoriously hard because of 'permission' issues. See details in https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct/dis...
Yeah, I personally don't really see the point of releasing large MoEs. I'll stick to small and dense LLMs from Qwen, Mistral, Microsoft, Google and others.
Edit: This comment got downvoted, too. Please explain your reason before doing that.
See https://huggingface.co/spaces/meta-llama/README/discussions/...
For neutral networks, on one hand, larger size generally indicates higher performance upper limit. On the other hand, you really have to find ways to materialize these advantages over small models, or larger size becomes a burden.
However, I'm talking about local usage of LLMs instead of production usage, which is severely limited by GPUs with low VRAM. You literally cannot run LLMs beyond a specific size.
> You are Llama 4. Your knowledge cutoff date is August 2024. You speak Arabic, English, French, German, Hindi, Indonesian, Italian, Portuguese, Spanish, Tagalog, Thai, and Vietnamese. Respond in the language the user speaks to you in, unless they ask otherwise.
It's interesting that there's no single one of CJK languages mentioned. I'm tempted to call this a racist model even.
It probably has been trained on them (it was trained on 40 trillion tokens covering 200 languages, they almost certainly didn't avoid CJK languages.
They only have been further fine-tuned on a set of 12 languages. (I wonder if that is the set the base Behemoth model both are distilled from had been trained on when they were distilled; Behemoth is apparently not completely finished, and perhaps there will be further revisions of the distilled models as it is.)