- https://ianbarber.blog/blogroll
- https://ianbarber.blog/archive
- https://ianbarber.blog/posts
- none of the above links work
- i really dont want to scroll 200 pages just to see what your blog articles are
When a technique or technology is new people are making massive gains by just applying it to some use case, or gathering more data for training, or giving it more resources.
As time goes on those "bitter lesson" gains start to hit the shallow part of the logistic curve and companies have to start investing more and more effort into engineering for each small, incremental gain.
The known-good thing has been heavily optimized for performance, making it much harder for new technologies to prove that they are better. This is similar to the problem of gas vs electric engines - we had a century of optimization and ecosystem development around gas engines, which creates an uphill battle for electric motors even though they are (eventually) superior on every way /except/ having that massive ecosystem.
The problem isn't as bad here, because software is much more flexible than hardware, and scaling laws give a reasonable way to try things out at smaller scale before going whole hog.
Or maybe this already effectively covered by context caching and the gains would be minimal (stateless, but if you pass in the same context or the same head context, it’s already in GPU memory and doesn’t need to be loaded?).
For example, tiered pricing for cached context relies on state, even if the models don’t.
https://sebastianraschka.com/llm-architecture-gallery/?compa...
If you look at it, the diagrams are very similar, but the main differences are that the feedforward is replaced with a MoE (router to multiple feedforwards) and the model has a different attention implementation.
-- works very well on high-RAM Macs
The entire point of the comparison is that LLMs look vastly different today than before. Comparing more similar LLMs would detract from the point I thought the author was trying to make.
I think the point stands: MoE, a myriad of complex attention approaches, shared layers, you name it. And making it all work together well is a huge trial-and-error pain even for small models, never mind getting to efficient hardware utilization.
The page links to the same site you do. No wonder it is similar -- the source is the same!
The very first sentence
> Back in 2022 and 2023 there were two big branches of machine learning happening at Meta.
is unmistakably human. That's not how a LLM would phrase this sentence, and if it did, it would have put a comma after 2023.
I am a professional writer and have been for over 30 years. (I do not use any form of LLM ever.) This means I read a lot. This also means that I have 30+ years of experience of readers not understanding what I wrote, or not getting further than the title, or not getting the main message, or inverting it in their heads, or inserting their own message and then complaining when I diverge, and an endless list of Ways People Do Not Get It.
I am also a trained TESOL teacher. Ability to capture gist is a skill we test for and measure, and many, maybe the majority, of native speakers don't have it and don't know.
In recent years I constantly see people going "this is written by AI" and I have yet to see a single of of them able to coherently prove their point. It's all just feelings and hunches.
So I am calling you on this:
How do you know? Show your working. Demonstrate your case.
Some days, I spend over 4 hours a day reading walls of text written by Claude. If I couldn't recognize Claude's default "voice" by now, something would be wrong. It would be like a Hemingway fan not being able to recognize Hemingway. Except more so, because Claude's writing style is getting worse from version to version, descending into self parody.
On the statistical side, Pangram's model identifies AI-authored text with a 1-in-5,000 false positive rate, measured against hold-out texts from before 2022. My "ear" also agrees closely with Pangram. If I think something sounds AI written, Pangram virtually always comes back with "AI, confidence: high."
> On top of that, LLM writing is often bad in a very particular way: it's weak on actual things to say, but with an overheated style.
This point is interesting because it raises the question of what "LLM writing" actually is. If it is expanding a smaller prompt into a larger article then yes, by construction the information density is low. But it can also be used to take a semi-coherent stream of consciousness and turn it into something readable and the people using it that way might already have started to slip under the radar.
This is a lot like how the criminals seem especially stupid because the ones who get caught are disproportionately the stupid ones. The easily detectable LLM writers are going to be the lazy ones.
To an extent, true. There are a lot of lazy ones though. And even for those who take steps, it sometimes leaves enough of a trace to at least raise the question.
The end product is something much more polished than anything I'd writ eon my own, but still comes off as being genuinely from me. At least that's what people have told me when I've asked.
Also, FWIW, Pangram scores my writing as entirely human.
Claude's writing isn't easy to identify because it uses em-dashes and bulleted lists. Claude's distinctive style goes much deeper than that.
In other words, correlation != causation
(As of now, that four-word low-effort comment has generated over a thousand words in response, none of which improve this article's discussion.)
Edit: You know how you can recognise someone just from their gait while they walk towards you? I would struggle to describe that for an individual person but it doesn't mean I can't identify them from that alone.
But AI written pieces do have a certain feeling. A sort of saccatto in the succession of ideas that does not feel natural. They emphasize certain points, and you as a reader, you just wonder why is that. There is the “This thing, not just that thing”. There are also the three successive propositions (mostly in one sentences) to accentuate an idea and “Negation. Strong positive idea in the same direction”.
In general try reading one (vocally) to yourself and it will feel really weird.
Everytime someone claims that they have always written like this I grab a pre-2022 post of theirs and five both to a few SOTA chatbots and ask "did the same writer author both these texts".
Thus far I have never gotten a "likely" response.
If the author truly did not use an AI to write something, then it is more likely that theybhave spent so much time conversing with their LLM than reading human authored material that they now sound like an LLM.
This specific article, though, doesn't look anything like LLM output.
PS. Isn't it odd how all LLMs have converged on the same speech patterns, patterns which resemble almost no human authored material outside of high-pressure sales techniques?
You start to be interested by the title of an article or a book cover, and then you start reading it and it’s just vapor. Nothing tangible to be gained. It’s like buying something expensive and finding out a cheap trinket under the wrapping.
After a couple of times, you will develop a certain kind of heuristics for this kind of texts. It will not be perfect and will have some false positives, but that’s the only way to keep your sanity.
I like your vapor term. For me it's about the content anyways. If it's just some sort of vapid, pointless drivel then I'm not going to like it regardless of who or what wrote it. And in my experience text that strongly correlates with AI tells also strongly correlate with having sparse substance and lots of fluff.
In other words, if people don't like something, just don't read it. Shouting at people for AI generated text just makes you look foolish if the text is not in fact AI generated. And the person shouting "AI slop!" has no way to prove it other than vibes.
LLMs were trained with a lot of synthetic data to transform them from a complete this text into a chatbot, I suspect that this tons of synthetic data that forces the LLM to answer questions into a specific ways also forced them to have this "synthetic/robotic" language. Claude users would have noticed the "belter and suspenders" phrase just started popping out after an update and I am sure is nto because lots of developers used it in their blogs and Anthropic scrapped those blogs in that update.