My guess is years of RLHF optimized for rater's first impressions of how well-written something was. It uses lists! It has headings! It uses fun phrases like 'smoking gun'! It's long and informative! These things are relatively fast and easy to identify and select as a preference.
But none of this really covers whether the response is _useful for the audience_. If you're not the audience, it can be hard to tell what is and isn't useful.
And pretty much universally, they were awful. It was like somebody had blindly taken an English writing textbook and applied every little helper rule to every clause of every sentence, paying no attention whatsoever to deeper concepts like pacing, drawing attention to key words, and moderating emotion.
And I think that's basically what's going on in all of the frontier models as well. When they are being fine tuned, some reward score is telling them to be concise, not to use filler words, to avoid run-on sentences, etc. And there's no counter-balance in the fine-tuning that looks at the writing and flow as a whole.
I remember when I first used Microsoft Word maybe some 31 years ago (Windows 95 was that long ago, huh), and I had squigglies everywhere because of some grammar checker. "Don't use passive voice!"
At least back in those days, it was advice that I could research more about, and decide whether I wanted to act on.
It likely that no response at all or "This looks good, no suggestions" are not possible responses. They haven't figured out how to detect when a conversation should just end or that no response is necessary and include that in the training data.
"Google shipped their" -> "Google shipped its" (I use 'their' to make it clear I'm talking about the people who build features at Google, and not merely about a faceless corporation)
"feature or whatever" -> "feature" ('or whatever' is chosen to tell everyone that I'm not actually sure what the name of the feature is, and I can't really be bothered to figure it out)
"gmail" -> "Gmail" (meh)
"I wrote, it had useful tips" -> "I wrote received useful tips" (if you remove the comma, the pacing changes, the reader doesn't linger on the word 'wrote', and you invite some ambiguity around the writing and the receiving of useful tips. I feel like this suggestion is particularly brain damaged, I simply can't understand my own writing when I make this change)
(and so on. I now write emails in notepad, and copy-paste them into Gmail when I'm done, that's how much the suggestions annoy me).
ChatGPT, Gemini, and whatever else do not in fact have the capability to understand that interaction between organization and translation. They process the information linearly, organize it linearly, and translate it linearly. That's what makes it so strange to read. A must lead to B which must lead to C. A cannot connect to C and then circle back to B which is connected to both A and C obliquely. It's a failure in understanding language and how it is used.
The suggestions are valid and correct, we (generic "we") are just used to not following proper grammar. This is how languages evolve but it is also what puts these assistance tools in a peculiar spot, because they help with accuracy, not everyday expressions.
I'm certain there are folks out there who have a hard time with written English that could benefit from an AI editor, but I would almost always rather read casual English written by a human, even if idiosyncratic and "incorrect" sometimes.
And, to be clear: Hard to read English can never be correct, no matter how carefully it follows grammatical rules, and AI often writes hard to read English.
A side effect of studying programming languages in my youth was that I began understanding grammar better. Suddenly words belonged to "classes" and when thinking about "object references" in sentences, and the rules that followed, it helped me read and write more clearly.
I think I might be slightly opposite of you, but not very far opposite. I prefer reading a precise and accurate sentence that - when understood - clearly describes the intent without a shred of doubt much more than an easy to understand sentence that I find ambiguous because, to understand it, it requires context and a social agreement that nobody is trying to trick people.
I had a friend once that kept using open-ended phrasings and ambiguous descriptions because (I think) he enjoyed the feeling of power or control that the confusion and miscommunication introduced. Like if he were to "run to the stores" you could ask what he was buying and he'd follow up with a "oh, I didn't say I was buying anything, I'm just going for a run". If I have to guess which of the variations of meanings is the intended one, then the communication has failed in my book, regardless of how easy it was to understand the sentence.
A failure to express intent, not a failure of grammar. Perfect grammar can mislead just as easily.
"The ManU have won three out of their last four matches."
"The Parliament have had heated debates over the NIH budget."
"Tesla introduced their newest EV model last week."
Any competent English teacher should know this.
Also, there’s no "correct" language outside of how it’s actually used. Linguists are the first to explain that their job is to describe, not to prescribe.
"When to use a singular verb
- When all the members of a collective noun are performing an action as a unit (and that’s usually the case), use a singular verb.
- The chamber orchestra often plays at the Art Centre.
When to use a plural verb
When the members of a collective noun are performing an action as individuals, use a plural verb. In this case, all or some members of the group are doing something independently of the other members; the group is not acting together as a unit.
- The orchestra are tuning their instruments.
- The cast have been practising their lines."
https://our-languages.canada.ca/en/writing-tips-plus/verb-ag...
So what would you suggest a writing assistance tool should focus on between correctness, common use and likelihood of the reader understanding the message? I've had plenty of Russian and Chinese colleagues over the years who will butcher and utterly destroy the word "the" from their written English messages, with a passion, but I managed to understand what they were writing regardless.
A smart writing assistant will give advice that’s appropriate to the context, register, the recipient’s culture, communication style and tastes, and the author’s relationship with the recipient. Sometimes writing very normatively and by the book is part of the message. Sometimes the exact opposite is true. If the assistant is unaware of all the context relevant to the communication, it’s likely to be a bad assistant.
Dropping articles is wrong because English as natively spoken doesn’t work like that. Except when it does, like news headlines, condensed "telegram" or SMS style, and so on.
And that's why it's so difficult to read the things that LLMs write. Because they follow the rules of language without embracing the chaotic nature of the fact that it's meant to convey information rather than to "be correct".
In Swedish -
> Vi går till mitt hus
"Let's go to my house" - this is correct but not really said
> Vi går hem till mig
"Let's go to my place" - incorrect but the actual way to say it
Definitely not a chair, but a corporation is also "just" a group of people. "Their" makes perfect sense for a corp, even if it isn't perfectly accurate
Everyone who understands English at a professional level (and probably even lower levels, too - or perhaps especially those) will understand what you mean if you say "the corporation announced their financial report last weekend", but it won't be the correct use of "their", as far as I'm aware.
Similarly you can't, as far as I understand, append an "s" to describe a relationship of an inanimate thing, e.g., this would be incorrect: "the corporations employees went to the summer party" again because "corporation" is a thing. But on the flipside phrases like "the employees of the corporation went to..." sound a bit too - what's the word - posh?
What is "a thing", exactly ? Corporations are certainly ambiguous in that regard: they have legal personhood[1] and even free speech[2] in US, apparently.
In the other hand, no corporation ever really "announce its financial report". Someone does it on behalf of the corporation, technically.
[1] https://en.wikipedia.org/wiki/Corporate_personhood [2] https://en.wikipedia.org/wiki/Citizens_United_v._FEC
Yes, yes it will be.
See collective nouns, and "Metonymic merging of grammatical number" https://en.wikipedia.org/wiki/Collective_noun
> this would be incorrect: "the corporations employees went to the summer party" again because "corporation" is a thing
You can easily say "the corporation's financial report for Q4 was worse than predicted".
I think you're focusing too much on specific and hyper-specific rigid rules in a single textbook. As others said, prescriptivism vs. descriptivism. Descriptivism is always better, as it describes how language works and may evolve. Prescriptivism assumes languages are rigid unchanging things that can be precisely described.
The full rules for a working language take people years to learn at universities, and they are never as simple as "an inanimate object can't have a possessive form with 's".
Prescriptivism also works in schools/learning as it helps people not to drift too much from the norm. But too many take it to heart.
(Sorry. I blame Claude.)
At least in the case of ChatGPT - it was (and mostly still is) objectively optimized for engagement. Since ChatGPT was leading the way, and much distilling has occurred, there are at least partial remnants of that problem everywhere.
The Chinese models may or may not really care about this at all, but they're doing a lot of distilling, so they're getting it whether they want it or not.
The American models for the most part care about engagement - some a lot more than others.
Things that are easy for humans, like remembering a character is holding a gun during a conversation, gets forgotten, because there is no clear architectural/spacial understanding of how the words are describing an underlying reality. And if the LLM gets this right it might instead forget which character has a Southern drawl, or make an illogical metaphor.
This is all more clear with programming, but the problem is the same - a lack of interwoven semantic understanding and friction between the real world and the self-contained world of the LLM's context.
It also insisted that I start an hour early even though it was adding in 3 hours of rest time at the end.
Human judgment it is not
Chat: https://chatgpt.com/share/6add218a-3bc8-83ea-a2d7-613444eb7c...
Maybe we'll get assistant models that will get optimized for human readable prose, but I think the labs have deluded themselves when it comes to utility/coding models and think the humam isn't going to be a variable soon.
I have a jira-esque ticketing system for agents/subagents and the tickets are basically unreadable to me. I hate it. But if I force them to write the tickets in a prose I prefer their preformance degrades. Its almost like this gibberish hard to read prose is designed for llms to communicate betweem themselves probably because the llms are doing most of the RLHF (RLAIF??) for training now.
[0] An example would be something like:
"The user presses a button, and the form submits."
There are valid intransitive uses of "submit," but none that make sense in this context. In this case "submit" either requires an object (like "the form submits data"--plus maybe an indirect object, like "the form submits data to the server") or should be swapped for "the form is submitted" if you absolutely don't want to supply an object.
LLM-generated text is hard to understand because while it's perfectly good language that follows valid cognitive and informational patterns, it does not reflect the intent or mental model of another mind.
This is entirely speculative and highly disputed, even if you are doing the more sophisticated thing here and by Language you mean something like mentalese.
Much cognition is embodied and makes use of visual, tactile, auditory, kinaesthetic, and other sense modality representations. "Thought is mostly a Language" has been a wordcel kind of argument with little empirical or philosophical support for a couple of decades now, IMO. Language is how we communicate thought, not primarily how we think.
Not that this refutes your main point: LLMs are trained almost entirely on (written) language, and since language is only a single, limited aspect of real cognition, this makes them so annoying and alien to read.
But LLMs DO show us that language is a big deal. It isn't just a serialization protocol for real thought -- it's a material component of how we think.
In many ways I think LLMs are the strongest possible proof of structuralist and post-structuralist theory which argues that human cognition is conditioned on the language and concepts to which we have access.
The neat thing about LLMs is that in a way they proved that only the truth turns out to be grammatical (statistically), largely because there are so many more ways to be wrong than right that the mode is almost always right.
But people also lie and bullshit in consistent ways, with the intention of fooling people with well-known empty arguments. Part of being a "good writer" is to make the meaningless sound meaningful - and the techniques are ancient.
Train on a bunch of people trying to push their framework and get rich, whether in online garbage, literary garbage, or in ostensibly serious nonfiction or scientific works, and you'll end up with modal language that is trying to play deceptive games with reasoning, but has nothing in particular it's trying to convince you of (no motives.)
People would either fall for that language (if they weren't thinking much and the context was one that comforted them), or they would methodically try to put that language into a model (let's say a series of syllogisms for an argument about what to do or what to believe, but it could be any model), and see that it failed. And speaking again of the context, the context of humans interacting with human content implies motives that help the listener focus on particular parts of their model to look for rhetorical tricks. The LLM, again having no motive, just does this in random places.
The latest advances in AI have been in the tooling around AI, which I think largely reduces to recursively creating models and populating them in outputs short enough that hopefully the text generation won't go off the rails.
Symbolic AI is coming back, it's just that the mortar is random word association.
Call me crazy, but I think Georges Bataille saw all of this. In my interpretation, he simultaneously thought of random associations as the most important component of reason and also completely irrelevant in the reality of the world. The only way you could discover the structure of things is through random collisions; your view of the universe was entirely constructed by your prejudices, lusts, and fears; but whether that view was wrong or right was based on how it worked and the particular content of the id-garbage that got you there was ultimately irrelevant. The fabric of the world is completely id-less, but can only be digested by id-driven beings. LLMs (without the tooling around them) are all id.
>We take these results as a proof of concept that a length penalty used during reinforcement learning with GRPO can directly affect grammatical features. We note that the model was not directly instructed to use compressed or dense language, and was not directly given information about the availability of syntactic alternations. Rather, the penalty resulted in an emergent preference or shorter members of syntactic alternations. [...] We speculate that the use of verifiable rewards creates an environment where correctness is decoupled from humanlike communication, leading models to converge toward a novel, potentially less comprehensible distribution of language use.
This sounds like common sense to anyone who tinkered with the models and training, however someone still needs to quantify it so it's good they did. I also want to note that this is only visible in Claude's default register, it's still able to write differently when prompted.
If we had talking centipedes the results would probably be similar, but for different reasons.
Which gives a relative incentive to hard to understand text.
LLMs text generation is much closer to the way humans speak than to the way good writers write.
(e.g., "baseline system state transition latency validation") instead of unpacking them with prepositions ("validating the latency of state transitions in the baseline system").
For me, I can't read books at all, because I forget what was said two paragraphs before. I bet people that are good at skimming text have less problems with AI generated text.
"Baselinesystemstatetransitionlatencyvalidation" is surprisingly enough less confusing to me because I know it's one word describing one thing, and my eyes registers each joined word with (years of) training, even though what my eyes look at might be the following chunks in order:
1) [ Baseline ] systemstatetransitionlatencyvalidation
2) Baseline [ systemsta ] tetransitionlatencyvalidation
3) Baselinesystems [ tatetransitio ] nlatencyvalidation
4) Baselinesystemstatetransiti [ onlatency ] validation
5) Baselinesystemstatetransitionlaten [ cyvalidation ]
My eyes essentially scan the word, and although it might be a tad bid slower than reading "baseline system state transition latency validation", reading and understanding is faster because I already know it's one thing before I know which thing it is.
Edit: The example would be easier to demonstrate if HN supported bold font, but I don't think it does (or know how).
Is it? I mean they get payed more if they return longer answers.
If we had a way to write a test suite for English that could reliably answer "is this sentence useful to my target audience" or "does this paragraph make any goddamn sense", you might be able to run some sort of agentic loop to approach good writing, but so far we don't.
I've been struggling with this because I do believe maintainability of code is important. But the way it is going, if AI is writing all the code and no human needs to read it anymore, does it really need to be maintainable? Does it matter if it looks like alien code since we can throw it away when using the next AI model? I have no answers of my own to these questions...
- For throwaway prototypes I pretty much vibe code, only making manual tweaks occasionally if the agent gets stuck or something looks wrong visually. - For production code I stay in the loop, reviewing and iterating on the code with an agent.
This leads to me having experience with “messy” prototype code and “maintainable” production code. Whenever my prototype code lives longer than expected it gets to a point where the agent starts introducing regressions or churning on changes.
This happens much less often with the “maintainable” code. From my perspective it seems pretty clear that LLMs are faster, more effective, and less error-prone in a well structured codebase
But yes, in my experience the code is often quite bad in other, harder-to-quantify directions: readability, maintainability, simplicity.
With humans language, sometimes "wrong" is in fact "right."
I've been iterating on this writing clarity prompt. The term we came up with was "Asyndetic parataxis with referential ellipsis and repeated rhetorical closure." It doesn't fully describe the exact issue I have with modern AI writing, but it's close.
Or reading patents.
Terrible ... like someone put in effort to make it unreadable.
AI language, it seems, is optimized to support reasoning from the interior monologue.
[0] https://github.com/cursor/plugins/blob/main/pstack/skills/un... [1] https://github.com/cursor/plugins/blob/main/pstack/skills/te...
People who are knowledgeable in a particular obscure area tend to do the same thing.
Of course if you optimize for natural-sounding language it will learn to sound more natural.
In contrast, most human communication and writing is intended for other humans, and we constantly hit failures here that we learn to iterate on and improve. Humans who do math do so with the constraints that their math needs to be reviewed by another human and in the context that all their writing needs to be understood by another human.
The process causes a warped and compressed version of language.
I seems to remember a distinct time during which Claude for example became incomprehensible and weird with its writing, whereas before it was reasonably OK, even if still not that good
Yes, this is a problem.
AI-generated text is used internally by the AI to instruct agents, etc. And for feedback loops in reasoning. If you optimize for the result only, the AI will increasingly speak "AI", not human.
I mean, maybe that's all right for when it's producing code - code is not beautifully flowing prose in the first place.
Code is not prose, yes, but as others pointed out, it is certainly nice if code is maintainable by humans.
So there's the usual structure of "it's not x it's actually y" signs.
It uses a lot of metaphors. It references stuff implicitly - a LOT - and assumes pretty much all the time that the thing it did is what you saw, which you didn't. But if you just.. read the stuff it says, and know a few old-school sayings and thesaurusize your reading style, it usually says things quite meaningfully.
I think a lot of people just don't read what it writes. So many folks I've watched used LLMs don't write in complete sentences, don't read the sentences that come back, don't watch the turns and tool calls in the middle, and then just respond to the message as if it says something it doesn't. I'm saying all this before I read the paper, but I've done a lot of personal and team experiments with this, so I'm quite curious to see what they say. I'll edit this after I go through it.
--
Fascinating! I did actually run into a lot of this with my personal 'engineering and communication styles' experiments. Most of the papers out there talk about models moving toward human language inside of a context window, and not the other way around (much smaller effect for human->model). I had a few years of conversations I ran on local and 'frontier' Claude and found that, yes, in general, information density is what each response turned out to be geared toward, whether it was everyday health discussions or deeper engineering topics and research. The component tests they ran I've been exposed to already, but I think their conclusion nailed it: ```This discrepancy may help explain why many frontier LLMs use linguistic constructions which are technically allowed by the grammar of English, yet are out-of- distribution for human language and are associated with comprehension difficulty.```
The best approach is just to write the outline and initial text in a very messy fashion yourself. And then have it clean it up and add detail. And then edit again by hand.
I'm to the point where I don't trust it to do any writing. It's only allowed to tell me what it thinks I missed. Because inevitably when I ask it to edit my prose, it destroys the text and I basically need to rewrite it all again.
Basically, there are often different ways of putting a particular idea into words. Sometimes the longer way might be clearer, and the shorter way might be more confusing. If you care about being understood, sometimes you will use the wordy option. This is what humans usually do.
However, LLMs have context windows. All else being equal, verbosity eats up more context. If the LLM is trained (with reinforcement learning) to reason most effectively, it can do that by using terse phrasing. Especially on certain tasks that really need a large context window. Why? Because being terse is like getting a larger context window for free.
But in another sense, it's not really free. You're trading off clarity for context window. Sometimes you have to choose between verbose+clear and terse+unclear. If you're biased toward terse, that will also shift you toward unclear.
I do agree that LLMs are verbose sometimes, though. I think they are also trained to please the user, and they see sharing extra knowledge and insights as a way to do that. I don't think this idea is in conflict with what the paper says. If an LLM's training rewards both spitting out neat bonus info and doing reasoning that requires making the most of its limited context window, then that LLM may generate a large quantity of text which is written in a slightly terse and confusing style.
I.e. a high frequency 3-gram is "by the way".
A low frequency 3-gram is "elephants speak french".
They’re difficult for humans because if a sequence of words hasn’t previously been seen, you have to expend more effort to make sense of it.
Something like “customer onboarding workflow automation" might qualify - it makes sense, but it’s not a commonly used phrase. AIs use a lot of these jargony sequences.
I think the difficulty in comprehension was mostly an Anthropic problem… partly because Anthropic employees stopped reading Opus output around the time Fable came out and stopped noticing RLVR going off the rails with compression.
The paper was a good read, in any case. Maybe I gotta start sending this article back to LLMs when they start compressing text too much :)
I regularly prompt it now to "summarize again without the jargon". That helps, for now, to bring it back to using plain language. Together with my basic prompt to reply using ASD-STE100 Simplified Technical English (https://en.wikipedia.org/wiki/Simplified_Technical_English)
So I tried it out, and it really sometimes works, and I've overcome some real obstacles that Tech Support was unable to assist me with. However, Gemini can also become "too big for its britches" and begin explaining glitches from a systems-programming perspective, as if it has deep knowledge of the Android codebase, and deep engineer's knowledge of the Pixel system hardware. And it will spit out absolute nonsense.
I have retorted to it a few times, "that sounds more like Star Trek than a sound appraisal of the subsystem" and then Gemini smirks and says "You're right to call me out on that!" and blithely continues on its way.
So yeah. Sometimes the LLM confabulations are deeply cloaked in highly technical jargon that could not even be questioned, unless you already know it's absolute bullshit. And then the LLM freely admits it! LLMs have no shame, that's for damn sure.
But I also wonder why AI loves to add slogans, catch phrases, and witty sentence fragments. It’s grating and a dead giveaway of slop.
As far as I can tell, it's only fluent readers who have trouble.
inanimate subjects with animate verbs
more stacked nouns
long syntactic dependencies
That makes sense, because fluent readers are using more System 1 processing to read - they’re not thinking about it for the most part, they’re just pattern matching instinctively based on text they’ve been exposed to previously (“trained on”). AI writing breaks those patterns so compromises their fluency.
Whereas a second language reader or poor reader is exerting more conscious attention to understand the text, so it doesn’t matter as much if it doesn’t conform to standard patterns.
It only falls apart when you actually try to figure out what's being said.
I find the most passionate advocates of LLMs right now to be the people who are
1) also the most passionate advocates of not reading or reviewing their output in any detail. They seem to be actively annoyed by the idea that people think they should. And
2) people who have a strange theory of mind about LLMs that excuses the low quality of their output. I've heard many times people advancing the theory that LLMs are coding in a particular way because the way that LLMs code is the way that they (the LLMs themselves!) are best able to work with the code.
In fact, I get the impression from these people that they think that their touching the code would actually ruin this delicate balance that the LLM is trying to make. This is actually crazy thinking. On a pretty obvious level, they already think that any LLM they encounter is a super-intelligent God acting in mysterious ways, and that they only way you can ruin its plan is by questioning it.
Whether or not that's true, you have to keep in mind that it's not true of everyone.
I'm a passionate advocate of LLMs. I implemented my first neural net, for image recognition, in the late 1980s (based on Bart Kosko's work, I'm not claiming to have invented anything.) Now, I use LLMs daily for coding and other tasks. I don't usually "vibe code" unless it's a simple enough project that the code quality really doesn't matter - a simple example would be data integration, where as long as the data in the target system matches the data in the source system, I don't care much about how the sausage in between is made.
But otherwise, I'm approving or denying every tool call, like a caveman, and when they mess up, I explain to them why. I interrupt them and redirect them when they need it. I change code myself and let them know. With all that, they're still incredibly useful - more useful than most "junior programmers", realistically. No human software developer can hope to match LLMs on many significant dimensions of development. But that doesn't mean experienced and competent humans have no role, at least currently.
The disconnect here is that your characterization of "the low quality of their output" is too simplistic. The quality of their natural language writing is different, for the kinds of reasons stated in the OP. Some of the accusations of "lower quality" really boil down to "too compressed for humans to easily understand", again as the OP points out. And the quality of their code has a lot to do with how they're prompted.
One of the biggest barriers to good LLM output - both writing and code generation - is that they're trained on data generated by humans who fit George Carlin's observation that half of them are stupider than average. When it comes to coding, on average they don't have enough education not to think that imperative code with minimally-restricted mutation all over the place is a good idea. They think dynamic typing, so that code can't be statically verified, is a good idea. They reflect our own collective limitations. If you recognize that and redirect them to sources of good practices, they can be much better.
In short, they're much more of a mirror than most people realize or admit.
With all this noise, it should not be forgotten that we appear to have figured out roughly how a significant component of intelligence works, and have real life, useful proofs of concept. All the technical criticism, to me, is just coming from people who don't seem to understand how intellectual advancement works. The political criticism is another matter.
This is likely relative to my existing knowledge. Humans are also LLMs, and a human without prior training will be able to transfer-learn slop from a computer LLM, but a human with prior training may find the attempt at transfer-learning conflicts with what they have already learned, and reject it.
In less ML terms I think a human who reads slop and has no existing knowledge of the subject will just learn the slop by rote and confuse it for actual knowledge.
Dunno about any test, but I get the meaning behind the words in a way that I can take appropriate action (as someone using LLMs heavily for dev and always following the transcript so I can properly steer them). And often I see people here screaming "AI text" or similar regarding some linked article, and having read it I found that I just bypassed all the "tells" without realizing, because they didn't affect my absorption of the meaning of the article. Makes me think people are consciously looking for those tells so they can scream and shake their pitchforks.