- Slow PR review
- Nitpicks on style, not substance
- Async waiting for review or re-review after making changes, harder to stay in what passes for a "flow state" nowadays
- A lot of a ceremony around straightforward changes
We use LLM reviews at my company and it's caught things human reviews have not (as in in the early days we human-reviewed code and then did tests running the LLM afterwards and it caught things we didn't). It can review code at any time of the day and re-review it just as quickly as you can address the feedback.
I predict we (my company) are less than a year out from having a fully automated pipeline for reviews with only humans getting called in when it touches certain parts of the code. We already have a crude system for detecting certain file paths that we want a human to review if touched and we tweak that list over time as needed.
a) spend my valuable developer time (this is my free time) on reading machine
output which indeed might be completely wrong and get angry and upset for
reasons which have been listed in great detail by those opposed to LLM-use in
Debian on debian-vote
b) not read the machine-generate review and have the resources that were
required to generate it not only be wasted, but let my upload become another
item in a statistic which shows how many tokens Debian is using up because the
more tokens we use the better our software, right?
They also have a reward system that punishes people for ignoring the llm review (by pushing their submission lower in the queue).
This here might hit the nail on the head exactly in terms of what the biggest FOSSes gripe with LLMs with might be: not the fact people use it but rather how it hinders the communication.
If you use it to help the write code, but you explain things in your own words or, at very least, redact and trim LLMs output, it's fine.
If you, however, just drop a wall of text at someone and expect they act on its b/s, this is lazy to the point it borders with microagression.
You wouldn't expect a distinguished professor explain basic maths to a student, and we shouldn't expect actual developers address Copilot's stupid, unredacted hallucinations or otherwise unrelated remarks.
This works for patches submission, too: you should be outright banned for submitting stuff you didn't bother to double check and properly document.
Lazy or incompetent colleagues always forced people around them to overcompensate and correct. AI increased their firepower and blast radius.
But to upload to debian new's queue you already need to have pre-approval, it's a different situation from e.g. a public git repository that anyone can push to.
Why would you use an LLM instead of a fixed message?
Another situation where I would use llms to generate messages is if we've mutually agreed to do that. For example automated review messages at work, if we've agreed that it would be a part of our workflow.
A good rule of thumb might be unsolicited vs solicited llm messages.
For example, mine is a technical writing skill based on the old book "The Art of Scientific Writing": Front load information as much as possible - the first paragraph of a document should be a summary, the first sentence of a paragraph should summarize. Short sentences building on each other. Always show evidence.
Someone else spent quite a while optimizing a standardized PR description template, with a section for intent, expected changes, risk, rollout coordination and such.
That made the slop a lot more tolerable.
If you mutually agreed of course it's fine
This is what I fear will push its way into every part of society soon, too.
"We're not forcing you to do your banking through our AI interface instead of a normal one, you're free to opt out, but multiple functions will be missing and we'll charge you a monthly fee."
"You don't have to have an LLM grade your exam, but beware that the human grader will be overworked and grumpy for the extra task."
If you’ve applied to jobs recently, you must have seen the companies that transcribe and summarize interviews using AI (and probably share data across their clients in an indirect, technically legal way).
You can opt out, of course, but do you really want to be the one candidate for which the interviewer needs to take notes manually? And they just spent a lot of money on this hiring AI tool, will they hire the one candidate for which they couldn’t use it?
This is gonna get so much worse. It's like the smartphone revolution on steroids – and we've seemingly learned nothing! We're in dire straits.
Sure LLMs are not nearly as rigorous about their output but that doesn't mean it's entirely not useful.
At a minimum you can have your LLM summarize the other LLMs output and go from there.
LLM output may or may not be relevant, may or may not tell me something new and by the way it is structure is often hard to quickly oarse for the relevant piece in a wall of text.
A script is reproducible, and fixable. They won't waste your time hallucinating issues. Note that this is different from "being wrong". A script can be wrong, and you can spend time debugging it, but that time is not wasted. You can fix the script, remove wrong rules, whatever.
Also, nothing exists in a vacuum. Script-generated review comments were never a widespread societal issue, LLMs unfeterred content generation is.
Not even convenience at times, I think that for some the amount of technical-sounding text LLMs can output on demand is comforting. It feels serious. It feels like it's doing something important.
Meanwhile the rest of us have to deal with these low-signal/high-noise walls of text in commit messages and PR comments. Or ask another LLM to summarize them, I suppose. "Doing and undoing, it's all work", as we say in French.
I think it’s actually very similar to a script. It takes an input and produces an output. Certainly it has more variation than a traditional script, but it's generally going to return a very similar answer when the same input is given repeatedly, like when you ask an employee at a restaurant where the bathroom is. The verbiage may not be the same every time but the answer is always the same.
It just needs ca 5 seconds to realize "oh damn, I would not want to be at the receiving end of this" and conclude that it's probably a bad idea.
Why does this keep happening?
Especially since the idea of having such automated reviews probably isn't bad. But why mess up the execution like this basically every single time?
The way I see it the devs that put that in place are saying, listen before even reading your contribution I am going to feed it to a LLM with these instructions and see what it spits out. Now I will extricate my self from being the LLM proxy so you deal with this output and when it's okay I will look at it.
Same as "listen bud until it passes linting I won't even look at it." Just a much more flexible and fuzzy linter.
It's just such an obviously terrible execution. Why would you throw it into production when doing literally anything else - and that includes just setting the whole server on fire - is less damaging.
(Not hyperbole. Loss of trust and loss of trust in ones own judgement is the highest and harshest price in this industry)
GR resulted in Proposal E, Choice 5: Responsible Use of Generative AI. Clearly stating: "Any such automated process should be overseen by a human who remains accountable for its behavior and output."
It seems like the question is where to fit it in to the pipeline. The team responsible for reviewing submissions could leverage it to speed up the process, and still respond personally.
It became actually useful only with the advent of Opus 5 and its competitors. Nowadays it usually makes a few good points, but in instances where it doesn't, it's quite the stubborn asshole.
This one time I had to paste a function from a library because it couldn't get it to its digital head that its assuming the function has a certain implementation based on its name only.
/* @code-review ignore */ [...] /* @code-review endignore */
I mean you shouldn't have to, but I'm wondering if you can.
Lesson for us all is that if we do not own the thing/project/etc, things can suddenly change without our decision, and all time we put into it might become a waste-of-effort retroactively
https://www.debian.org/devel/constitution
It sounds like in this case someone is not following an agreed-upon policy. And of course, even with governance like this one, the rest of the maintainers can move in a direction you don't agree with.
I contribute to open source software because it improves stuff I use and/or care about.
It seems too pessimistic, IMO, to abstain because things could potentially change to the worse, sometime in the future.
Patience and persistence helps, but it can be daunting IMO
I replied as I did, because lukeify implied that lack of ownership is
> one of the primary reasons I can never get interested in contributing to open source
In recent versions of iTerm2 for example, when using the plain old menubar and hovering over some items, panels appear showing awful sloppy AI images. WTF? https://files.catbox.moe/5ivkhw.png - That makes me question everything... Was the developer always tasteless? Or did they get a taste bypass?
With AI, creating the tasteless thing became easy.
If you give people the ability to select a font, they will select Comic Sans.
That AI image is pretty baffling though. Setting aside the pointlessness of displaying an image in the context menu, it even looks like it was made with ancient stable diffusion from 2024 or something, modern image generators are "better" than that. I wonder when it was put in.
I've never considered updating iTerm. Apparently that lazieness was the right call?
Jesus. What the hell. What does that sudden AI image even offer? I guess it would make some sense in a "docs page" context (even though there it would too be slop), but as a tooltip of a menu option?
Gotta say though.. that is so unhinged, one has to respect the human ingenuity behind it. No LLM can come up with an idea so batshit insane. They're too mid for that.
The question is not even about the setup in terms of hardware/software. The question is about organizational structuring of its processes.
Even something as simple as writing the phrase "No LLMs please" somewhere in the upload, that if present, prevents LLMs being triggered. Nothing elaborate and it preserves the default-on behavior the team seems to want.
The problem would be better solved with an opt-in.
But it looks like the team decided to shoot themselves in the foot there and both implement and communicate the change badly...
There is no alternative to Debian for me. I’m ready to justy throw out all my computers and become a goat herder at this point.
PS I’ll remove memory and storage from said computers to pay for land and goats before I chuck them out of my window.
> I rather wait a year before my package makes it through NEW (yes, that happened) than have my strong opposition to LLMs be ignored and all my uploads being automatically screened without my consent.
so those entries go in the "abstain-from-LLM-queue", where there is no guarantee of timeliness at all, and human volunteers can process that queue manually as they are available
tldr just tag the entries as "OK for LLM" or not. it's just like organic and non-organic produce, it can be done.
Picking at random a successful and unsuccessful LLM-generated review from the dashboard, the successful review consists of 734 words (~2.5 printed pages) and the unsuccessful review weighs in at a whopping 1,535 words (~5 printed pages). I haven't tried to fully read either report, nor spend brain cycles checking if the LLM output is legitimate or not. I just want to emphasize the many orders of magnitude change in review size that's being generated, and that uploaders may need to spend time trying to review and understand in this new process.
A bit rude to expect people to swallow 5 pages of slop to figure out why review failed.
EDIT: made my comment more constructive, but originally asked if any distro has adopted a hard anti-AI stance.
Yes all the BSD's, and i think with the best explanation, since one cannot proof what license the llm's output has, one cannot legally implement the output under the BSD-License.
But what about Linux? They just dont care, they have the money, corporate backing and with that, all the power.
Linux itself is in any case tainted, but some of us care to contain the plague however possible.
That said, the LLM presence and involvement should be confined to the team using it. "We don't know what it says, it is not a part of judgement, it may make mistakes, but here's a wall of text from it for you to read for who knows what benefit" is quite a dick move. Don't throw slop into people's faces like that.
