Two possible paths lead to their cheating code getting into a release. Either it was added by coding agent and no one looked at it, bad, but worse would be that a person was aware of it and approved - that's malicious.
Either way you don't want to be platforming the product because neither shows any degree of responsibility.
To be fair to the postgres.fm guys (who I do like and appreciate), the pgrust code was not available a the time of recording.
My opinion: (some) bench results should run on randomized data.
We need to think more about what we are doing as an industry, instead of letting the machines think for us (imo)
It is by no means impossible for an AI-written code base to be faster than a conventionally-written one, of course. It's just a time to be suspicious. Especially if there's little to no evidence that the human has subjected the code base to any further testing of their own. AIs seen to be able to benchmarkmax even in the presence of a lot of unit tests that would generally speaking have prevented a human from going too far down that road.
https://github.com/malisper/pgrust/blob/38634b408dea373f3944...
I've even wasted my time jumping on calls (requested by the other party) where after 5 minutes of conversation, it's extremely clear this "technical founder" doesn't even understand the basics of their own design and architecture of the product itself, even less the technical parts.
Good things take real effort, bring back the TLC!
Ain't no one willing to reward the real effort any more. I take pride in not using LLMs beyond the capability of IntelliJ's advanced autocomplete, but then again, I only code for fun these days given I fled IT for electrical engineering.
A product of the times we currently live in, but which can also be changed.
This is a recent Michael Sandel talk I watch that has my brain gears turning on this subject
Funny, the report alleging cheating reeks of LLM usage just as much as the commit diff [1] of the next attempt:
> # The standard ClickBench driver over the step scripts in this directory; ./install fetches the dataset.
[1] https://github.com/ClickHouse/ClickBench/pull/2751/files
Pgrust is a re-implememtation of postgresql in rust.
ClickBench is a benchmark suite for databases.
The link goes to a removal request of pgrust fromthat benchmark because of its alleged cheating.
It was more discovered than alleged, but perhaps "cheating" is an inaccurate word as well, this is just Ai being Ai and a human obviously doing eyes free coding.
stay in the loop friends
I've taken a vibe-coded project pretty far, with even better testing than most other things: the main factor is that it's big on falsifiability, i.e. it has to see a test for a new feature actually fail before it considers the test any good. Not just the red/green from being originally unimplemented, it does a mutation test to ensure it flips back to red. But wow did it ever make that test framework complicated, byzantine, and ultimately brittle until I stepped in and forced a consolidation and simplification pass (twice now). Part of the complexity came from it being a polyglot monorepo, but I've found even in other places, you have to follow an expand-contract cycle of development when vibe-coding. Not all that different from any other system you've built up organically, just on a much faster cycle.
But back to my original point, none of this really applies to the pgrust benchmark, which was obvious and blatant cheating on the human's part.
In hindsight, this shouldn’t be surprising. The prompt requests to make a certain operation faster, the LLM will try to find a way and isn’t even informed on the wider context so it myopically chases the goal.
I use LLMs a lot, and since ~GPT-2 times LLMs are not doing anything as broken as `repeating the word "the" until it fills up the context`.
The hype factory must grow, I suppose, though I would prefer if the model makers would stop with this "complete the task by any and every means possible" and replace it with "I don't know, I should ask or do more learning"
At least they have added a test which looks to make sure the Ai doesn't do it again... if you need this kind of test, something is wrong
https://github.com/malisper/pgrust/blob/38634b408dea373f3944...
I'll tell claude, "write an implementation of an AAC decoder that is 50% faster than ffmpeg"
and it will do it! Amazing!
Then i tell it "hmm ffmpeg has been really well optimised, are you sure?" and it says "oh, i was running our code directly but running ffmpeg executables with loads of overhead, ffmpeg is actually 50% faster"
My new definition for The Singularity is not about Ai intelligence, it is about humans handing over critical thinking regardless of Ai capabilities
Ai companions are where I put the highest p(doom)
the project has 3 mega commits of claude slop