There's a reason why we talk about software development lifecycle, design, architecture, testing ... It's because it's been the most reliable way to build and ship software. We shouldn't expect discard this and expect agents to perform well outside of this.
I'm treating LLM agents as junior devs who happen to have vast knowledge of software engineering. As their team leader i make them go through planning, implementation, bug sweeping cycles using strict workflows. And it works quite well, i've been working on several large projects (1M+ LOC java,typescript,c/c++) and by any measure the projects are healthy. Sure the code isn't that beautiful, sure i'd have written things differently but it's pretty good nonetheless.
Shameless plug here: i've been also working on https://kodfactory.com, the code factory i've built to work on these large projects with workflows, reviews, etc ... I'm cleaning things up to open source it later.
But I don't really want to play a part in a simulation, trying to cajole my scene partners into saying the lines I need them to say. I want to use a tool the same way I would use any other tool. If this is AI it should just do the thing. Anything else is an imperfection of the technology.
But at the same time, language is a vague communication medium. We have a precise language for describing forms of computation, but that's code so we're back at square one. We still haven't nailed the right amount of follow up and correction and interrupt-ability of these coding agents.
And we may never figure it out. It may simply be impossible. But it doesn't mean this weird anthropomorphization of AI is something I want to do. If I wanted to be a manager, I would be a manager.
because that's how agents are marketed.
heaps of people on this site expect them to be omnipotent then claim it’s fake when it doesn’t read minds
- https://openai.com/index/introducing-the-codex-app/
- https://www.anthropic.com/news/claude-3-7-sonnet
anthropic specifically brags about how good claude code is every annoucement of a new model. I will surrender that none of them claim its "to perfection", but IMO its implied because no one would claim that their model one-shots any issue to dog shit quality.
no where does this document suggest that codex can "one shot everything to perfection with just a prompt". It describes using a prompt plus agent skills (which are essentially many other prompts) to develop a playable game.. nothing about it being perfect or anything more than being in a playable state.
Besides, other people's claims about something doesn't give you license to abandon all critical thinking. Though it's evident they don't claim what you say they are.
To be clear that's not what I'm thinking, even Fable 5 produces some hilariously bad results under some conditions and sonnet 5 produced great results under others.
But you didn't provide the evidence for that. You shared some links and then admitted they didn't claim it.
It kinda seems like "because I think they're a little too positive about their product, I can set my expectations to anything I want and la-la-la it's their fault."
And I don't see the problem with agents building test scaffolding as they go. It might be too defensive at times, like testing a shell script you don't run often, but big deal. It's kinda cool imo, and it's trivial to make it stop.
Every time you see a benchmark for "how long the agent can go without asking for human intervention", that's encouraging vibe coding.
because that's the end goal? and for simple small stuff they're already there?
That is also because I have been obsessed with this synth for almost 30 years. I built clones of it 20 years ago in reaktor. I know how to spec out every aspect of this synth and I gave Claude a 150 page pdf on digital filter design too.
The results are far different than someone who has never used a virus prompting "make me an access virus B synth as a single html page".
We are calling both of these processes "one shot" but this is not even close to the same process.
I suspect this is the LLM discourse in a nutshell. People are using the same vocabulary for wildly different processes.
Sweet, I can't believe some people don't love this.
They do often enough that it's not a surprising event, depending on prompt quality, context available, ability for the result to be objectively judged and iterate on by the agent, etc. For frontiers on very high settings at least.
My agents have never created code that is straight-up garbage and I have never flushed a week's worth of tokens down the toilet. I just can't identify with all the constant complaints about AI-assisted coding.
And my biggest project isn't some hello world app. It's a self-hosted, privacy-focused personal financial management application that I intend to open source. It's about 126k LOC against 240k LOC of regression tests and 30k LOC of CI/CD pipeline. I'm doing 24x7 mutation testing on a dedicated box against the accounting engine and temporal systems. I even have specialized agents doing audits against Regulation Z (US banking law) criteria so the app models the required behavior of banks.
Most of my complaints about everything are nits, like the overly verbose and dense way LLMs communicate with me. Or their predisposition to add, add, and add more stuff when proper engineering practices are more often about subtraction (but I've built mitigation guardrails against a lot of that).
> a… personal financial management application… about 126k LOC against 240k LOC of regression tests and 30k LOC of CI/CD pipeline
Just how much functionality are you getting out of that? It's hard for me to imagine that people want that much out of such a program. I just keep a spreadsheet. (Yes, LibreOffice is also very bloated.)
In my defense though, it does way more than a spreadsheet. Stuff like OCRing screenshots of bank transactions with a specialized, locally-hosted LLM to avoid data harvesters like Plaid. This became an entirely separate subsystem with verification, automated model benchmarking, prompt provenance, etc.
You'd be surprised at how quickly edge cases start to pile up when an accounting system makes contact with the real world. (If you buy something on a credit card and then return it after your statement closes but before your payment is due, do you still owe a minimum payment based on that purchase? Well... depends on your bank. Capital One and Chase: yes, US Bank: no.)
> Just how much functionality are you getting out of that?
I'm still dogfooding it. It's a pretty opinionated app that has things a month-end closing ceremony, reconciliation processes, envelope-based budgeting cycles. So unfortunately my feedback cycle is largely locked to the calendar. But my wife absolutely loves it so far.
> 126k LOC against 240k LOC of regression tests and 30k LOC of CI/CD pipeline
I mean...
Yeah, that's pretty self-explanatory why you don't identify complains about AI-assisted coding.
If you ask for a particular thing, they'll do it, even if it's not a good idea. When you start running into issues, they'll try to solve those issues for you. They'll do that as long as you keep asking, even if there's no good way to properly fix the issues, because the initial approach was wrong.
When an agent is struggling to produce something, I switch to asking it to re-evaluate the approach itself, and ask it to suggest a less brittle approach. I then chat through the various options, and choose the best approach that makes sense, and then the agent is back on track, producing properly working code without the issues.
Some people just keep pushing through on bad approaches, without questioning it and then blame the agent for being unable to finish it.
2. ask an LLM to do the needful and never ever look at the results except to count LOC
In effect, I’ve always wanted a pair programmer agent, not a zero to one programming agent. Unfortunately models these days are mostly of the latter kind and it has caused a major disruption in the way I work. I’d much rather appreciate a small model making fast and specific edits that I ask if it, rather than ingesting 20 files to make changes, and then starting to write tests, etc.
I think seasoned developers, over time, learn how to work a code base and design components with well defined interfaces, where the implementation is isolated in small well contained classes. SRP etc. more junior programmers can work on those smaller components/services in isolation.
For me this also seems to be a productive way to work along side an agent. Break up functionally into well defined chunks, and let the agent work on each small problem. Take more of a lead in the architecture I suppose.
I definitely prefer (c). But I get why (b) can feel necessary. If your competition is using (b) there can be pressure to do the same just to keep up.
The software has a plugin API. I asked Muse Glimmer to recommend a plugin — it found one but I tested it and it didn't work for unclear reasons (among other things the software installed version is old, the plugin older). I then asked it to outline how to implement a simple word filter, it gave me an overview of some hooks that looked right from dim-and-distant-past recollection of reading the docs when I installed it. I asked it some questions, it did the research.
I then set it off generating the skeleton of a filter plugin, went to the shops to buy food, came back and worked through filling it in and finishing it off. There was a bug. It found the solution.
It's only about 100 lines of code but it is a random old webapp and it had to look stuff up to finish it, and I think it did rather well. All on my Mac.
I am deeply cynical of the one-shot code, "nobody codes anymore" hype culture idea and that distaste put me off AI and agentic coding for ages. Like you, I want an assistant but as a freelancer I have to stay in control. I have no interest in the "just specify loops" BS and it will be bad for my business anyway.
I worry about code that I don't have a good working overivew of, and I worry that I might forget what I have done (I have pretty bad issues with focus and memory). But in this particular case, I don't really care if I forget, because there's documented code and I have no intention of specialising in this app. So it was a nice little test case.
I also don't really want to sit around waiting for Qwen 3.8 27B on this machine. Muse Glimmer is fine, actually. Gets to the solution as quickly as Qwen 3.6 35B-A3B.
This gives me a little hope that local AI will give me the sort of responsive developer sidekick I actually want.
-- PHB
The current trend in state-of-art LLM coding agents is giving more output, thinking longer and checking the results more to catch mistakes. Be it an economics inventive to make users burn through their quota or show increase in usage for shareholders, or a market demand of users liking the ability of models to do independent work without intervention or oversight; the result is what the article seem to call the Vibe Tax.
I myself asked Claude code recently to review a somewhat large PR, to see what it would find. I didn't expect much, but also didn't quite realize how the model would interpret my request; I burned $20 in 3 minutes in API usage, as it ran 2 sub-agents which themselves spun up 5 more each. Most sub-agents were manually checking for things clang-tidy would catch without actually calling clang-tidy. This behavior rose as i changed from sonnet/opus 4.6 to 4.8 and now 5.0.
I don't want to run a agent independently in this way; i ask targeted questions about specific things and review the result. But model development is targeted towards a more hands-off "vibe" workflow, because that's where the money and hype is. As a result, i find the models more frustrating, less trustworthy and more costly to my work. (I've even started using haiku more, since it remains to-the-point without steering away from what i ask)
That said, the companies are incentivized to sell you tokens, and therefore to have the models use as many tokens as they think you'll let them get away with for a given task / level of performance.
As if this wasn't bad enough, it also was not smart enough to regenerate the evidence in these contracts as it changed the underlying source code. So it would get in a loop where it would update code -> commit -> 15 minutes later CI would error citing the contracts weren't updated -> it would fix the contracts -> 15 minutes later CI would error because the fix was wrong -> it would fix the fix and commit -> 15 minutes later contracts would fail -> contracts were fixed again and this time maybe 30 minutes later it would pass, maybe it errors again.
This loop could go on all day every day if someone wasn't paying attention because the agent has no concept of time or wasted work. It's an AI livelock of sorts, but it will eventually converge in my experience. It'll just take 10x longer (literally like 20+ hours) than if you just intervene and tell it knock it off, so it feels like lighting money on fire (hence the tax).
That's why I feel like this vibe coding stuff has to actually be monitored, like a Tesla system -- because like a Tesla system it cannot be trusted to not crash into the proverbial code wall.
If the general idea is that these agents write too many tests, sure I guess? ‘Too many tests’ doesn’t sound like a failure case of engineering to me; typically software has had too few tests. Also, a lot of the power of these agents is their ability to self-verify and correct, which the test loop is a part of.
Nobody is making you pay this supposed tax. Just tell it not to write tests.
So I'd rather micromanage the process step by step. It takes more of my time, but the result is much, much closer to what I actually wanted.
They handle edge cases, catch bugs, and write tests that I'd never write.
Even if, however, this leads to the average piece of software improving, this one-shot complexity has the same issues as any large project. The more code, the longer it takes to steer the ship.
This "rising tide lifts all boats" mentality will make exceptional software even rarer than it is today.
Excited for the Roller Coaster Tycoons of tomorrow[0].
[0]https://en.wikipedia.org/wiki/RollerCoaster_Tycoon_(video_ga...
I now have my own link catalog, read-latter app and an RSS reader. Tailored to work exactly how I like. Hardened, with automated backup, and external users for the RSS app. It works. It takes learning, some knowledge of terms and very high-level practices, plus design thinking, but I haven't written a line of code for these.
I'd say that's the correct way to describe frontier models. They were trained with reinforcement learning based on human feedback. And obviously, humans prefer the bug-free variant. That's why models are now super verbose and spam tests like crazy. In their training environment, tokens were effectively free. And the humans that got asked never saw the price. If you ask people to choose the better offer and both are free, you end up with bloat. It's like people over-filling their plate at a buffet, then leaving leftovers. Except in this case, it's AI models burning through your wallet.
In case it helps anyone, this is how I did describe my desired harness, from scratch. I didn't know nor wanted to write all the ".vscode/skills" files, or the AGENTS.md file or any of that, so I asked Opus to "write a Harness and all related skills as needed, to follow this procedure on absolutely every change"... It (at least on VSCode) already comes with a harness/agent creation skill by default, so it has the ability to write a very good standarized process for you.
I've been playing with AI seriously for the first time, with a Python app that reads a spreadsheet with investment bookkeeping records and generates a pre-filled tax form. The harness prompt was somewhat like this:
----
1. A "Technical Spec Writer" subagent notes down every requested change to a SPEC.md file. This spec includes functional and behavioral descriptions, together with detailed technical documentation, includes software architecture, data models, API boundary definitions, etc. It then reviews everything for inconsistencies, mistakes, and text consolidation opportunities.
2. A "Tax Law Expert" subagent makes a due diligence review of the spec corpus, and raises any concerns it has wrt. what the actual Law mandates vs. what the spec docs say. Any concern is a blocker which gets documented and must be resolved by the owner (me) before proceeding. Ask me for clarifications, rulings, reference documentation, etc. as needed.
3. A "Software Engineer" subagent takes the spec and implements it. Reviews for obvious mistakes, variable misuses, unhandled errors. Finally, reviews the code to find DRY or refactoring opportunities.
4. A "Quality Assurance" subagent makes a final pass on the code, ensuring full compliance of the codebase with the specification. Also, tests are passed and verified.
----
I would have never imagined how deep the "Tax Expert" would make me go until "it" was satisfied with the results. The resulting spec is by no means a replacement of a human expert reviewing the tax declaration, but I am 98% confident that much more than the "happy path" of what I particularly want to cover is actually right. It asked me for clarifications or references (actual URLs so it could read them) to jurispridence on corner cases that I had not even anticipated for my own declarations.
In comparison, the actual "software engineering" must have been like 15% of the time/tokens.
It definitely helped me do a much deeper dive on the legalese than I would have done otherwise when writing something like this. (Still no replacement for an actual expert)
* the dev wants to describe an app in natural language then fall asleep while an AI works on it
* the dev wishes that the same AI would write less comprehensive tests
What exactly is a vibe coder to this dev?
(If you give claude an inch these days it'll just steamroll through a whole program of work without checking what it should be doing - a kind of overenthusiastic pull towards the first draft that is often detrimental and definitely wastes tokens, and even for very basic tasks it's using many more tokens than it should because it's doing this full belt and braces thing for everything, just in case you're an idiot).
But also... it's something you can easily reign in if you want to.
I've been able to get such positive returns out of LLMs. I am working for 3 different remote jobs concurrently with it, I've shipped a few apps thats doing six digits a month, I found a life partner after I used LLM to really work on myself. I am also experimenting with hardware prototypes and will likely have funding to launch it all with LLMs.
Why am I able to get so much out of "vibe coding" but others seemingly do not? I am not a genius, I am not a artisan, I am just very persistent and clear on what I ask LLMs but more importantly I don't try to place any other sort of unrealistic expectations on what it can and can't do.
You read comments on HN and read these articles and you might come across feeling a sense of peril and doom which are all completely fictional for the most part. A lot can be achieved with LLMs, much more than what the constant doomers will try to drag you down to.
if i wanna write a web-app or python script; the models are better than ever. If i want to fix a specific bug in a established and trusted legacy cobe-base; Haiku 4.6 does a better job than Opus 5.0, because it does what it's told and nothing more.
The author wanted a todo-list starting-point; realistically 200 rows of html+CSS without the back-end. Heck, they may not even want to make a todo-app, but thought a todo-app would be a decent starting-point. So why would we ever want a model to spend a weeks worth of tokens on everything except the request the user asked? This is not a cost issue; this is a control issue.
Choose what makes you happy!