The skill I learned is exactly as you describe: understand the system. For me, this means "boxes and lines", the components and how they interact. I've encouraged developers on my team to make sequence diagrams part of their deliverable, because it helps both of us put the code in the context of the system.
The boxes can be opaque because I know the boundaries and expected inputs and outputs. This tells me when I need to go in a box: is it sending a malformed message? I can trace that back into the code.
I think that's the skill we have to cultivate now that coding agents produce far more than we can review line-by-line. The system view is the map, and the boundaries tell us where to go hunting inside the box.
Even if you're relying on extremely thorough test and validation, well someone who understands things at a low level has to write said tests and validations.
So long as humans are legally liable, this approach will only work until someone gets sued. Which means human experts will need to still exist and be trained, perhaps their training will just have to be more explicit rather than gained through the act of building products. Companies will hate this.
How do juniors develop it? Yeah, that's the tough part. Those years spent slogging through very tedious work got me here. You have to exercise to build muscle. Maybe we need to find some way to let people exercise the skills without the hands-on slog? I'm honestly not sure, how are you seeing it handled where you are?
I'm not seeing post-AI training of juniors handled at all, anywhere. The closest thing is what my employer does where for liability reasons every command executed by an AI agent, even as simple as an ls, has to be manually approved by the developer. This has the side effect of walking you through the AI's analysis, which can be quite educational.
Given the general culture of corporate cost-cutting (the whole appeal of AI is it doesn't need a benefits package), it's possible we're heading into a future where people trained pre-AI turn into a dedicated guild of professional troubleshooters, demanding fat paychecks to parachute in and put out fires when a company's giant blob of agents fails and their in-house staff lacks the deep knowledge to figure it out.
Now the juniors are cooked anyway, but not because they didn't learn the skill of software development.
Yes, we should be building systems.
No, it’s not equivalent to managing a team, because the AI has no skin in the game.
A decision an AI makes is not equivalent to a decision a human makes because the consequences are not equivalent. An AI cannot be fired; cannot be sexually rejected; cannot disappoint its loved ones; cannot spend a cold night on the streets. A human who makes decisions with complete disregard to material consequences is a human you need to get off your team ASAP because they’re a ticking time bomb: a liability.
So, in building your systems, you need to be sensitive and aware to the fact that AI has no skin in the game.
You raise another point though: agents are not human. The things that drive agentic loops are not at all the things that motivate humans to do good work.
That's probably where the human-management skill ends in this new world. The techniques that get high quality from humans do not apply to agents.
The hiring restaurant knows you can and will walk as soon as you land another engineering gig.
The failure that actually matters here is product failure. Angry users. Security incidents. Downtime. Broken workflows.
Stressed and uncomfortable developers have, historically, been a leading indicator of product failure. Software is complex and when devs can't handle the breadth of ownership + the depth of expertise at the same time, it means bad work that leads to product failure. It's not clear that agents fail in the same way, or that our collective intuition about how and when to trust agents is sufficiently developed to predict product failure in the same way.
As anecdata: I know a bunch of growth-SaaS/last-batch-of-IPO companies where the devs are all stressed, the feature ship rate is up, and incidents are down. All at the same time.
So yes, quick learners and just-in-time depth is important, but so is intuition about where discomfort is going to produce true failure. The discomfort isn't a feeling to throw away, simply trusting that you'll learn everything in time. Ownership requires holding a model of the system in your head and proactively steering where you dive.
I think that the whole premise of that failure affects things is based on a older model. You kind of assume that:
1) users are able to easily gain information about alternatives. We have seen that since LLMs replaced search engines, we have much less churn.
2) users are not behaviorally addicted/pushed into patterns. We have patterns that utilize users to keep using the app by exploiting their cognitive weaknesses
3) users have the mental fortitude and stamina to learn new tools instead of accepting flaws in the one they are used to
Some failures and bugs in prod don't matters as much
I'll go out on a limb and guess that at a hyperscaler, shipping a bug to prod means hitting canary or 1% of traffic. Seems fine!
I think my point still stands, even as the shape of failures shift. There's certain ones that still hurt (data corruption, exploited security holes, etc.) because they're hard to notice. It might even be that the primary goal of ownership is to develop intuition about where those failures are likely to happen.
A small bug implemented by an agent in the way R-multiples, stop loss distance, or P&L heatmaps are computed will never be complained about by the user—either he will trade incorrectly and lose his money, or see that single off-by-one difference compared to exchange ledger and churn forever.
I hope you are aware that you are outright evil. I also hope that you will manage to stop justifying it as acceptable.
The real alignment issue isn't in the challenges getting AI to align to human values.
It's in humans understanding what is worth aligning to.
"He chose... poorly."
If this is the face of our industry, that might explain a lot of things...
Absolutely, and the feeling of discomfort is a signal that you need to dive, a signal that your mental models might need updating. I think that builds up over time through many dives
But I also care about the complexity classes of the solution. I prod it about the time and space complexity, and about cache locality.
I do a lot of black box testing: take the solution, see if it does what I expected, then bake in those expectations in a document.
I find that if I do those things, it produces better code than I could do in the same time.
I also think it's more or less the same as how I used to work, just more thorough and with fewer time-consuming, trivial bugs.
should we stop treating the idea that humans must possess some capability AI can never replace as the starting point for discussions about humanity's future?
I'm inclined to think we should.
Preserving human agency is a moral and institutional choice, not something that needs to be justified by human cognitive superiority
It's the problem of delegation: you can do more of something, but worse, the more people you bring in. LLMs are just a very cheap contractor with a zero liability clause and some unchecked sycophantic tendencies.
The issue is that what's considered "moral" by the majority of society does not stop those already in power from doing as they please (the institutions).
There is a fundamental inequity that exists in our society that allows those with more to have an outsized vote in determining how the finite resources (human/land/capital/etc.) on this planet are distributed.
If you don't know what your schema should be, then I think this is the dead canary in the vibe coding coal mine. Back up to the design phase and talk to the robot about the domain until you have a really strong conviction about its tuples and relations.
Data and its shape is one such rigid structure. It will stay with you long after the code around it has been replaced. You do want to be intentional about it and build intuition for it.
Once you have that in place, you can stretch out a web of agentic fabric across it, and pin it down with tests.
Code is soft nowadays. Data isn't.
Naming, in general, is very similar to what I expect from colleagues who have weak English-as-a-second-language skills.
You let it run too wild and you’re suddenly left trying to reason why your LLM is trying to add an interaction between an Intermediator and a FromSpinMaxWriter.
Reverse Gell-man? https://en.wiktionary.org/wiki/Gell-Mann_amnesia_effect#Engl...
Before agents that is what every high performing person I've ever worked with was like. It was the kind of person I wanted to hire when I was in charge of that sort of thing. Enough breadth to know where to dive deep and enough willingness and curiosity to dive deep.
One thing that puts me squarely in the Optimist camp is that such a map can often be created by AI just-in-time. Sure you might not know every detail of what it said, but you can simply follow up and build your understanding.
"The CI tool was flagging an issue, perhaps correct, perhaps incorrect. But this is a real system so I had to dive in."
By diving in he ultimately just uses the AI to fix the problem:
"So I told Codex to make SonarQube happy, which it did by refactoring and splitting things up into 10 files".
Why isn't there a "shallowbot", or some CI configuration that tries to fix CI mistakes once before humans dive in? Plus it's not that deep, it's just shit code that was fixable with a "make sonarqube happy" prompt. I think the author is ashamed to admit that he's as generalist as everyone else now.