Good stuff.
> "It’s the record of what the code was supposed to do ... written down somewhere a human or an agent can read them back. Ours are markdown files in git..."
I keep getting surprised how many quite technical people I know are doing real work with AI but have never touched markdown.
If you are one of those people, allow me to offer a suggestion:
Stop. No more sending questions to ChatGPT or Claude until you do the following:
- Install Typora on your Windows or Mac machine.
- Write a .md (markdown text) file a full page long using Typora (or another editor that understands markdown, but Typora will do just fine). Make use of Heading 1, Heading 2, and underline formatting, at least.
- Make the one page file be a complex request to an AI. Something like "research x, compare it to y, make a guess what will happen in the future, here are some of my opinions and guesses, make your replies short, provide references".
Hand that .md file to your favorite AI. (Bonus points for using drag and drop or pointing the AI at the dir the file is in.) Look at your results. Make a change to your .md file. Hand it to the AI again, see the different results.
In my opinion, if you work on a keyboard and you've been making use of AI for the last 8 months, you should have done the above 4 months ago. Until you do, you will have a hard time understanding how drastically AI has changed almost all developer's lives, and how it is changing more and more fields that are not software development.
The crafting of the requirements is better suited to an interactive process. You will have superior results and a quicker turn around by having the AI research the change implications, what current implementations actually do and where (so you can look at the parts you want to), etc.
When working in a BIG codebase, having it detail up a flow of how all the parts fit together is great and helps get you setup to work. Then work through your planned constraints with the AI and make sure the perception matches your intent (with prevents a lot of issues that a fire and forget document creates).
Then when you are satisfied that it sees the problem as you do, have the model capture it and then let it rip.
I also really like telling the AI, "while we are hacking our way through the jungle trying to figure out how this thing should be built, at the same time, I want you to keep a design document up to date so that, if I asked you to rebuild the whole system again but without all the false starts etc we went through while we were figuring it out together, you could do so."
I probably won't use any of the "zombie clone" versions of my app that get built from the AI-generated design doc, but the fact that the zombie app (and design doc) look good at a casual glance, and the design doc is not 200 pages of AI bloatware verbiage, seems like good evidence that if a human expert needed to step in and save the system from some big surprise, that design spec would be found to be exactly what that human needed to help find the problem.
Honestly using markdown is really inefficient. Code is much more expressive and terse. Why do you need all these markdown files when you have code already? I could see markdown if you can’t read code, but otherwise…
Don’t get me wrong docs are useful when code isn’t sufficient (for more esoteric things), but in many cases code is much better.
That's one of the things that has drastically changed, in my opinion. One way to look at it could be: in the past it was okay to have a minimal design spec doc, if your programmer wrote excellent code and comments, and the system wasn't too big.
But now, you can't complete if you try to do it that way while not using AI. You need to take all that effort that used to go into good code and good comments, and put it into a good spec doc, and hand that file to the AI.
So that once the AI has built your system, you can say either "hey AI, a bug just showed up in the widget area, go look at your strange spaghetti code of yours and the design doc and fix it", or, "hey AI, the whole sub-system seems to have gone critical mess, take the code, spec doc, lessons learned, and the following few high-level design decision changes, and re-write the whole thing!"
Apologies for bluntness, but there just isn't going to be much need in this world for people that are excellent at hand-building intricate watches, in a world where $20 quartz-movement watches have appeared.
This is inefficient. If quality matters at all, you’re going to miss things in your English design doc and/or use 100 words where 1 line of code will do.
If quality (as in product, not code) doesn’t matter then sure, do whatever. LLM will fill in the gaps.
> world where $20 quartz-movement watches have appeared
Much like in real life, if you’re paying for a cheap knock off you get what you pay for. Also much like real life, plenty of people will pay for $20 knock offs and be content.
The real use for markdown for me and why i have a TON of markdown usage, is for planning tasks. these aren't "permanent" or anything but end up in a very much archived section of the repo once concluded but will have commit ids and such attached so intent can be retrieved at a later date if needed, as well as session ids for whatever model/harness i was using for that particular task.
It also serves another purpose of allowing me to nicely chunk up work as i see fit, so if i start a task in CC, and would prefer a section be done with some other model i can easily do so, or if i feel the work is suited to parallel sessions i can spin those up as well, with whatever models/harnesses i want.
I’ll get a rough markdown plan to start but don’t treat it as much more than a launch point.
Who's still typing?
I keep getting surprised how many people don't realize Linux is a thing.
Note that I said "quite technical people", not "fully technical people" or developers.
Houses are still standing built with hand saws and elementary-school level tools; we don't tear them down, just cut new pieces with power tools and bolt on new additions built with new pieces made from advanced tools.
Isn't AI just another purity test? nasty VSCode users instead of pure elite Vim users?
While the maintainers surely will be AI, the last bastion is the architects. Claude can write the scaffolding, but does Claude know how the parts go together? Even the concept of fuzz-testing existing means that even humans can't envision all the failure paths, so it would be unfair to hold the metric against AI. What we are going to see more of is the architectural failures: massive data breaches because the architecture aroung the breach will be reveaed to be laughably simple, from prompts that just didn't know what to ask.
I'm guessing that when he goes to debug a complex problem that spans across multiple systems for his app, his brain is paging through the equivalent of a well-written 3GiB markdown design document file.
And my opinion is, yes, pretty soon SOTAs will be able to be that guy, if we are willing to hand that much control to an AI. But I sure hope we don't--it seems like way too much power for an AI. However: Larry sitting next to Fable working together? A team like that could change the world. I hope he's digging in to Claude or something, I'll have to give him a call.