Most of the conversation about AI and software is still stuck on one question: can the model write the code? It can. It has been able to for a while now, and it gets noticeably better every few months. If your mental picture of AI-assisted development is still “will it produce something that works,” you’re arguing about something that has already been settled.
The question that actually decides whether these tools help or hurt on serious software is a different one, and a much harder one. Once the AI has written something, how do you know you can trust it enough to keep it?
That sounds like a small distinction. It isn’t. It’s the whole game, and it gets a fraction of the attention, because it’s a lot less exciting than watching a feature appear out of a prompt.
Let me explain why it matters so much, and then what I’ve actually found works.
Why trust is the bottleneck, not generation
Think about the difference between a weekend project and a system a lot of people depend on for a long time.
On the weekend project, what happens if the AI writes something subtly wrong? You find out fast, the blast radius is tiny, and fixing it costs you an afternoon. Trust barely matters, because being wrong is so cheap. You can just try things.
On a system that has to keep working (real users, real data, years of accumulated decisions) the math flips completely. A subtly wrong change can sit there quietly and surface months later as corrupted data, a security hole, or an architectural decision that makes everything after it a little harder. Being wrong stops costing an afternoon and starts costing something large, delayed, and often impossible to trace back to the moment it was introduced.
And here’s the trap: the AI is equally confident in both cases. The code comes back plausible, well-structured and confident-looking whether it’s right or catastrophically wrong, because the fluency is constant and the correctness is not. So on serious software the effort doesn’t move from you to the machine so much as relocate — out of typing and into judgment. Out of “writing the code” and into “verifying I can trust the code,” which on a complex system was often the harder and more expensive half to begin with.
So the naive version of AI-assisted development (prompt, accept, ship, repeat) doesn’t scale to high-stakes work, not because the code is bad, but because it quietly transfers unverified risk into your system faster than you can check it. You end up with more to review, less context on each piece (you didn’t write it), and a tool that is cheerfully generating in whatever direction you pointed it — including off a cliff.
The teams that will use these tools well on software that matters aren’t the ones with the best prompts. They’re the ones who build an environment where the AI’s output can be trusted by construction and verified cheaply, so that going fast doesn’t mean flying blind.
Here’s what that has actually looked like for me: four things, none of them about writing better prompts.
Make the channel narrow
The first is resisting the urge to let the AI touch everything freely, and instead making it work through a narrow, well-defined channel.
The wider and vaguer the surface you let it operate on, the more places it can be subtly, invisibly wrong. But what if you route its work through a constrained interface, a strict configuration schema, a defined contract, a small set of shapes the output is allowed to take? Then a whole class of mistakes becomes impossible rather than merely unlikely, and the constraint does some of the checking for you.
This is just good engineering, honestly, and we’ve always known that narrow interfaces contain blast radius. AI raises the stakes, because now you have a fast, confident, tireless thing producing code, and the constraint is what keeps its confidence from becoming your problem. A narrow channel turns “I have to carefully review everything it could possibly have done” into “it can only do things of a shape I’ve already decided is safe.”
Make the changes cheap to review
The second is the one that has changed my workflow the most, and it’s less about the AI than about what surrounds it.
Why do people either white-knuckle every AI edit or, more often, quietly stop reviewing them? Not because they stopped believing in review. Reviewing is expensive and tedious, and under time pressure the review is the thing that silently gets skipped. And an unreviewed AI change on a serious system is exactly the unverified risk we’re trying to avoid.
So the highest-leverage thing I’ve done is make the review cheap and immediate. I let the AI edit fairly freely, and I can loosen the reins precisely because I see every change it made as a clean visual diff and go through them quickly, hunk by hunk. Because the review is fast, I actually do it. All of it. Which is the whole point: the goal was never to review less, it was to review everything without it being so painful that I cut corners. Pair that with frequent, small commits and you get a genuine balance — the speed of AI generation on one side, the review-and-verify discipline I actually want on the other, held together by the fact that checking its work costs me almost nothing.
The insight underneath it: the speed of AI development is only safe if the speed of review keeps up. Invest in making the review effortless and you can safely let the generation run fast. Skip that, and speed just becomes unverified risk arriving faster.
Let evidence do the trusting, not your eyes
The third is verification you don’t have to perform by hand: tests, and increasingly, AI-run tests.
Eyeballing a diff catches a lot, but not everything. Does this change actually behave correctly? Human review of that question is both fallible and exhausting. The more of it you can turn into something automated, a check that passes or fails on its own, the more your trust is grounded in evidence rather than vigilance. Lately I’ve found real value in having the AI run tests directly against my local development environment as part of the loop: it makes a change, exercises it, and I see whether it actually works, not just whether it looks right. That isn’t the only testing I do, and it doesn’t replace my own judgment, but semi-automating that verification layer means a whole class of “looks fine, is broken” problems gets caught by the process instead of by luck.
The principle: trust should be earned by evidence, not extended on the strength of confident-looking output. Every piece of verification you can automate is a piece of trust you no longer have to grant on faith.
Make suspicion a move, not a mood
The last one I used to describe as a posture. I’ve come to think that’s exactly what’s wrong with it.
The instinct is right. When the AI produces something and it looks clean and the change seems small and everything appears to just work, that is the moment to get more careful, not less. Some of the worst problems hide behind output that looks too easy.
But look at what I argued two sections ago about review. People don’t stop reviewing because they stopped believing in it. They stop because it’s expensive, so under pressure it silently gets skipped. I don’t get to make that argument about review and then quietly exempt suspicion from it. Vigilance is a mood, and moods don’t survive a busy Thursday.
And it fails at the worst possible moment. The dangerous point isn’t when the AI hands back something obviously strange (you catch that). It’s when the change works. Once the thing you set out to do is done, the attention attached to checking it evaporates, because finishing is what tells your brain to stop. Success is what switches the suspicion off.
So what do you do with an instinct that’s right but can’t be relied on? What I actually practice is a move rather than a feeling. When a change comes back smaller or cleaner than I expected, I make the AI account for it: this is so minimal it’s almost concerning — walk me through why this actually works. Not because I think it’s wrong. Because I want the reasoning on the table where I can see it, and because I don’t trust myself to remember to want that at five o’clock on a Thursday.
This isn’t cynicism about the tools. I use them constantly and they’ve genuinely changed how I work. It’s the ordinary discipline of anyone responsible for a system that matters, with one addition I’ve earned the hard way: the tool’s confidence is not evidence, and neither is my own sense that I’m being careful. Only the verification is.
What this actually is
Step back and notice what the narrow channel, the cheap review, the automated tests and the forced accounting have in common. None of them is about getting more out of the AI. They’re all about being able to trust what it gives you: preventing whole classes of errors, making verification so cheap you actually do it, grounding trust in evidence instead of appearances, and refusing to let confident output stand in for correct output.
That reframes the whole skill. The valuable thing isn’t prompting, which is easy and getting easier. The valuable thing is judgment about verification: knowing where the blast radius lives, what “correct” means for this system, which checks can be automated and which can’t, and which clean-looking answers deserve a second look. Where does that judgment come from? Not from the tool, but from having been responsible for software that had to keep working, and from paying attention to how it broke.
Which leads to the thing I keep coming back to. I’d rather state it and be wrong than leave it open and be safe:
As these tools get better at writing code, the scarce skill isn’t writing code, and it isn’t prompting either. It’s building the environment that lets you trust what they wrote — and being the person who can tell when the confident, clean-looking answer is quietly wrong.
So what would change my mind? The whole claim rests on one condition: that a human is still the one who ships. If verification itself ever becomes something you can hand off, a checker reliable enough to certify correctness rather than just flag suspects, then this is a skill with a shelf life like plenty of others. I don’t see that coming; the failure modes I run into today are the same ones as two years ago, produced faster and phrased more convincingly. But that’s the seam, and it’s where I’d look first if I’m wrong.
Until then, I don’t think that skill is going away, and if anything the better the tools get, the more it’s worth. But it’s a strange, hard-to-measure kind of value: the disasters you prevented, the risk you didn’t take on, the review you made cheap enough to actually do. Easy to feel. Hard to point at. Worth building anyway.