When you can’t “ship and see”

Most of what we know about shipping software assumes you can “watch it run”.

You put the change behind a flag. You send it to 1% of traffic. You stare at a dashboard for twenty minutes, then 5%, then 25%, and if the error rate moves you roll it back and nobody outside the building ever knows. Production isn’t just where the code ends up. It’s the last and most final test environment you have.

Almost everything written about AI-assisted development assumes this in some way. Generate more, ship faster, let the system tell you if you were wrong.

Now take that away.

Except when you can’t

I spent a decade building systems where the call being handled is someone’s worst day. You do not canary a 9–1–1 call. There is no acceptable error budget when the error is a person not reaching help. And there is no “we’ll catch it in prod”, because that means it already impacted someone.

This isn’t unique to emergency services: aviation, medical devices, rail signaling, industrial safety, and any field where the distance between “wrong” and “harm” is shorter than your rollback time.

In those systems, verification has to happen before the code runs rather than after. That work is expensive and mostly human. It happens one change at a time, and it has been the real constraint on delivery for as long as I’ve worked in them.

What agents actually change

Agentic tools make generation nearly free. Not just better, not just cheap. Free. The cost of producing a plausible implementation of a well-specified change has fallen further and faster than anything I’ve seen in twenty years. It would be naive of me to think that I can write better code than a team of staff engineers.

But if generation was 10% of your delivery cost and verification was 90%, making generation “free” buys you 11%. That’s Amdahl’s law.

What actually happens is worse than “no improvement”. The generation side speeds up, the verification side doesn’t, and the queue between them grows. More and more code arrives at a review process that was already the bottleneck, produced by something that doesn’t get tired and doesn’t self-limit. And even worse, it’s code and thought processes we may not understand.

Teams describe this as moving faster than ever, right up until somebody notices that nothing is reaching production any quicker than it did before the age of agentic development.

About that 17x

I’ve quoted a number publicly: roughly 17x throughput on scoped coding tasks. It’s real and it was measured. But “scoped” seems to carry a lot of weight and I think it needs to be explained.

The number makes sense when the change is bounded and the requirement is unambiguous. All nice, an engineer would be happy. With enough existing coverage it comes down to reading the code and running the test suite.

Where it begins to fail is where the correct behavior is defined by a standards document (~800 pages) and fifteen years of operational reality rather than a JIRA ticket. Now, the right behavior depends more on knowing what a dispatcher does at 3am rather than technical correctness.

Those are the changes that matter most. They’re also the ones where an agent will hand you something that looks pretty good (and passes the tests!), and is wrong in a way you only catch if you know the industry.

So the true version of the number is this: “17x on the half of the work that was never the hard part”.

So what do you actually build?

If verification is the constraint, that’s where the money goes (or should go). Concretely, roughly in order of payoff:

Tests derived from the requirement, not from the code. You ask an agent to implement something, then ask it to write the tests, and it writes tests that pass against what it just wrote. That isn’t verification, it’s just a form of confirmation bias with good coverage numbers. The test has to come from the requirement, independently of the implementation. Ideally it gets written first, by something that has never seen the code.

A way to see without deploying. This is a big one and the expensive one. If you can’t canary, you need a simulation environment fed by, ideally, recorded real traffic, where you can replay a day of production behavior against a candidate build and diff the outcomes. Most companies in this category eventually build this, usually years later than they should have. Agents make it urgent, because the volume of change needing verification goes up while your ability to verify stays flat. Sometimes, in the case of emergency services, this can be tricky, since a simulation environment is not a simple stream of JSON messages.

Origin. Know which code came from an agent and which didn’t. Not because agent-written code is worse. As I said, it’s probably better than my own! But it fails differently and we don’t understand it as well, and you want to direct attention accordingly.

Written architectural decisions. Agents are excellent at local correctness and completely blind to the decision you made two years ago and the reason for it (I would encourage you now more than ever to keep an engineering journal). If that reason lives only in someone’s head, an agent will cheerfully and confidently undo it. This is how you end up with a “You’re right to push back” in the middle of the night.

Cheaper review, not more thorough review. When more code starts piling up, the impulse is to review harder. That tightens the constraint. The move should be the opposite: make each review cheaper. In practice that means smaller changes with more context attached, and every mechanical check pushed onto tooling before a person ever sees the diff. What’s left for the reviewer is the part that actually needs a person. I wonder if this is what Apple is now facing with their App Store review process nightmares.

What I don’t know yet

Whether verification capacity can itself be automated, or whether it’s irreducibly human. Or to what extend (maybe there’s more gray and not all black or white). Some years ago, I wrote about that 10% still missing. I think we’re pretty close to 1% now: the loop is closing.

I can make review cheaper or shift more of it into tooling. What I haven’t managed to do is remove the person who understands what the system is for. Every time I’ve tried, the thing I lost was the judgment that catches the “could be, but incorrect” change. I suspect that’s a real limit for now rather than a gap in the tools.

So… what?

Most organizations adopting agentic development are investing almost entirely in the generation half. They will report large multiples based on how much code is being produced (hey, I will admit I have done it myself!), and they will plateau somewhere around 2x on actual delivery, because in an ordinary shop, generation is roughly half the true cost of shipping.

And the companies that get the real multiple will be the ones that were already forced, by regulation or by consequence, to take verification seriously long before any of this agentic engineering happened. The safety-critical people. Which would be a genuinely funny outcome, given we’re usually the ones accused of moving too slowly.

Ismael Arenzana led engineering at one of the top US NG9–1–1 providers. He writes about mission-critical systems, technology and everything in between at arenzana.org.