Your Agent's Output Was Valid. It Was Still Wrong.

I ran a production agent that owned the end-to-end create and update flow for full-screen promotional modals on our team's internal content operations platform. It acted on real user-facing surfaces, and we gated its output with a set of evaluation checks. Nothing shipped unless the evals were green.

The promotion launched to users outside its intended group. What made it unsettling wasn't that it failed; it's that it passed. Every eval was green.

The reason was a gap we hadn't named. Our evals confirmed the agent produced a valid configuration: well-formed, structurally correct, internally consistent. But validity said nothing about containment. The eval implicitly assumed the human approver had reviewed the full setup and understood who would actually see the modal. That assumption was the hole. Reading the config told you what the agent intended. It couldn't tell you the blast radius: how many real users this would reach, and whether any of them were outside the target group. We were evaluating the artifact. The failure lived in its consequences.

The uncomfortable part: a human was already approving these launches before the leak. The gate existed. It still let the modal through.

This is the failure mode people gloss over when they say “human-in-the-loop” as if it settles the question. A human approving a raw config isn't reviewing the outcome. They're reviewing an artifact and silently trusting that it maps to what they intend. Configs don't announce their blast radius. The detail that mattered was easy to overlook precisely because nothing forced it into view. The approval was real; the information behind it wasn't.

A human gate is only as good as what you put in front of the human. Ours was asking someone to catch a problem we hadn't made visible.

We didn't fix this by making the agent smarter. We built containment around an agent we don't fully trust: three layers, each catching what the last one couldn't.

First, prediction. Before launch, the system estimates blast radius from the current config, historical runs, and dry runs where needed. The agent's intent became a number.

Second, a human gate with something to see. Where approval had been a rubber stamp on a config, the approver now gets a concrete claim: this promotion will reach roughly X million impressions. If that matches the intended group, they approve. If it doesn't, the mismatch is visible before anything ships, which is exactly what was invisible the first time.

Third, a runtime circuit breaker. A monitor watches the live promotion against the estimated threshold. Cross it, and the system pages on-call and opens an approval request to halt the rollout. The prediction that gates the launch also bounds it in production.

Two things I didn't believe before this incident.

Eval sets encode the failures you've already imagined. Ours checked everything we'd thought to check, and production still found the one dimension we hadn't: targeting containment. You will never enumerate every failure mode in advance, which means pre-launch evaluation is necessary and never sufficient. You also need runtime containment: something watching the live consequence, empowered to stop it.

And human-in-the-loop is theater unless the human is given the specific thing to catch. A gate that reviews an artifact instead of its outcome isn't a safety mechanism, it's a signature. The fix wasn't adding a human; the human was already there. The fix was putting the blast radius in front of them so approval became a decision instead of a formality.

Validity is not correctness. For agents acting on the real world, the output can be perfectly well-formed and still wrong in the one dimension you forgot to evaluate. Assume you forgot one. Build for the moment it shows up.