Your AI agents pass gates they should fail

Teams shipping LLM agents almost always add a check: a validator, a schema, an exit code, a review step. The check passes. The work looks done.

Then something wrong reaches production anyway, and the postmortem finds the check was never capable of catching it. It wasn't bypassed. It wasn't broken. It was asked a question that couldn't tell right from wrong.

This is the failure mode I work on, and it is badly under-measured because every symptom of it looks like success.

Evidence, from systems I run

5 fabricated values out of 81 — and the gate returned PASS, exit code 0 A model merging two columns invented values that were never in the input. Every invented value came from a closed vocabulary, so every token-level assertion the validator ran was satisfied. The gate did exactly what it was written to do, and the output was wrong.

A guard caught 1 item out of 120 It was in place, running, and green. Measuring what it actually caught showed it was a crutch, not a control — the coverage lived somewhere else entirely.

A usage counter was blind to 100% of the surface it was meant to measure It reported a routing rule at 0.03% adoption. The reader was never opening the files where that work lived. The number was an artifact, and it drove a month of decisions before anyone re-ran it.

Two independent classification passes agreed 84.4% — and 20 of 28 disagreements leaned one way Systematic, not random. A single pass would have mislabelled 19 items out of 180. Anything downstream treating one pass as ground truth inherits that bias silently.

The rule underneath all four: a check that has never been made to fail has not been shown to pass. Most teams test that their guard accepts good input. Far fewer test that it rejects bad input — and that is the half that actually protects you.

What I do

Engagement	What you get
Reliability audit 1–2 weeks	I take your agent pipeline and find where it reports success while producing wrong output. Deliverable: reproducible failure cases with the exact input, the green signal it produced, and why the existing check couldn't see it.
Gate hardening 2–4 weeks	I build the missing layer: deterministic checks, negative controls that prove each guard can fail, and comparison against the input where output should be a pure function of it. Zero-noise, fail-open, no added latency at generation time.
Adoption measurement 1 week	You wrote the practices doc. I tell you, with a number, how many of them your agents actually follow — read from the telemetry you already generate.

How I work

Evidence over assertion. Every finding ships with a reproduction. If I can't reproduce it, I tell you it didn't reproduce — including when that undercuts something I said earlier.
Negative controls by default. Every guard I build is shown failing on material it must reject before it is trusted on material it should pass.
Your stack, not mine. I don't sell a framework. The layer lives in your codebase and outlives the engagement.
Nothing leaves without a gate. I've built the approval and provenance layers I'd want applied to my own work — hash-locked artifacts, human confirmation on outbound actions, audit trails.

Where I come from

I run four production systems on a fleet of coding agents, with the enforcement layer I built underneath them. The measurements above are from that work, not from a lab. I write about it publicly — including the times the measurement itself turned out to be wrong.

Read the write-ups →

Start with the audit

Tell me about your agent pipeline and where you suspect it's reporting success it hasn't earned. If I don't think I can find anything worth your money, I'll say so before you spend any.

ahmadyaseen35@gmail.com