An AI fix you cannot reproduce is a guess with a diff attached
Give a coding agent a stack trace and it will write you a fix. It will be syntactically valid, stylistically consistent with the surrounding code, and accompanied by a confident explanation. Whether it addresses the failure is a separate question, and the explanation is not evidence either way.
This is the part of autonomous repair that is easy to skip, because skipping it makes a demo much faster.
What a stack trace does not tell you
A trace says where a program stopped. It does not say what state got it there, which request triggered it, or which of the last forty commits made it possible. An agent handed only a trace is pattern-matching on the shape of the error.
Sometimes that is right. The problem is that you cannot tell which times from the patch alone, and neither can the agent.
Reproduction turns a guess into a test
The loop we settled on:
- Attribute the failure to the deploy that shipped before it started.
- Build a container at that exact revision.
- Replay the captured request against it.
- If it does not fail — stop, and say so.
Step four is the one that earns the rest. A failure that will not reproduce at the revision blamed for it means the attribution is wrong, and repairing on a wrong attribution produces a change that touches the wrong file for a reason nobody will be able to reconstruct in six months.
When it does reproduce, something more useful has happened than confirmation: the reproduction becomes the acceptance test. The repair is not finished when the agent stops writing. It is finished when the replay that failed now passes.
This changes what "verified" can mean
Every claim in our record carries how it was established:
- verified — a check ran and was watched to pass
- observed — telemetry says so
- inferred — a model concluded it
- approved — an identified human signed off
- unresolved — not known, and said so
Attribution is inferred. It is a model looking at a trace and a diff. That is exactly why reproduction exists: it converts an inferred claim into a verified one, or it refuses.
Without the reproduction step, "verified" can only mean "the tests still pass" — which says nothing about the failure you set out to fix. A repair can leave every test green and the original error occurring at the same rate.
The honest cost
Reproduction is slow, and it does not always work. Some failures depend on production data you should not copy, on timing, or on a third party that behaves differently on a Tuesday. Roughly, the ones that reproduce cleanly are deterministic failures in request handling — which is a large share of what actually pages people, but not all of it.
When reproduction fails, the honest output is unresolved and a
stop. A tool that degrades to "repaired it anyway" the moment verification
gets inconvenient has traded the only thing that made it trustworthy for a
higher completion rate.
The question to ask any autonomous-fix tool
What did you run, and did you watch it fail first?
If the answer is a diff and a paragraph of reasoning, it is a suggestion. That can still be useful. It is just not the same thing as a repair, and the difference matters most at three in the morning when nobody is reading carefully.