CI stayed green for months while production broke weekly. The most useful thing those tests did at the time was reassure me.
For a while my CI was always green while production broke weekly. I made a list: tests were lying to me in six ways.
Mocks accept any argument you hand them. One renderer had the signature (self) and its caller passed an extra user_id. In the unit test a MagicMock stand-in took the surplus argument without a sound; in production it raised TypeError, an outer except swallowed it, and an entire block of context vanished. Mocks don’t check signatures, so at every seam between the code under test and the outside world at least one case has to run against the real object.
Then there are the ones nobody runs. The repo carried a batch of verification scripts covering critical paths, sitting outside pytest’s collection roots while CI runs pytest, so they stayed red for a long time without anyone knowing. Green CI only means the collected tests are green, a distinction I later wrote into the README. Same story, costlier version: I wrote ninety-eight cases for a tools module and discovered before committing that the file itself was not tracked, blocked by an ignore rule, so committing it would have turned CI red on a missing module instead. Ninety-eight cases, written for nothing.
Snapshots absorb bugs. Regenerating a baseline snapshot of a prompt makes the current output the expectation. If that output already carries a bug, the snapshot cements it, and every run afterwards comes back green, convincingly green. A snapshot can tell you the output changed, not that it was wrong to begin with, so every re-record needs somebody reading the diff.
Anything time-related is flaky by construction. Sleeping a fixed interval to wait on async work, depending on the real timezone, depending on whether it happens to be daytime: all three make the same code green today and red tomorrow. Inject time instead, so the code under test receives a now rather than calling one. Where that isn’t possible, monkeypatch, but the patch has to cover the whole execution including any deferred coroutines.
Coverage is not behaviour. After a split-and-refactor, line coverage can hit 100% because every line was imported and called, while the behaviour has quietly shifted: argument order, defaults, exception branches.
And there is the kind that won’t even let you finish a run. A shadow mode defaulted to on, so unit tests genuinely dialled an external API, and the first full run hung for thirty-four minutes without printing a single line; adding an are-we-under-pytest check brought it to one minute ten. Stranger still, the same command once hung for thirty-eight minutes, and the three runs after that took three minutes, thirty-seven seconds, and one minute forty-five. It never reproduced. You cannot write an assertion against that kind of thing. You can only remember it happened.
01The most expensive one
Once I fixed the proactive-care prompt and nothing about the behaviour changed. Eight files edited in rotation, two rounds of deduplication, three rounds of trimming; nothing moved. In the end I went and read the real user data file, and found a unit test writing into the actual plan.json. Not a temp directory, the production path, with an entry titled something like priority integration test, restored by a finally block. And in this environment pytest gets killed hard from outside often enough that finally never gets to run.
Counting them up, ten real plan.json files had been polluted over that stretch, leaving fifteen stray entries, one still marked pending. My prompt edits could not move anything because the schedule the model read had test entries mixed into it, and it was faithfully talking about those entries.
“The finally block will clean up” only holds when the process exits normally, and test processes in this environment were often killed outright.
Validating a tool-using agent has its equivalent: the easiest false pass is building a stand-in that lacks the mode marker, so the tool is never selected and the model replies in chat mode with something perfectly plausible. The test passes and the feature was never exercised. So for any change touching models, tools, or prompts, I do one real run at the end: a full persona, a real input, and watch whether it actually places that tool call. Unit tests cannot catch a path it never took.
These days I judge whether tests are enough by whether I dare to press merge after a refactor. If I don’t, what’s usually missing is the one test proving the behaviour stayed the same.