A plan the user checked off comes back the next day. The logs say the write succeeded and the file was written; it was just written onto a projection.
A user unchecks a plan item in the app, and the next day it is back. The logs show the write succeeded, and the file confirms it was written. The problem is where it was written. plan.json is the source of truth and plan.md is a human-readable projection; the app’s check endpoint updated only the md, while the backend regenerates md from json every hour and overwrote the change. Each side’s logic was fine on its own; the md file meant for reading should never have been writable.
I found the second example much later. One day I noticed two identically named data directories sitting in the workspace, one at the project root and one under core/. The one at the root held 24 historical events; the one under core/ held four recent ones.
The cause was a path computed one level short. The module lives under core/services/dual_role/, which is three levels below the project root, but the code went up two. It had been copied from another module sitting in core/utils/, where two levels lands you home and here it only reaches core/. From that day on the data went to the wrong place, without a single error along the way. Worse, persistence used a full overwrite, so the copy at the root was never appended to again and froze at its historical state. The project already had a get_dual_role_data_dir() helper. This module didn’t use it.
The third is about metadata. One conversation’s chat history landed correctly in the new character’s own directory, while the scope field on every event still named the old character. The directory pointed one way and the metadata another, and everything downstream doing permissions or retrieval by scope went wrong with it. The write path used a dynamic registry and the metadata used a hardcoded legacy label map.
In all three, the code looked fine on its own: it ran, nothing errored, yet the same data existed twice in the system. The second copy usually starts as a temporary compatibility shim and becomes something nobody dares remove. The same project had several more of these: sleep state had two determinations, one from the scheduler phase and one from a good-night keyword in chat, capable of opposite conclusions during the day; one activity’s display text had two parallel maps, so adding an activity meant editing two places and missing one leaked a raw English key into the UI; and there was that line in the documentation reading 307 total when an actual count gave 517, because a summary is a projection too and nothing guaranteed it would ever be recomputed.
01Label everything
Since then I make myself label every concept first: source of truth, projection, or cache. A comment or a doc line will do, somebody just has to know which one it is. The test can be cruder still: delete this data and see whether the system grows it back by itself.
- It comes back: it is a projection. Projections derive one way only and must never be written to.
- It doesn’t come back: it is the source of truth. The whole system gets exactly one write path.
- It comes back slowly: it is a cache. It needs invalidation keys and must be droppable wholesale.
Being careful next time doesn’t work. What I do now is make a second implementation fail the checks: define the same concept name in two modules and CI fails. It’s crude, but it doesn’t rely on anyone remembering.
And one more that gets used far more often: for any bug where the user changed something and it got changed back, don’t start with concurrency. Ask whether that endpoint writes to the source of truth or to a projection.
Caches are still worth using. But if deleting a cache loses data, it has already become a second source of truth.