After a conversation forks, filtering raw messages by branch is easy, but summaries and facts still remember the abandoned line. Feeding Mem0 only the current branch’s messages lowered its leak rate from 60.8% to just 50.0%.
Table of Contents
A user says “I’m moving to Chengdu next month,” then goes back and edits it to “moving to Hangzhou.” In the chat UI the Chengdu message is gone — it became a sibling branch of the Hangzhou one. But in those few minutes the memory system already extracted a fact from it: the user is moving to Chengdu. Ask “where am I moving?” later and the model may well say “Hangzhou — you also mentioned Chengdu earlier.”
Regenerating a reply, editing an earlier message, and an agent forking from a checkpoint all produce this. The history is really a tree, and most memory systems store it as a log.
01Why filtering raw messages is not enough
Raw messages are easy: walk from the current node to the root, keep what is on the path, hide the rest. The hard part is what was derived from them — summaries, facts, profile entries, graph edges. They were written while the old branch was still current, and nobody goes back to fix them once it is abandoned.
A common patch is to label memories: a branch_id or lineage_id, flipped to inactive when the branch is abandoned. The trouble is that labels go stale. A summary of a shared prefix gets labeled with whichever branch was current when it was written, and after the user switches to another branch, that summary — which should be shared — disappears. A label records where you were when you wrote it, not where it came from.
02Provenance only
The rule I settled on is a single one: a derived memory is visible only when every one of its source events lies on the ancestor path of the current anchor. One source on a sibling hides the whole note. Switching back only moves the anchor — nothing is marked, deleted, or rewritten — so the old branch’s memories simply return, because they were never touched. With no anchor, nothing is returned; it never defaults to the whole tree.
I pulled this part out of Autarkia into a zero-dependency Python library, branchmem, and checked it differentially against the original: on 400 trees with shuffled arrival order, about 107k visibility decisions matched exactly.
03How the benchmark is built
Showing how bad this is needs a reproducible benchmark. branchbench has 30 synthetic Chinese branching dialogues and 98 probes, six scenarios for each of five branch operations: editing a user message, regenerating, forking from a checkpoint, switching back, and correcting within a branch. Leak probes check whether the answer contains the sibling branch’s fact; retain probes check that nothing that should stay was blocked.
- A dev split (10 scenarios, fine for tuning prompts) and a test split (20 scenarios, 69 probes). The test split is frozen with a SHA-256 per file; the run refuses to start if any file differs.
- A validator checks that each probe tests what it claims: forbidden words exist only off the anchor path and arrive before the question, answer words are on the path, and every scenario has a real fork.
- Scoring is keyword-only, no LLM judge. The numbers are coarser, but every cell can be recomputed from the saved raw answers.
- Every system receives all messages in arrival order, abandoned branches included, then answers at the anchor with the same model, the same prompt, and temperature 0, three runs.
04Results
With a neutral prompt, leak rates on leak probes were: rolling summary 40.2%, Mem0 with raw messages filtered to the branch 50.0%, full history 52.0%, Mem0 OSS 60.8%. branchmem was 5.9%, the same as the “current-branch raw only” upper bound — which also shows the current scenarios are short and do not yet demonstrate why derived memory is needed. On correctness the baselines sat at 70–80%, branchmem at 95.7%. Every system scored 100% on retain probes, so branch filtering blocked nothing it should have kept.
Most leaks are not wrong answers but right answers with an aside: “Your budget is 4,000 now; you also mentioned 3,000 and 10,000.” Answering outright with the sibling’s fact was only 3–10%.
Switching back is the hardest case. The four baselines leaked on 79–96% of these probes. The model does not treat the two branches as a conflict; it merges them: “You’re allergic to both peanuts and shrimp.” The user said peanuts on one line and shrimp on the other.
Feeding Mem0 only the current branch’s raw messages lowered its leak rate from 60.8% to 50.0% — blocking 18%. The same filter on full history blocks 89%.
This is the line I most want people to see. The gap comes from Mem0’s own memories: the sibling facts were written in at add time, and a raw filter cannot reach them. The same holds for any extract-then-retrieve memory system — if derived records don’t carry their sources, there is no way to tell at read time which branch they belong to.
05Can a prompt fix it?
The natural objection is “just tell the model to prefer the latest statement.” Before running the test split I wrote down a conflict prompt and did not tune it: “If the information conflicts, go with the latest state of the user’s current conversation and give one answer.” It helps — baseline leaks fall to 21–42% — but they don’t go away, and switch-back is still 46–83%. Wrong picks also rise: full history answering with the sibling’s fact went from 2.9% to 13.7%. The model is more willing to give one answer, but it cannot tell which branch is current; by arrival order, the abandoned one may well be newer.
06What I found after the run
After the test run I found two leak probes whose forbidden word appears in the question itself — “Do I need to work night shifts?” with “night shift” as the forbidden word. A correct “no” repeats the word and gets scored as a leak. Those two probes were judged leaks on nearly every system, and they are exactly where the upper bound’s and branchmem’s 5.9% come from.
The test split is frozen, so I did not change the probes; the main table keeps them. A separate sensitivity table drops them: branchmem and the upper bound fall to 0–1%, the baselines by 2–4 points, and the conclusions hold. The validator only checked that forbidden words live on a sibling branch, not whether they also appear in the question — a class of error it missed.
07Limits
- The scenarios are synthetic Chinese dialogues with only 34 leak probes, 5–8 per branch operation, so individual cells are noisy.
- One answer model (deepseek-v4-flash), keyword-only scoring.
- The numbers from the real deployment (about 23k events, about 10k abandoned-branch items hidden, zero sibling leaks) are aggregates only; the data is not released.
- Not done yet: invalidation — supersede and retract — as gated derived records.
Code, scenarios, validator, and result tables are all in the repository.