The Synthetic Set Picked C. Forty-Four Real Questions Picked B.

Autarkia’s long-term memory runs once a night, compressing the day’s conversations into entries that can be retrieved later. This revision had to decide how big one memory should be. There were three candidates: A writes a one-line summary per message; B cuts the conversation into segments and writes one synopsis per segment; C keeps the raw turns searchable and additionally breaks each segment into atomic facts.

01The synthetic answer

The first comparison used a synthetic set I wrote by hand: 10 cases, 90 messages, 20 questions, 25 gold facts, covering knowledge updates, temporal reasoning, multi-session recall, and questions that should be refused. The result was clean. End to end, A answered 13/20, B 17/20, C 19/20; rerun on a different model, C still scored 19/20. I wrote “candidate shape = C” in the log, along with one caveat: twenty synthetic questions need to be checked against a real annotated set.

The real set was 44 questions drawn from two long-running conversation threads, with the same question types as the synthetic set. Every question records where its answer lives (session and event ID), and I reviewed each one myself. The set stays local and never enters the repository.

02All three options at 15/44

On the first real run, all three arms scored 15/44. The convenient reading is “granularity doesn’t matter” and move on. The result was invalid: extraction used a reasoning model, whose thinking tokens count against max_tokens, and I had carried over the synthetic set’s budget. Of arm C’s 237 calls, 169 returned an empty body — the whole allowance went to thinking. None of the three arms had enough allowance, so this round compared nothing at all.

Going back over the prototype turned up two more problems in the metrics themselves, both already present during the synthetic round:

• nDCG returned 1.0 when not a single relevant result was retrieved. The worst possible retrieval scored perfectly, so the nDCG in the synthetic report was inflated.
• Segments that arm C failed to parse were silently dropped, so its parse-failure rate was always zero.

After the fixes I set new budgets (A 1024, B 3072, C 4096), added hybrid retrieval combining BM25 and vectors, and gave every comparison a Wilson interval and an exact paired McNemar test. Forty-four questions is not many; without a test, a gap of two or three questions is easy to read as a finding.

03The real answer

The conclusion flipped. B was best under both BM25 and hybrid retrieval: with hybrid retrieval B answered 27/44, against A at 14/44 (p=0.002) and C at 20/44 (p=0.039). Two ablations pointed the same way. Keeping only atomic facts scored 15/44. Putting raw turns and synopses in one pool and taking top-k dropped to 20/44 — the raw entries crowded the synopses out.

Why the synthetic set picked C I can only explain, not prove: in hand-written cases each fact is stated explicitly, once, and atomic facts map almost one-to-one onto questions, so finer is simply better. In real chat a single fact is spread over several turns, wrapped in context, and sometimes corrected halfway. Atomizing loses the context, raw turns are too many and too fragmented, and one synopsis per segment ends up closest to what the questions actually ask.

Production now follows B: one synopsis per segment, and raw text no longer competes in the same pool — it is expanded from the source events after a synopsis matches. Hybrid retrieval comes in at a later stage.

The limits of this conclusion are on record too: 44 questions, sampled rather than full corpus, all answered by the same model. Knowledge updates scored only 5/9 and temporal reasoning 3/8; those two are weak under every granularity.

The near miss was the first real run. Three scores of 15/44 looked a lot like a result; had I believed it, I would have decided granularity didn’t matter and shipped C, the synthetic set’s pick.