TAG
Measurement
Everything filed under this tag.
4 entries01Writing
The Synthetic Set Picked C. Forty-Four Real Questions Picked B.2026-10-27A synthetic evaluation set gave a clean answer to what granularity memory should be stored at. Real questions from real chats gave a different one — and along the way turned up three measurement problems.#Memory#Evaluation#Measurement#LLMReply Timing: How a Median of Eight Seconds Was Wrong2026-10-27Calibrating how long a role takes to reply against real chat logs, the first pass pooled more than a year of data and got the conclusion backwards. With the method fixed, it turned out the system was missing an entire middle tier.#Agent#Product#MeasurementSample Age, Catch-up Sampling, and Measurement Validity2026-10-22In periodic sampling the probe eats into its own window. A 756 ms interval means the curve describes the probe, not the GPU.#Measurement#GPU#SystemsA Failed Batch Stays Failed2026-10-22A run whose failures.jsonl is non-empty can never become a passing record. Fix it and rerun under a new run id; do not fill the gap.#Measurement#Data#Systems