A run whose failures.jsonl is non-empty can never become a passing record. Fix it and rerun under a new run id; do not fill the gap.
This continues “Sample Age, Catch-up Sampling, and Measurement Validity”, in the same EdgeSched experiment chain. The previous post covered when a single sample counts; this one covers whether a failed run can be repaired.
In this experiment chain a run does not produce a summary table. It produces a fixed set of artifacts: run config, manifest, per-iteration JSONL, events, timeline, summary, CSV, and failures.jsonl. The rule about them is single and strict: a non-empty failures.jsonl disqualifies the run permanently. No amount of backfilling changes what that run is.
The reasoning is short. A repaired harness is not the harness that failed. Backfilled iterations run on the new code while the failed ones ran on the old, and nothing in the curve marks the difference. Plot them together and you get a line that never existed as a whole.
So the cost of fixing is made explicit: the matrix is rerun under a new run id. The resume mechanism accepts only identical parameters, source commit, Python version, platform, and model manifest; existing (context, config, run) combinations are skipped, and records outside the matrix or duplicating one are rejected. The design separates backfilling from rerunning mechanically rather than by discipline.
A sibling prohibition: curves from different telemetry backends are never spliced. CLI-polling curves and ctypes direct-NVML curves belong to separate families; merging them yields no publishable threshold, only a line of unclear provenance. The legacy batch was retired wholesale, with three artifact sets deleted outright on 2026-09-17, their numbers no longer citable for anything.
One more boundary separates two kinds of gates. A sampling-jitter threshold is a performance target under test, not a validity gate, so the formal matrix must run in observe mode: offending iterations are written to performance_target_checks and kept in the full curve, never hidden or rewritten into passes. A validity-gate failure is a different event: the console prints the failed check name and the iteration counts as failed. Confusing the two either buries bad data or mistakes a slow system for an untrustworthy one.
Provenance is immutable in the same way. git archive HEAD packs committed content only, without modified or untracked files from the work tree, so using it as a dirty-source snapshot yields false provenance; the status entry count must be read live rather than copied from the previous run. Dirty-run artifacts are for debugging only; formal evidence is rerun on a clean commit.
Every number that reaches the paper passes the same promotion conditions: real backend with no fallback, frozen parameters, complete iterations, manifest present, consistent aggregation, rebuildable charts, traceable citations. The cost is far higher than running and believing. The benefit is singular: each number still points at the run that produced it.