In periodic sampling the probe eats into its own window. A 756 ms interval means the curve describes the probe, not the GPU.
Besides Autarkia I work on EdgeSched, a scheduling research prototype: on a single GPU an LLM has to stay resident, image generation arrives in bursts, and CPU work backfills the gaps, and the question is how they coexist. The main output of the experiments is residency curves — memory and utilization over time — and the conclusions go into a thesis, so every curve first has to prove it can be trusted.
A GPU residency curve samples memory and utilization at a fixed period. The first question about such a curve is not whether the numbers look plausible but how far each sample sits from the moment it claims to describe. This experiment chain calls that distance sample age, caps it at 750 milliseconds, and voids an entire batch when any sample exceeds the cap.
The gate exists because of one measured distortion. Early GPU telemetry polled the nvidia-smi CLI in a subprocess. In a long matrix of configs, contexts, and rounds, the driver query at the 4096-token context point showed a 512 ms tail, close to the sampling period itself. After each query the sampler still waited for the next fixed grid point, stretching the effective interval to 756 ms. The curve still looked complete and smooth, but the probe’s own latency had leaked into it.
01Three layers of repair
The fix has three layers, and the gate fails without any one of them. The first replaces the backend: ctypes calls NVML directly, no subprocess is started, and the manifest, summary, and per-iteration checks are required to record gpu_telemetry_backend=direct_nvml with gpu_cli_subprocess_used=false. Silent fallback to the CLI is prohibited.
The second is catch-up sampling. Once a periodic probe overruns the next tick, it samples immediately on completion instead of waiting for the following grid point, and every catch-up increments gpu_deadline_recovery_count, making the observer effect itself an audited quantity.
The third is field ordering. The memory reading is queried after utilization, power, and temperature, so the sample timestamp sits as close as possible to the memory value, and the remaining fields contribute less systematic skew.
One validity rule is easy to miss: enough samples and a correct backend do not make a batch valid. A batch on direct NVML with a full sample count still fails if every sample inside the window is stale. A correct backend and valid data are two separate verdicts.
The measurement infrastructure is held to the same gate. The resource timeline does not start until the first GPU sample lands, and the first CPU sample is marked invalid and excluded from aggregation. One two-second acceptance run required 41 of 41 points available, with a measured maximum sample age of 422.719 ms against the 750 ms cap, before the harness itself could be called verified.
What these gates prove is narrow: every number on the timeline genuinely occurred near the moment it claims. They do not prove that a configuration performs better, and they do not claim to capture fluctuations faster than the sampling interval. The 750 ms figure is a trade-off between validity and sampling cost, not a statement about GPU behaviour.