PROJECT · 02
EdgeSched
LLM residency, image bursts, and CPU backfill on a single GPU
EdgeSched studies how several local models share one GPU. The LLM stays resident to keep first-sentence latency low, image generation arrives in bursts, and CPU work backfills the gaps; all three compete for the same memory and compute, and the scheduler decides who runs first, who queues, and who falls back to the CPU.
The implementation splits into a control plane and an execution plane. On the Python side, GlobalTaskScheduler manages request lifecycle, admission control, backpressure, and cancellation, with task state written to an mmap backplane that can be recovered after a process crash. On the C++ side, ResourceIsolationScheduler is reached through pybind11 and owns native queues, workers, monitoring, and the llama.cpp KV cache. If the C++ extension is not built, the whole path falls back to Python.
Scheduling distinguishes tasks by phase: resident LLM work, burst image generation, and phases that can share the GPU each get their own lease. A migration cost model compares swapping out, waiting, and bypassing before moving a model between GPU and CPU. While the GPU is leased, CPU-affine work goes to the backfill queue.
The experiments produce residency curves of memory and utilization over time, and the conclusions go into a thesis, so the measurement itself passes a validity gate first: sample age capped at 750 ms, telemetry through direct NVML, and catch-up sampling with a counter. A run whose failures.jsonl is non-empty can never become a passing record; after a fix the whole matrix is rerun under a new run id. Two posts on this site, “Sample Age, Catch-up Sampling, and Measurement Validity” and “A Failed Batch Stays Failed”, describe these gates.
It is currently a research prototype with the thesis in progress. One formal comparison between hybrid residency and sequential execution is complete, and the hybrid setup leads on first-sentence readiness and application-level supply gap. Output text, sentence count, and token count are not yet controlled, and the device clock is still being verified, so the claims stay narrow rather than being stated as final.