vLLM throughput collapses as the scheduler preempts and recomputes the same requests

When KV cache blocks run out, the vLLM scheduler evicts in-flight sequences and recomputes them later. Under sustained load the recomputation consumes the cache and compute that would have retired other requests, so evictions beget evictions and a large share of GPU time goes to work that has already been done once.

A stable degraded regime in which vLLM's KV cache is oversubscribed, so the scheduler repeatedly evicts and re-prefills the same sequences and spends an increasing share of GPU time on repeated work rather than on completing requests.

The scheduler will admit more sequences than the cache can serve, because admission and capacity are governed by different limits. When the cache runs dry it must evict something, and eviction throws away decode progress rather than pausing it. The evicted request comes back needing a full prefill, so the cache pressure that caused the eviction is reproduced by the recovery from it.

• Throughput falls sharply while GPU utilisation stays high, so the card looks busy and productive
• Latency becomes erratic, with individual requests taking many times their usual time
• Nothing errors, so no alert fires and no exception appears in the logs
• Adding load makes total completed requests go down rather than up

• vLLM V1, where RECOMPUTE is the default preemption mode because it is cheaper than swapping in that architecture
• vLLM V0, where the same warning names PreemptionMode.SWAP instead
• Deployments admitting more concurrent sequences than the KV cache can hold at the served context length
• Long-context workloads, where each sequence reserves capacity for its full length

Apply the following checklist to a small reproduction: each box below is a positive signal that you are looking at this exact failure rather than a sibling in the same taxonomy.

• ✓Scrape vllm:num_preemptions_total twice a minute apart. A counter that keeps climbing under steady traffic indicates a thrashing loop rather than an occasional spike.
• ✓Halve --max-num-seqs and re-measure completed requests per second. If total throughput rises while concurrency falls, the deployment was thrashing.
• ✓Compare average generation throughput against prompt throughput in the periodic stats line. Prefill work far exceeding what the arriving traffic can account for is recomputation.

WARNING: Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space. This can affect the end-to-end performance. Increase gpu_memory_utilization or tensor_parallel_size to provide more KV cache memory.
INFO: Avg prompt throughput: 2140.0 tokens/s, Avg generation throughput: 41.2 tokens/s, Running: 48 reqs, Preempted: 31 reqs

Timestamps and exact values vary across runs, but the pattern. An info-level start, an early WARN, an ERROR carrying the symptom. Is the actual fingerprint you should alert on. The Denpex platform flags this combination automatically.

Reducing the admission limit means the working set fits, so nothing has to be evicted and no work is repeated. Enlarging the cache can also work, but it moves the threshold rather than removing the feedback, so a traffic increase puts the deployment straight back into the loop. Capping concurrency is the change that makes the regime unreachable.

// Typical pattern:
import torch.optim as optim
optimizer = optim.AdamW(model.parameters(), lr=3e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_training_steps)
for step in range(num_training_steps):
    optimizer.zero_grad()
    loss = model(batch)
    loss.backward()
    optimizer.step()
    scheduler.step()

Adapt the snippet to your framework. The same pattern holds for PyTorch Lightning, Hugging Face Trainer, DeepSpeed, Megatron-LM, and vLLM training wrappers. Where the wrapper exposes a config flag (for examplelr_scheduler_type in Trainer), prefer the flag over the imperative API to keep the schedule declarative and reproducible.