vLLM throughput collapses as the scheduler preempts and recomputes the same requests

When KV cache blocks run out, the vLLM scheduler evicts in-flight sequences and recomputes them later. Under sustained load the recomputation consumes the cache and compute that would have retired other requests, so evictions beget evictions and a large share of GPU time goes to work that has already been done once.

Quick answer

Lower --max-num-seqs. The instinct is to add memory, but capping admission ends the loop directly, and no error will ever fire to tell you it is happening, only the preemption counter.

Symptom: WARNING: Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space. This can affect the end-to-end performance. Increase gpu_memory_utilization or tensor_parallel_size to provide more KV cache memory.
Root cause: Preemption is a safety valve that keeps the server up when the cache cannot hold every batched request. It is not free: an evicted sequence loses the decode work it had already done and is prefilled again from the beginning when it is re-admitted. That makes the failure self-sustaining.
Recommended fix: Lower --max-num-seqs so the scheduler admits fewer concurrent sequences than the cache can hold. Capping admission is the direct fix for a thrashing loop, because it stops the condition rather than enlarging the resource.
How Denpex helps: Denpex matches vLLM throughput collapses as the scheduler preempts and recomputes the same requests across every rank in a distributed run and reports which rank failed first, so you act on the initiating node instead of the loudest one.

Performance#vllm#preemption#recompute#kv-cache#scheduler#throughput

What this failure is

A stable degraded regime in which vLLM's KV cache is oversubscribed, so the scheduler repeatedly evicts and re-prefills the same sequences and spends an increasing share of GPU time on repeated work rather than on completing requests.

Live diagnosis, no signup

Is this what broke your run? Paste your log.

You're reading about vLLM throughput collapses as the scheduler preempts and recomputes the same requests. Paste your own crash log or traceback below and get the real root cause for YOUR run, not this generic entry. No account, no card. Logs are masked at ingress and never saved to account history.

training_logs.txt

No log to hand? Try one:

3 free diagnoses/day

Why it happens (the mechanism)

The scheduler will admit more sequences than the cache can serve, because admission and capacity are governed by different limits. When the cache runs dry it must evict something, and eviction throws away decode progress rather than pausing it. The evicted request comes back needing a full prefill, so the cache pressure that caused the eviction is reproduced by the recovery from it.

What you'll observe

Throughput falls sharply while GPU utilisation stays high, so the card looks busy and productive
Latency becomes erratic, with individual requests taking many times their usual time
Nothing errors, so no alert fires and no exception appears in the logs
Adding load makes total completed requests go down rather than up

Common symptoms and what they mean

Symptom	Why it happens
Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space.	Preemption is a safety valve that keeps the server up when the cache cannot hold every batched request. It is not free: an evicted sequence loses the decode work it had already done and is prefilled again from the beginning when it is re-admitted.
The same warning repeating continuously rather than appearing in occasional bursts	That makes the failure self-sustaining. Re-admitted sequences need cache and compute to redo their prefill, which is exactly the pressure that caused the eviction, so the scheduler evicts again. Once the loop closes, a growing fraction of each step is spent recomputing rather than retiring requests, and the fleet does less work the more traffic it is given.
A steadily climbing vllm:num_preemptions_total in the Prometheus metrics	It is easy to miss because nothing fails. There is no exception and no error rate to alert on; the only signals are a warning line and a counter, and utilisation stays high throughout because the GPU is genuinely busy doing the wasted work.
total_cumulative_preemption_cnt rising in the periodic stats line	Preemption is a safety valve that keeps the server up when the cache cannot hold every batched request. It is not free: an evicted sequence loses the decode work it had already done and is prefilled again from the beginning when it is re-admitted.

Which systems are affected

vLLM V1, where RECOMPUTE is the default preemption mode because it is cheaper than swapping in that architecture
vLLM V0, where the same warning names PreemptionMode.SWAP instead
Deployments admitting more concurrent sequences than the KV cache can hold at the served context length
Long-context workloads, where each sequence reserves capacity for its full length

How to confirm this is the problem

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

✓Scrape vllm:num_preemptions_total twice a minute apart. A counter that keeps climbing under steady traffic indicates a thrashing loop rather than an occasional spike.
✓Halve --max-num-seqs and re-measure completed requests per second. If total throughput rises while concurrency falls, the deployment was thrashing.
✓Compare average generation throughput against prompt throughput in the periodic stats line. Prefill work far exceeding what the arriving traffic can account for is recomputation.

Root cause

Preemption is a safety valve that keeps the server up when the cache cannot hold every batched request. It is not free: an evicted sequence loses the decode work it had already done and is prefilled again from the beginning when it is re-admitted.
That makes the failure self-sustaining. Re-admitted sequences need cache and compute to redo their prefill, which is exactly the pressure that caused the eviction, so the scheduler evicts again. Once the loop closes, a growing fraction of each step is spent recomputing rather than retiring requests, and the fleet does less work the more traffic it is given.
It is easy to miss because nothing fails. There is no exception and no error rate to alert on; the only signals are a warning line and a counter, and utilisation stays high throughout because the GPU is genuinely busy doing the wasted work.

The fix and how to prevent it

Searchable error signature

search key

WARNING: Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space. This can affect the end-to-end performance. Increase gpu_memory_utilization or tensor_parallel_size to provide more KV cache memory.
INFO: Avg prompt throughput: 2140.0 tokens/s, Avg generation throughput: 41.2 tokens/s, Running: 48 reqs, Preempted: 31 reqs

Use this text as a lookup key in logs and upstream issue trackers. It is not presented as a captured customer log. Confirm the cause from your own preceding events, versions, configuration and the cited references.

The fix and the prevention pattern

The root cause is on this page and stays free. A free account adds the exact remediation steps, saved history, and the fix on every entry in the encyclopedia.

No credit card. Daily allowance follows verified trust tier. Instant access.

Why the recommended fix works

Reducing the admission limit means the working set fits, so nothing has to be evicted and no work is repeated. Enlarging the cache can also work, but it moves the threshold rather than removing the feedback, so a traffic increase puts the deployment straight back into the loop. Capping concurrency is the change that makes the regime unreachable.

Best practices by model family

Model / Stack	Recommendation	Notes
Thrashing under steady traffic	Lower --max-num-seqs first	Ends the feedback loop rather than moving its threshold.
Context window far above workload need	Lower --max-model-len	Each sequence reserves capacity for its full declared length.
Genuinely needs the concurrency	Raise tensor parallel size or quantize the KV cache	Adds real capacity instead of trading against activation headroom.
Considering chunked prefill	Expect no memory relief from it	It addresses head-of-line blocking; capacity is still reserved for the full sequence length.

With the fix vs without the fix

Dimension	With the fix	Without the fix
How the failure announces itself	A warning line and a rising counter	Assumed an error would be raised
What GPU utilisation shows	High, because recomputation is real work	Read as evidence the hardware is saturated
Effect of adding traffic	Completed requests per second falls	Assumed to rise until a plateau

Diagnostic note

“This is the failure most likely to be mistaken for needing bigger hardware. Utilisation is high, the GPU is hot, and the obvious reading is that the card is saturated, so teams add replicas. Adding replicas does help, by splitting the traffic each one sees, which reinforces the wrong conclusion. Halving the admission limit on one replica and watching completed requests per second rise is the cheap experiment that settles it.”

Visual fingerprint

How eviction reproduces its own cause

  admit more sequences than the cache holds
            |
            v
     cache exhausted  ->  evict a sequence, discarding its decode progress
            ^                          |
            |                          v
  re-prefill consumes cache  <-  re-admit the evicted sequence

Recovering from an eviction requires a full prefill, which consumes the same cache whose exhaustion caused the eviction. Once traffic sustains the cycle, the proportion of each step spent on repeated work grows and completed requests per second falls.

Diagnose this failure in VS Code

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

Install the free VS Code extension

vLLM errors in context

vLLM failures cross request, model, KV cache, worker, kernel and host boundaries. The hub compares common literal signatures and the first control that separates their causal owners.

Compare every vllm error side by side

Related failures to investigate next

The same cache budget, exhausted at startup instead of during service, prevents the engine from starting at all. vLLM cache block allocation failure →

KV cache memory growth describes how the per-sequence footprint accumulates and decides how many sequences fit. KV cache memory growth →

Raising the prefill token budget to improve throughput trades against the same memory this failure is short of. vLLM prefill token budget out-of-memory →

Frequently asked questions

Questions engineers and on-call staff commonly ask about this failure.

Nothing is erroring. Is this really a failure?

Yes. The server stays correct and gets progressively less done. The only signals are the preemption warning and the counter, which is why it survives so long in production.

GPU utilisation is high. Do I need more GPUs?

Not necessarily. Recomputation is real work, so it keeps utilisation high while producing nothing new. Halve --max-num-seqs on one replica and see whether completed requests per second rises before adding hardware.

Should I just raise gpu_memory_utilization?

It buys cache but takes headroom from activations and graph buffers, and can convert preemption into a hard out-of-memory failure. Raise it incrementally and validate at peak load.

Will chunked prefill fix it?

No. It batches prefill chunks with decode and helps head-of-line blocking, but capacity is still reserved for each sequence's full length, so the memory pressure is unchanged.

References

Don't just read the fix, diagnose your run

The encyclopedia tells you what went wrong. Denpex tells you what went wrong in YOUR training run. With your logs, your config, and your stack.

Diagnose your logs free Compare a solved incident Browse all failures

Related Performance errors

CPU Affinity Misconfiguration

Performance · medium

GPU Utilization Low

Performance · medium

DeepSpeed ZeRO-3 Slow (Synchronous Param Prefetch)

Performance · medium

Unsloth Qwen3-30B-A3B MoE Fine-Tuning Extremely Slow / Low GPU Utilization

Performance · medium