vLLM runs out of GPU memory during serving after the prefill token budget is raised

Raising the number of tokens vLLM may batch per step improves time to first token and throughput, and it also raises the peak activation memory a prefill step needs. When that peak collides with the memory already reserved for the KV cache, the failure arrives during serving rather than at startup, after the deployment has looked healthy.

A serving-time out-of-memory failure caused by the per-step prefill token budget rather than by the KV cache reservation, appearing only once arriving prompts are long enough to build a maximum-sized batch.

The batched token budget decides how much prefill work goes into one step, and activation memory for that step scales with it. That memory is not part of the pool the KV cache was sized against, so raising the budget quietly increases peak usage above what startup profiling measured. Profiling runs before any real prompt has arrived, so the configuration that fails under long prompts is one that was never exercised at its own limit.

• The server starts cleanly and fails only once real traffic arrives
• The failure correlates with long prompts rather than with the number of concurrent users
• A configuration tuned for throughput on one model fails on another of similar size
• Reducing the served context length does not help, because the prompts were already within it

• vLLM V1, where chunked prefill is enabled by default whenever possible
• Deployments tuned toward a batched token budget above 8192 for throughput
• Mixed traffic combining long prompts with many short decodes in the same step
• Configurations where max_num_seqs multiplied by max_model_len already consumes most of the cache budget

Apply the following checklist to a small reproduction: each box below is a positive signal that you are looking at this exact failure rather than a sibling in the same taxonomy.

• ✓Check whether the failure began with a change to --max-num-batched-tokens or --enable-chunked-prefill rather than with a change in traffic volume.
• ✓Correlate the failures against prompt length rather than request rate. A token budget only binds once a step is large enough to reach it.
• ✓Restart with the budget set to 2048 and replay the same traffic. Success at the lower budget with everything else unchanged confirms the step size rather than the cache was the cause.

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.83 GiB. GPU 0 has a total capacity of 79.15 GiB of which 2.11 GiB is free.
INFO: Chunked prefill is enabled with max_num_batched_tokens=16384.
WARNING: If out-of-memory occurs during cudagraph capture, consider decreasing gpu_memory_utilization or switching to eager mode.

Timestamps and exact values vary across runs, but the pattern. An info-level start, an early WARN, an ERROR carrying the symptom. Is the actual fingerprint you should alert on. The Denpex platform flags this combination automatically.

Lowering the token budget shrinks the largest step the scheduler can build, which lowers the activation peak that collides with the reserved cache. It costs time to first token rather than correctness, and it targets the quantity that actually changed, unlike lowering the served context length, which does nothing when the prompts were already inside it.

// Typical pattern:
import torch.optim as optim
optimizer = optim.AdamW(model.parameters(), lr=3e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_training_steps)
for step in range(num_training_steps):
    optimizer.zero_grad()
    loss = model(batch)
    loss.backward()
    optimizer.step()
    scheduler.step()

Adapt the snippet to your framework. The same pattern holds for PyTorch Lightning, Hugging Face Trainer, DeepSpeed, Megatron-LM, and vLLM training wrappers. Where the wrapper exposes a config flag (for examplelr_scheduler_type in Trainer), prefer the flag over the imperative API to keep the schedule declarative and reproducible.