Skip to content
vLLM error triage

CUDA out of memory during vLLM serving

CUDA out of memory

CUDA allocation fails mid-serving after earlier requests succeeded. Capture the first traceback and request shape to isolate cache or concurrency pressure.

Source checked against vLLM v0.32.0. Log wording and available flags may vary by release.

First action

Capture the first CUDA traceback, per-GPU memory, request lengths, and active sequences. Reproduce on a canary while reducing one concurrency or batching limit, such as --max-num-seqs, and keep the request shape fixed.

Verify the change. Repeat the failing shape at representative concurrency and watch allocation errors, preemption, and request latency before widening traffic.

Diagnose your vLLM log offline

Synthetic demonstration samples

Synthetic demonstration samples. Not customer telemetry.

Upload your log file. Large files are processed in your browser. Only selected evidence is submitted when you run diagnosis.

Your logs are processed entirely in your browser and are never uploaded by this tool.

Add host kernel / dmesg or nvidia-smi context (optional)

100% local browser engine (same deterministic core as the Open VSX & VS Code extension). 0 bytes of log data uploaded.

Handling a production incident? Replay it with your team.