Skip to content

Operator task guide

TensorRT-LLM, SGLang, and vLLM multi-GPU serving triage

Multi-GPU inference crashes across TensorRT-LLM, SGLang, and vLLM almost always fall into four buckets: TP/PP KV-cache startup starvation, custom all-reduce P2P/IPC failures inside containers, FP8/NVFP4 quantization workspace OOMs, or MPI/NCCL rank teardown deadlocks.

Capture the worker subprocess log where engine initialization first throws rather than the HTTP frontend returning 500 EngineCore died or MPI_ABORT.

Reviewed . Reference guidance is not a diagnosis of your workload.

Step-by-step method

  1. 1

    Separate KV-cache startup budget failures from CUDA illegal memory accesses

    In TensorRT-LLM (--kv_cache_free_gpu_memory_fraction), SGLang (--mem-fraction-static), and vLLM (--gpu-memory-utilization), memory headroom reserved at init must cover model weights, activation workspace, CUDA graphs, and communication buffers.

    nvidia-smi --query-gpu=index,memory.used,memory.total,pstate --format=csv
    python -c "import torch; print(torch.cuda.device_count(), torch.cuda.get_device_capability(0))"
  2. 2

    Verify container SHM and P2P custom all-reduce transport across TP ranks

    Tensor parallel inference uses heavy CUDA IPC and /dev/shm transfers. Missing --ipc=host or blocked P2P causes custom all-reduce hangs on H100/B200 containers.

    df -h /dev/shm
    export NCCL_DEBUG=INFO
    export VLLM_CONFIGURE_LOGGING=1
  3. 3

    Validate quantized engine checkpoints (FP8 / NVFP4) against GPU compute capability

    SM90 (Hopper) vs SM100 (Blackwell) kernels fail cleanly only when verified at load time; mismatched FlashInfer, Triton, or CUTLASS kernels raise illegal instruction or unsupported architecture traps mid-prefill.

    export CUDA_LAUNCH_BLOCKING=1
    # Test single-request deterministic prefill before enabling CUDA graphs (--enforce-eager / disable cuda graph)

Serving engine multi-GPU failure matrix

Signals, meanings and actions for tensorrt-llm, sglang, and vllm multi-gpu serving triage.
SignalWhat it meansNext action
TensorRT-LLM MPI_ABORT with out of memory during engine build or initWorkspace + KV cache + weights exceed HBM per TP rankIncrease --tp_size, reduce max_batch_size/max_num_tokens, or lower kv_cache_free_gpu_memory_fraction.
SGLang CUDA error: an illegal memory access in FlashInfer or RadixAttentionKernel SM mismatch, ragged page boundary, or corrupted RoPE/max_seq_len configTest with --disable-cuda-graph and inspect exact model config.json vs engine image release.
vLLM EngineCore died or ZMQ IPC hang between TP workersWorker GPU rank died on OOM, P2P IPC block, or Xid faultCheck worker rank stderr and dmesg Xid before restarting the API server loop.
Custom all-reduce hang on TP >= 4 in Kubernetes podContainer missing --ipc=host or sufficient /dev/shm sizeMount emptyDir medium: Memory (>= 16Gi for TP8) and test NCCL fallback.

Evidence checklist

  • Worker TP-rank stdout/stderr (not just HTTP API gateway logs)
  • Exact --tensor-parallel-size, --mem-fraction-static / --gpu-memory-utilization flags
  • Installed vllm, sglang, tensorrt-llm, torch, flashinfer, and CUDA versions
  • Host df -h /dev/shm and dmesg NVRM Xid records

Common mistakes

Debugging only the HTTP proxy 503/500 traceback

When EngineCore or an MPI rank dies, only the worker subprocess holds the CUDA kernel or OOM allocation size that killed it.

Sizing only model weights without CUDA Graph and FP8/NVFP4 workspace overhead

Serving frameworks pre-allocate activation graphs and cache pages upfront; verify multi-GPU headroom with the /tools/vram-calculator first.

Frequently asked questions

Why do TensorRT-LLM or SGLang TP8 servers hang on the first request in Kubernetes?

Multi-GPU serving relies on CUDA IPC peer-to-peer memory handles and /dev/shm synchronization between TP workers. Undersized Kubernetes /dev/shm or restrictive pod IPC settings block custom all-reduce buffers immediately.

How do I choose between increasing TP vs adjusting KV-cache utilization?

If model weights plus CUDA graph buffers leave under 10% free HBM per GPU, increase tensor parallelism. Otherwise adjust --gpu-memory-utilization (vLLM), --mem-fraction-static (SGLang), or kv_cache_free_gpu_memory_fraction (TensorRT-LLM).

Apply the method to your incident

Use the three free diagnoses to review your error and relevant evidence. Keep reference guidance separate from the cause and recovery status of your own workload.

Diagnose your incident