Debugging only the HTTP proxy 503/500 traceback
When EngineCore or an MPI rank dies, only the worker subprocess holds the CUDA kernel or OOM allocation size that killed it.
Operator task guide
Multi-GPU inference crashes across TensorRT-LLM, SGLang, and vLLM almost always fall into four buckets: TP/PP KV-cache startup starvation, custom all-reduce P2P/IPC failures inside containers, FP8/NVFP4 quantization workspace OOMs, or MPI/NCCL rank teardown deadlocks.
Capture the worker subprocess log where engine initialization first throws rather than the HTTP frontend returning 500 EngineCore died or MPI_ABORT.
Reviewed . Reference guidance is not a diagnosis of your workload.
In TensorRT-LLM (--kv_cache_free_gpu_memory_fraction), SGLang (--mem-fraction-static), and vLLM (--gpu-memory-utilization), memory headroom reserved at init must cover model weights, activation workspace, CUDA graphs, and communication buffers.
nvidia-smi --query-gpu=index,memory.used,memory.total,pstate --format=csv
python -c "import torch; print(torch.cuda.device_count(), torch.cuda.get_device_capability(0))"Tensor parallel inference uses heavy CUDA IPC and /dev/shm transfers. Missing --ipc=host or blocked P2P causes custom all-reduce hangs on H100/B200 containers.
df -h /dev/shm
export NCCL_DEBUG=INFO
export VLLM_CONFIGURE_LOGGING=1SM90 (Hopper) vs SM100 (Blackwell) kernels fail cleanly only when verified at load time; mismatched FlashInfer, Triton, or CUTLASS kernels raise illegal instruction or unsupported architecture traps mid-prefill.
export CUDA_LAUNCH_BLOCKING=1
# Test single-request deterministic prefill before enabling CUDA graphs (--enforce-eager / disable cuda graph)| Signal | What it means | Next action |
|---|---|---|
| TensorRT-LLM MPI_ABORT with out of memory during engine build or init | Workspace + KV cache + weights exceed HBM per TP rank | Increase --tp_size, reduce max_batch_size/max_num_tokens, or lower kv_cache_free_gpu_memory_fraction. |
| SGLang CUDA error: an illegal memory access in FlashInfer or RadixAttention | Kernel SM mismatch, ragged page boundary, or corrupted RoPE/max_seq_len config | Test with --disable-cuda-graph and inspect exact model config.json vs engine image release. |
| vLLM EngineCore died or ZMQ IPC hang between TP workers | Worker GPU rank died on OOM, P2P IPC block, or Xid fault | Check worker rank stderr and dmesg Xid before restarting the API server loop. |
| Custom all-reduce hang on TP >= 4 in Kubernetes pod | Container missing --ipc=host or sufficient /dev/shm size | Mount emptyDir medium: Memory (>= 16Gi for TP8) and test NCCL fallback. |
When EngineCore or an MPI rank dies, only the worker subprocess holds the CUDA kernel or OOM allocation size that killed it.
Serving frameworks pre-allocate activation graphs and cache pages upfront; verify multi-GPU headroom with the /tools/vram-calculator first.
Multi-GPU serving relies on CUDA IPC peer-to-peer memory handles and /dev/shm synchronization between TP workers. Undersized Kubernetes /dev/shm or restrictive pod IPC settings block custom all-reduce buffers immediately.
If model weights plus CUDA graph buffers leave under 10% free HBM per GPU, increase tensor parallelism. Otherwise adjust --gpu-memory-utilization (vLLM), --mem-fraction-static (SGLang), or kv_cache_free_gpu_memory_fraction (TensorRT-LLM).
Use the three free diagnoses to review your error and relevant evidence. Keep reference guidance separate from the cause and recovery status of your own workload.
Diagnose your incident