Upgrading only PyTorch without matching vLLM or SGLang custom kernels
Serving frameworks compile custom fused attention and quantization kernels against exact PyTorch and CUDA headers.
Operator task guide
CUDA 13.0 (cu130) and CUDA 13.5 toolkits require a Linux NVIDIA driver on branch 580 or newer on the host unless container forward-compatibility libraries match the compute capability and kernel interface.
Never mix cu128 PyTorch or FlashAttention extension wheels inside a CUDA 13.x runtime image without checking ABI symbols and bundled libcudart versions first.
Reviewed . Reference guidance is not a diagnosis of your workload.
Inspect the host kernel driver version and confirm the active libcuda.so interface supports CUDA 13.0 or 13.5 runtime calls.
nvidia-smi --query-gpu=driver_version,name,pci.bus_id --format=csv
nvcc --versionPrint torch.version.cuda, _GLIBCXX_USE_CXX11_ABI, and loaded shared objects before rebuilding custom CUDA extensions from source.
python -c "import torch; print(torch.__version__, torch.version.cuda, torch._C._GLIBCXX_USE_CXX11_ABI, torch.cuda.get_arch_list())"
pip list | rg -i 'torch|vllm|sglang|flash|triton'Confirm installed binaries include native SASS cubins for your GPU compute capability so inference engines do not stall on runtime PTX JIT compilation or abort with no kernel image.
python -c "import torch; print(torch.cuda.get_device_capability(0))"| Signal | What it means | Next action |
|---|---|---|
| CUDA driver version is insufficient for CUDA runtime version | Host driver is below 580.x for a native CUDA 13.x container without valid compat libraries | Upgrade the host driver to branch 580+ or pin the container image to cu128 / cu126 matching the host. |
| undefined symbol: _ZN3c10... after cu130 upgrade | Extension wheel (flash-attn, sgl-kernel, vllm C++ ops) linked against a mismatched PyTorch or CUDA minor version | Use /tools/cuda-pytorch-compatibility to select matched PyTorch, CUDA index, and ABI builds. |
| no kernel image is available for execution on the device | Binary wheel omitted sm_100 (Blackwell B200) or sm_90 (Hopper H100) code generation flags | Install a wheel built with TORCH_CUDA_ARCH_LIST including your device capability. |
Serving frameworks compile custom fused attention and quantization kernels against exact PyTorch and CUDA headers.
Forward-compatibility packages only bridge documented datacenter hardware and library boundaries; mismatch triggers immediate initialization failures.
Generate verified pip, uv, and Docker base commands for CUDA 13.0, 13.5, and 12.8.
Troubleshoot B200 hardware, NVLink, and software bringup failures.
Resolve user-space vs kernel driver splits after unattended upgrades.
Native CUDA 13.0 and 13.5 toolkits expect NVIDIA Linux driver branch 580 or newer on bare-metal hosts unless an officially supported CUDA forward-compatibility package is mounted cleanly in the container.
Mixing major CUDA toolkit ABIs across compiled extensions such as flash-attn, vllm, or sgl-kernel frequently produces undefined C++ symbol errors or segmented memory faults. Match the exact CUDA index across every native wheel.
Use the three free diagnoses to review your error and relevant evidence. Keep reference guidance separate from the cause and recovery status of your own workload.
Diagnose your incident