Skip to content
Free developer tool · no signup · no API key

CUDA, PyTorch & FlashAttention Compatibility Matrix & Resolver

Match your host NVIDIA kernel driver, container CUDA toolkit, PyTorch wheel index, and C++11 ABI flags. Generate deterministic pip, uv, and Docker install commands that pull prebuilt binaries instead of triggering 45-minute source compilations or runtime symbol crashes.

CUDA and PyTorch Compatibility Resolver

Verify driver versions, architecture support, toolkit versions, and installation commands instantly.

Compatibility Status

Configuration is fully compatible for NVIDIA H100 Tensor Core with PyTorch 2.5 and CUDA 12.4.

COMPATIBLE

Installation Commands

Standard Pip
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
UV Pip
uv pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
Docker Base Image
pytorch/pytorch:2.6.0-cuda12.4-cudnn9-runtime

How CUDA driver forward compatibility works in containerized workloads

Every GPU workload relies on two separate driver layers that are often confused during incident triage: the host kernel-mode driver (nvidia.ko and libcuda.so.1, mounted from the host into containers by the NVIDIA Container Toolkit) and the user-mode CUDA runtime (libcudart.so, bundled directly inside your PyTorch wheel or container image).

1. Minor-Version Backward Compatibility (CUDA 12.x Family)

Starting with CUDA 11.0 and continuing across CUDA 12.x, NVIDIA guarantees minor-version compatibility within the same major release family. However, PyTorch wheels also ship bundled user-space libraries such as cuBLAS, cuDNN, and NCCL that query minimum driver entrypoints. For production reliability without extra compat packages, keep the host driver at or above the minimum version listed for your target CUDA runtime below.

2. Datacenter Forward Compatibility (cuda-compat)

When a shared Kubernetes or Slurm cluster runs a locked enterprise driver branch (for example R535 LTS) but your container requires a CUDA 12.6 or 12.8 runtime, NVIDIA provides cuda-compat-12-* packages inside official nvcr.io/nvidia/pytorch images. These packages ship an updated user-space libcuda.so that bridges newer CUDA API calls onto the older kernel module on Data Center GPUs (V100, A100, L40S, H100, H200, B200). Note that forward-compatibility packages are not supported on consumer GeForce RTX drivers.

CUDA RuntimeMinimum Linux DriverPyTorch Wheel IndexTarget GPU Architectures
CUDA 12.8570.26+cu128Blackwell (B200, GB200 sm_100), Hopper (H100, H200 sm_90)
CUDA 12.6560.28.03+cu126Hopper (H100, H200 sm_90), Ada (L40S, RTX 4090 sm_89)
CUDA 12.4550.54.14+cu124Hopper (sm_90), Ada (sm_89), Ampere (A100, RTX 3090 sm_80/86)
CUDA 12.1530.30.02+cu121Hopper (sm_90), Ada (sm_89), Ampere (sm_80/86)
CUDA 11.8520.61.05+ (or 450.80.02+ with compat)cu118Ampere (sm_80/86), Turing (T4 sm_75), Volta (V100 sm_70)

Top 3 compatibility errors and their terminal signatures

Most broken GPU environments fail with one of three stack traces. Compare your terminal output against the signatures below to isolate whether the mismatch is at the kernel driver, C++ ABI, or SASS/PTX architecture layer.

1. Host Kernel Driver Older Than Bundled PyTorch CUDA Runtime

Terminal Signature

RuntimeError: The NVIDIA driver on your system is too old (found version 11080). Please update your GPU driver by downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver.

Why It Happens

The PyTorch wheel was installed from the default PyPI index (which defaults to CUDA 12.4+ in modern releases), while the bare-metal or Kubernetes host node is still running a 520.x or 470.x kernel driver that only exposes CUDA 11.8.

How To Fix It

Either upgrade the host NVIDIA kernel driver to branch 550+ for CUDA 12.4 support, install the cuda-compat-12-4 forward-compatibility package inside the container (supported on NVIDIA Tesla/Data Center GPUs), or reinstall PyTorch pinned to the cu118 wheel index.

One-Line Diagnostic Command

nvidia-smi --query-gpu=driver_version --format=csv,noheader && python -c "import torch; print('torch:', torch.__version__, 'cuda:', torch.version.cuda)"

2. FlashAttention C++11 ABI or PyTorch Minor Version Symbol Mismatch

Terminal Signature

ImportError: /usr/local/lib/python3.11/site-packages/flash_attn_2_cuda.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEi

Why It Happens

The installed flash_attn shared object was linked against a different PyTorch minor release (for example, torch 2.4 vs torch 2.6) or the wrong _GLIBCXX_USE_CXX11_ABI string representation (cxx11abiTRUE vs cxx11abiFALSE). PyTorch does not preserve C++ ABI stability across minor versions.

How To Fix It

Inspect your active torch.__version__ and torch._C._GLIBCXX_USE_CXX11_ABI, then install the exact matching prebuilt wheel URL from the Dao-AILab GitHub release assets using pip or uv with --no-build-isolation.

One-Line Diagnostic Command

python -c "import torch; print('Torch:', torch.__version__, 'CUDA:', torch.version.cuda, 'CXX11_ABI:', torch._C._GLIBCXX_USE_CXX11_ABI)"

3. Missing Compute Capability Architecture (sm_89, sm_90, sm_100)

Terminal Signature

RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.

Why It Happens

The GPU hardware compute capability (such as sm_90 on H100 or sm_100 on B200) was not included in TORCH_CUDA_ARCH_LIST when PyTorch or a third-party CUDA extension was compiled. Consequently, the binary contains neither SASS cubin code nor PTX for your GPU.

How To Fix It

Move to a PyTorch wheel built with a newer CUDA toolkit that supports your GPU generation (CUDA 12.4+ for Hopper sm_90, CUDA 12.8+ for Blackwell sm_100), or recompile custom extensions with TORCH_CUDA_ARCH_LIST explicitly set to your target architecture.

One-Line Diagnostic Command

python -c "import torch; print('Device capability:', torch.cuda.get_device_capability(0), 'Compiled arch list:', torch.cuda.get_arch_list())"

FlashAttention wheel ABI matching rules

Unlike pure Python packages, flash-attn links directly against PyTorch's internal libc10.so and libtorch_cuda.so shared libraries. Every prebuilt wheel encodes four strict ABI tokens in its filename. If even one token does not match your virtual environment, either pip falls back to a heavy source compilation (which frequently exhausts host RAM when MAX_JOBS is unset) or Python throws an undefined symbol error at import time.

Anatomy of a Prebuilt FlashAttention Wheel Tag

flash_attn-2.7.4.post1+cu12torch2.6cxx11abiFALSE-cp311-cp311-linux_x86_64.whl

1. CUDA Major Family Tag (cu118 vs cu12)

FlashAttention-2 groups CUDA 12.1, 12.4, 12.6, and 12.8 builds under the unified cu12 wheel tag because CUDA 12 maintains minor-version ABI stability. Verify that torch.version.cuda starts with 12. when pulling a +cu12 wheel.

2. PyTorch Major.Minor Tag (torch2.4, torch2.5, torch2.6)

PyTorch C++ symbol mangling and internal ATen dispatcher tables change between minor releases. A torch2.5 wheel cannot be imported under torch 2.6.0. Patch releases (such as 2.6.0 to 2.6.1) remain compatible with the torch2.6 tag.

3. Dual-ABI Flag (cxx11abiFALSE vs cxx11abiTRUE)

GCC's _GLIBCXX_USE_CXX11_ABI controls how std::string and std::list are mangled in C++ symbols. Official PyPI and download.pytorch.org wheels use cxx11abiFALSE. Custom source builds and NVIDIA NGC containers typically use cxx11abiTRUE.

4. Python Tag & CPU Architecture (cp310 to cp312, x86_64 vs aarch64)

Prebuilt wheels target CPython versions (such as cp310, cp311, and cp312) on linux_x86_64. On ARM64 Grace Hopper (GH200) or Grace Blackwell (GB200) nodes (aarch64), verify whether an ARM64 wheel is published for your release or compile with MAX_JOBS=8 pip install flash-attn --no-build-isolation.

Frequently asked questions

Does my container CUDA version need to match the host nvidia-smi CUDA version exactly?
No. The CUDA version shown in nvidia-smi represents the maximum CUDA runtime API version supported by the host kernel driver (libcuda.so). A container or PyTorch wheel can bundle an older CUDA runtime (libcudart.so) and run without issues. On datacenter GPUs such as the A100, H100, and L40S, NVIDIA CUDA forward-compatibility packages also allow newer CUDA 12.x toolkits to run on older production driver branches.
Why does pip compile flash-attn from source instead of downloading a prebuilt wheel?
Prebuilt FlashAttention wheels require an exact four-way match across your Python minor version, PyTorch major.minor release, CUDA major family, and the C++11 ABI flag (_GLIBCXX_USE_CXX11_ABI). If any of those four properties drift, or if ninja and packaging are missing when building metadata, pip falls back to compiling CUDA kernels from source with nvcc.
How do I check whether my installed PyTorch uses cxx11abiTRUE or cxx11abiFALSE?
Run python -c "import torch; print(torch._C._GLIBCXX_USE_CXX11_ABI)" in your environment. Official PyTorch wheels from download.pytorch.org and PyPI almost always return False (requiring cxx11abiFALSE wheels), whereas NVIDIA NGC PyTorch containers typically return True (requiring cxx11abiTRUE or the pre-installed container extension).
Do I need a full CUDA Toolkit installation on my host machine to run PyTorch on GPU?
No, not for running precompiled models or standard inference. Official PyTorch wheels bundle their own CUDA runtime, cuDNN, cuBLAS, and NCCL shared libraries. You only need a full nvcc CUDA Toolkit matching your PyTorch CUDA version when compiling custom C++ or CUDA extensions such as FlashAttention, Apex, or custom Triton/CUDA kernels from source.
Which CUDA and PyTorch versions are required for NVIDIA Blackwell (B200) and Hopper (H100) GPUs?
NVIDIA Hopper (H100, H200, compute capability sm_90) requires CUDA 11.8 or newer, with CUDA 12.4+ and PyTorch 2.4+ recommended for FP8 and FlashAttention-3 kernels. NVIDIA Blackwell (B100, B200, GB200, compute capability sm_100) requires CUDA 12.8 or newer and PyTorch 2.6+ or current nightly builds compiled with sm_100 kernel images.

Related GPU infrastructure tools

Continue diagnosing and configuring your training or inference stack with these zero-signup utilities.

View all developer tools
Triage & Root Cause

GPU & Distributed Training Log Analyzer

Paste raw NCCL timeouts, CUDA OOM tracebacks, Xid hardware errors, or FlashAttention build logs to pinpoint the failing rank and fix.

Open tool
Capacity Planning

LLM VRAM & KV Cache Calculator

Estimate activation memory, optimizer states, and KV cache footprint across FP16, BF16, FP8, and INT4 quantization before launching a job.

Open tool
Cluster Orchestration

Slurm Multi-Node SBATCH Script Generator

Generate production-ready sbatch scripts with torchrun, NCCL IB/RoCE flags, container mounts, and automatic fault-tolerant requeue traps.

Open tool