CUDA errors are reported asynchronously. A kernel launches, returns immediately, and fails later, so the error surfaces at whatever line happened to synchronise next. That is why the traceback so often points at innocent code.
The first move for almost any CUDA error is therefore the same: set CUDA_LAUNCH_BLOCKING=1 and re-run. It makes launches synchronous so the error is reported where it actually happened, at a large performance cost that does not matter while debugging.
The second thing worth knowing is that a CUDA error usually poisons the context. Once a kernel has faulted, subsequent calls on that device keep failing until the process restarts, so the cascade of errors after the first one carries no information.
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate ...
The allocator could not find a free block. The message reports total capacity, how much is already allocated and how much is free, and those three numbers usually explain it: fragmentation, not exhaustion, is a common cause when free memory looks sufficient.
First action: Read the numbers in the message before changing anything. If free memory exceeds the requested size, the arena is fragmented, and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True often resolves it without touching batch size.
CUDA error: an illegal memory access was encountered
A kernel read or wrote memory it does not own. An out-of-bounds index, a freed pointer, or a race. The context is destroyed, so everything after it also fails.
First action: compute-sanitizer --tool memcheck python train.py. It names the kernel and the offending access, which no Python traceback can.
An assert inside a kernel failed. In practice this is nearly always an index out of range, and nearly always an embedding lookup or a loss function receiving a label outside the valid class range.
First action: Run with CUDA_LAUNCH_BLOCKING=1 to get the real line, then check label and index ranges against vocabulary or class count. Off-by-one and an unmasked padding token are the classic causes.
The generic "the kernel died and CUDA cannot say why" error. It is a symptom of something else: often an illegal access, sometimes a hardware fault, occasionally a driver problem.
First action: Check dmesg for an Xid at the same timestamp FIRST. An Xid 48/79/94 next to this turns an ambiguous software error into a definite hardware one.
Misaligned address
Application bug
CUDA error: misaligned address
A kernel performed a vectorised load or store on an address that is not aligned to the required boundary. Common with custom kernels using float4 or half2 on tensors that are not contiguous.
First action: Call .contiguous() on the inputs to the custom kernel, and check any pointer arithmetic that offsets into a tensor.
No kernel image
Configuration
CUDA error: no kernel image is available for execution on the device
The binary contains no machine code for this GPU architecture. It was compiled for different compute capabilities than the one it is running on.
First action: Compare torch.cuda.get_device_capability() against the build. Rebuild with TORCH_CUDA_ARCH_LIST covering every architecture in your fleet.
CUDA driver version is insufficient for CUDA runtime version
The installed driver is older than the CUDA runtime the application was built against. The driver is on the host; the runtime usually ships inside the container, which is why this appears after an image update.
First action: nvidia-smi shows the driver version; the container shows the runtime. Either update the host driver or use an image built against an older CUDA.
CUDA error: operation not permitted when stream is capturing
Something inside a CUDA graph capture region did an operation graphs forbid, usually a synchronisation, a host allocation, or a .item() that reads back to CPU.
First action: Remove CPU-GPU syncs from the captured region: no .item(), .cpu(), print of a tensor value, or dynamic allocation.
Code asked for a GPU index that does not exist in this process. Nearly always CUDA_VISIBLE_DEVICES restricting the view while the code still uses a global index.
First action: Remember that CUDA_VISIBLE_DEVICES renumbers devices from 0. Inside the process, use the local index, not the physical one.
A kernel was launched with a grid or block size the device rejects, most often more than 1024 threads per block, or a shared-memory request above the per-block limit.
First action: Print the launch configuration and compare against the device limits from cudaGetDeviceProperties. Shared memory per block is the limit people forget.
The CUDA runtime could not initialise. In containers this is usually missing device nodes or a driver mismatch; after a fork it is CUDA being used in a child process that inherited a context.
First action: Check /dev/nvidia* exists inside the container. If it appears after forking, switch the multiprocessing start method to spawn.
GPU memory returned data ECC could detect but not repair. This is hardware, not code, and any result computed nearby is suspect.
First action: Drain the GPU and resume from a checkpoint written before the error. Check nvidia-smi -q -d ECC,ROW_REMAPPER and treat a remap failure as an RMA.
Most of these are diagnostics rather than fixes. If one makes a failure disappear, it has told you where the fault is, not removed it.
Environment variables and what each one does.
Variable
What it does
CUDA_LAUNCH_BLOCKING=1
Makes kernel launches synchronous so errors are reported where they happen. The first thing to set for any CUDA error. Slow, so debugging only.
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
Reduces allocator fragmentation. Frequently resolves an OOM where free memory already exceeds the requested block.
CUDA_VISIBLE_DEVICES
Restricts which GPUs a process sees, and RENUMBERS them from 0. The renumbering is the part that causes invalid-ordinal errors.
TORCH_USE_CUDA_DSA
Enables device-side assertions in PyTorch builds that support it, giving a more precise message than a bare device-side assert.
TORCH_SHOW_CPP_STACKTRACES=1
Adds the C++ stack to PyTorch errors, which matters when the fault is inside an extension rather than Python.
Frequently asked questions
Why does the traceback point at code that looks fine?
CUDA kernels launch asynchronously. The launch returns before the kernel runs, so a failure is only detected at the next synchronisation point, which may be many lines later and in unrelated code. Set CUDA_LAUNCH_BLOCKING=1 and the error will be reported at the line that actually caused it.
Is a CUDA error ever a hardware problem?
Sometimes, and the way to tell is dmesg. An Xid logged at the same timestamp turns an ambiguous CUDA error into a hardware one. "Unspecified launch failure" and "uncorrectable ECC" are the two most likely to be hardware; "device-side assert" and "illegal memory access" almost never are.
Why do I get a cascade of CUDA errors after the first one?
A faulting kernel poisons the CUDA context. Every subsequent call on that device fails until the process restarts, so only the FIRST error carries information. Everything after it is the context reporting that it is already broken.
I have free GPU memory but still get OOM. Why?
Fragmentation. The allocator needs a contiguous block, and repeated allocations of varying sizes leave the arena split into pieces that are individually too small. PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True lets the allocator grow segments instead, and usually fixes it without reducing batch size.
Does restarting fix a CUDA error?
It clears the poisoned context, so the job will start again. It does not address the cause, and if the cause was a bad index or a race, the error returns at the same point. If a restart genuinely fixes it permanently, suspect a hardware or thermal condition that the restart happened to reset.
Knowing the error is not knowing the cause
Which rank failed first, whether the checkpoint is safe to resume from, and whether this is your code or the hardware. Paste the log and Denpex answers all three, free and without an account.