Environment error references

Browse 102 published references for environment failures in GPU training and inference. Match the message and context from your run, then open the reference for possible causes, evidence checks and sources. A matching message alone does not confirm the cause.

• accelerate CannotDynamicallyBSZError Batch size must stay constant across gradient accumulation

  Batch size changed between gradient accumulation micro-steps. Accumulation sums gradients across micro-batches before one optimizer step, so a varying batch silently reweights the average. Accelerate refuses rather than corrupting the effective learning rate. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent huggingface failures.

• Adam Epsilon Underflow in FP16 Mixed Precision

  When using FP16 for the optimizer state (or when the epsilon value itself is cast to FP16), the default Adam epsilon of 1e-8 underflows to 0 (since the smallest representable subnormal in FP16 is ~6e-5). When the moving average of squared gradients approaches zero, the denominator becomes exactly zero, causing a division by zero that yields NaN weights.

• Apptainer/Singularity --nv GPU Binding Failures

  HPC containers lose GPU access when --nv cannot locate host driver libraries (nonstandard driver install paths, missing nvidia-container-cli, ldconfig cache staleness) or when overlay mounts fail on kernels without unprivileged overlayfs.

• Attribute Error (Python)

  Python attribute errors crash training when accessing attributes that don't exist on objects.

• Axolotl ModuleNotFoundError After a Successful pip Install

  A git/editable install of Axolotl reports success, but importing axolotl or running accelerate launch -m axolotl.cli.train fails with ModuleNotFoundError. The src-layout package never landed on the path (or installed into a different environment). Reinstall editable from the repo root or symlink the package.

• bitsandbytes matmul_4bit CUDA error: no kernel image is available for execution on the device

  The installed bitsandbytes CUDA binary does not contain a 4-bit kernel for the GPU compute capability in use. This is an architecture or binary compatibility failure, not a model-memory shortage. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent quantization failures.

• bitsandbytes was compiled without GPU support libbitsandbytes_cuda.so could not be loaded

  bitsandbytes loaded its CPU-only build, or the CUDA build it selected does not match the installed toolkit or the GPU architecture. Quantized (8-bit / 4-bit) paths silently fall back or fail outright. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent huggingface failures.

• Conda Environment Conflict

  Conda environment conflicts cause package version mismatches, broken dependencies, and hard-to-debug import errors.

• Connection Timeout

  Connection timeouts prevent model downloads, data fetches, and inter-node communication from completing.

• Container CUDA Runtime Mismatch with Host Driver

  The containerized CUDA runtime fails to execute kernels because the host NVIDIA driver is too old to support it.

• could not select device driver with capabilities gpu nvidia-container-cli initialization error

  The container runtime could not select or initialize the NVIDIA GPU runtime. The host toolkit, runtime configuration, driver visibility, or requested GPU capability is missing before the application starts. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent containers failures.

• CUDA Driver / Runtime Mismatch

  CUDA driver and runtime version mismatches prevent PyTorch from initializing CUDA.

• CUDA error: invalid configuration argument awq_gemm_kernel grid block dimensions exceed SM limit

  The AWQ GEMM kernel was launched with a grid or block configuration the device rejects. AWQ kernels assume shapes divisible by their tile size; an unusual hidden dimension or a batch of 1 can produce a degenerate launch. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent quantization failures.

• CUDA error: operation not permitted when stream is capturing

  Attempting to perform CPU synchronization or stream switching (e.g..item() or dynamic tensor operations) while a CUDA graph is actively capturing operations. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent cuda-graphs failures.

• CUDA graph replay detected invalidated memory buffer

  A CUDA graph was replayed after the memory it captured was freed or reallocated. Graphs record raw device pointers, so any tensor whose storage moved between capture and replay leaves the graph reading memory that is no longer its own. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent cuda-graphs failures.

• CUDA out of memory during AutoencoderKL tiling decode intermediate latents

  AutoencoderKL ran out of VRAM while decoding tiled latent intermediates. Tiling reduces the peak for the main VAE operation, but overlap buffers, output assembly, dtype, and concurrent model residency still consume memory. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent multimodal failures.

• CUDA out of memory during Inductor CUDA graph execution allocator private memory pool fragmentation

  TorchInductor exhausted a private CUDA Graph memory pool even though total free memory may appear sufficient. Captured graphs retain stable addresses, so inactive blocks in the private pool cannot always satisfy a differently shaped allocation. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent torch-compile failures.

• cudaErrorLaunchFailure an illegal instruction was encountered sm90a

  A kernel executed an instruction the GPU does not implement. Almost always an architecture mismatch: the binary contains SASS for a different compute capability, or was JIT-compiled from PTX targeting features this GPU lacks. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent quant failures.

• cuDNN error: CUDNN_STATUS_NOT_SUPPORTED cudnnConvolutionForward

  cuDNN convolution operation failed, unsupported algorithm, channel layout mismatch, or GPU compute capability limit. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent multimodal failures.

• DataLoader Deadlock from Silent Worker OOM

  When a PyTorch DataLoader worker exceeds available memory (often due to small Docker `/dev/shm`), it is silently killed by the OS OOM killer. The parent process is waiting on a multiprocessing Queue for data from this worker. Because the worker died without sending an EOF or error state, the parent hangs indefinitely.

• DeepSpeed async_io/aio Won't Compile on Windows

  Installing DeepSpeed on native Windows fails building the async_io op because it depends on Linux libaio, which has no Windows equivalent. DeepSpeed targets Linux; use WSL2 or a Linux host, or build with async_io disabled and avoid NVMe offload.

• DeepSpeed cpu_adam Build Fails: 'cusolverDn.h: No such file or directory'

  Building DeepSpeed's cpu_adam op fails with 'fatal error: cusolverDn.h: No such file or directory' and 'Error building extension cpu_adam'. PyTorch's conda package shipped its own nvcc that shadows the system CUDA toolkit, so CUDA headers aren't found. Point PATH/CUDA_HOME at a complete CUDA toolkit.

• DeepSpeed ImportError: '_disable_dynamo_if_unsupported' is not defined

  Importing DeepSpeed fails with NameError/ImportError for _disable_dynamo_if_unsupported because DeepSpeed references a torch internal symbol absent from the installed torch version. Aligning the torch and DeepSpeed versions (e.g. torch>=2.3.1 with deepspeed>=0.17.4) resolves it.

• DeepSpeed ImportError: cannot import '_get_socket_with_port' from torch.distributed.elastic

  DeepSpeed import/launch fails with 'cannot import name _get_socket_with_port from torch.distributed.elastic.agent.server.api' because DeepSpeed uses a torch internal symbol that was renamed/removed in the installed torch. Align the DeepSpeed and torch versions (upgrade DeepSpeed or match torch).

• DeepSpeed Windows Build Fails (LNK1181 aio.lib / missing stdint)

  Building a DeepSpeed wheel on Windows 11 fails with 'LINK : fatal error LNK1181: cannot open input file aio.lib' and header compile errors. async_io/libaio is Linux-only and some headers don't compile under MSVC without stdint. Disable aio, add the missing include, or build under WSL2.

• DeepSpeed ZeRO-3 Stale Sync during Grad Clipping

  In ZeRO Stage 3, parameters and gradients are partitioned. When computing the global gradient norm for clipping, all ranks must synchronize their local norms. If there is a communication timeout, or if some ranks process empty partitions in a specific mixed-precision context, a rank might send uninitialized or stale gradient buffers containing NaNs. The global all-reduce aggregates these NaNs, causing the entire step to fail.

• Docker GPU Passthrough Error

  Docker containers fail to access GPUs when nvidia-docker or GPU passthrough is not properly configured.

• Docker Image Mismatch

  Docker image mismatches between dev and production cause code to behave differently due to library versions, CUDA, or system dependencies.

• Docker Permission Error

  Docker permission errors prevent containers from accessing required resources like GPUs, files, or network.

• Environment Variable Not Set

  Missing environment variables (CUDA_VISIBLE_DEVICES, HF_TOKEN, MASTER_ADDR) cause silent failures or wrong behavior.

• ExLlamaV2 kernels do not support act-order desc_act with group_size

  The ExLlama kernel cannot serve this GPTQ checkpoint: act-order (desc_act=True) combined with a grouped quantization is unsupported by that kernel path. The model is fine; the kernel choice is not. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent quantization failures.

• FlashAttention Build Hang / OOM due to missing Ninja

  Building FlashAttention requires compiling massive CUDA templates. If the `ninja` build system is not installed, `setuptools` falls back to standard sequential compilation, which takes hours. Furthermore, if `pip` uses build isolation (the default), it creates a temporary environment without the user's installed `ninja` or specific `torch` version, leading to either a missing dependency error, extremely slow builds, or memory exhaustion during compilation.

• GCC Version Mismatch

  GCC version mismatches cause C++ extension compilation failures and runtime library incompatibilities.

• Hopper mma.sp Silent Data Corruption (SDC)

  Using custom PTX kernels that leverage the mma.sp sparse tensor core instruction on Hopper GPUs causes silent calculation errors on the 535-series NVIDIA drivers.

• Hugging Face Tokenizers Fork Warning and DataLoader Hangs

  Distinguish the tokenizers fork safety warning from an actual DataLoader hang. Check parallelism timing and verify worker progress and token outputs.

• HuggingFace Hub Error

  HuggingFace Hub errors occur when downloading models or datasets fails due to auth, network, or rate limits.

• HuggingFace Tokenizers Rust Error

  HuggingFace tokenizers library uses Rust backend, which can fail with native errors not visible in Python traceback.

• Import / Environment Error

  Import and environment errors crash training at startup due to missing or mismatched dependencies.

• Index Error (Python)

  Python index errors crash training when accessing list/tuple indices that don't exist.

• Inductor fails to compile cumsum on a 0-dim constant tensor with tuple index out of range

  Compilation aborts on a graph containing a cumulative sum over a scalar constant. A post-graph optimisation recognises the pattern, asks the tensor for the length of the dimension being summed, and a scalar has no dimensions to ask about, so indexing its empty shape raises before any code is generated.

• Key Error (Python Dictionary)

  Python key errors crash training when accessing dictionary keys that don't exist.

• Libc Version Mismatch

  Libc (glibc, musl) version mismatches between training and deployment cause binary incompatibility and runtime errors.

• llama_model_load: error loading model: unknown tensor type unsupported GGUF version

  llama.cpp/GGUF load failure, file is not GGUF (old GGML), GGUF version newer than the runtime, truncated download, or quantization type unsupported by this build. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent quantization failures.

• Megatron @jit_fuser Fails: 'Unknown type constructor Sequence'

  torch.jit.script on a Megatron @jit_fuser-decorated function fails with 'Unknown type constructor Sequence' on newer torch. TorchScript can't resolve typing.Sequence as used inside the fused function. Upgrade Megatron (the fix moved the vocab-range lookup out of the JITed function) or replace the unsupported annotation.

• Misaligned TCPStore and NCCL Initialization Timeouts

  During init_process_group, PyTorch uses a TCPStore to exchange initial connection information (like ncclUniqueId). The default timeout for TCPStore is often shorter than the NCCL watchdog or the time it takes for a massive model checkpoint to load from network storage. Fast ranks finish loading and wait at the barrier; slow ranks take too long, and the fast ranks trigger a timeout error.

• Multi-Allocator Zero-Copy MMU Fault

  A secondary memory allocator (like CuPy's memory pool) forcibly freed a block of memory that was zero-copied from PyTorch. PyTorch attempts to access the now-unmapped virtual address.

• NaN Inf loss detected in flash_attn_varlen_func with autocast fp16

  FlashAttention produced non-finite values under FP16 autocast. Overflow can originate in the inputs, attention logits, scaling, or an unsupported kernel path, so the first non-finite tensor determines the owner. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent multimodal failures.

• NCCL Shared Memory Namespace Error

  Displays as 'Call to open failed: No such file or directory' for /dev/shm. Caused by container environments lacking NUMA support or adequate shared memory limits.

• No Matching Distribution Found for bitsandbytes on macOS

  Installing a QLoRA/4-bit training stack (Axolotl, TRL, Unsloth) on macOS fails with 'No matching distribution found for bitsandbytes==0.43.0'. bitsandbytes historically shipped only CUDA Linux/Windows wheels; 4/8-bit quantization needs CUDA, so the real training must run on a Linux GPU host.

• NVDEC hardware decoder stream allocation failed maximum concurrent decode sessions exceeded

  The GPU's hardware video decoder ran out of concurrent sessions. NVDEC session count is a fixed hardware/driver limit, and consumer boards are capped far below data-center parts regardless of available memory. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent cuda-graphs failures.

• nvidia-container-cli device node error /dev/nvidiactl no such file or directory

  The container runtime could not find the NVIDIA device nodes on the host. The kernel modules are not loaded, or the node was created after the container runtime cached its view, the GPU is invisible inside every container on this host. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent containers failures.

• nvidia-device-plugin Failed to initialize NVML libnvidia-ml.so.1 cannot be found

  The NVIDIA device plugin could not load libnvidia-ml.so, so it never registered nvidia.com/gpu with the kubelet. Every GPU pod on the node stays Pending with "Insufficient nvidia.com/gpu" while the node itself looks healthy. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent containers failures.

• OMP: Error #15: Initializing libomp.so but found libgomp.so already initialized

  Two OpenMP runtimes were loaded into one process (Intel/LLVM libomp alongside GNU libgomp). Behaviour is undefined, in practice thread pools collide, causing deadlocks, oversubscription, or crashes in multi-threaded inference. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent quant failures.

• Only Tensors of floating point dtype can require gradients int8 LoRA

  A LoRA adapter was attached to a quantized (int8/int4) base layer and the training path tried to make the quantized weight itself require gradients. Only the floating-point adapter weights are trainable; the quantized base must stay frozen. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent huggingface failures.

• OpenMPI / MPI Issue

  OpenMPI and general MPI issues cause distributed training to fail at initialization or hang during collectives.

• Pre-built Wheel Undefined Symbol Error

  The user installed a pre-built wheel of `flash-attn` compiled against a specific PyTorch and CUDA version (e.g., PyTorch 2.1.0 + cu118), but their runtime environment has a different PyTorch version (e.g., PyTorch 2.2.0 + cu121). C++ extensions in PyTorch are tightly coupled to the PyTorch ABI (Application Binary Interface). Mismatched PyTorch versions lead to unresolved symbols at load time.

• Python 3.13 cannot pickle code objects, so a distributed job reports the wrong failure

  An interpreter upgrade changes what pickle will serialise, and traceback objects stop qualifying. Any machinery that ships an exception between ranks then fails while packing it, so the error the cluster reports is the packing failure and the original fault is never printed.

• Python Import Error

  Python import errors prevent training from starting when modules cannot be loaded.

• Python Multiprocessing Fork Issue

  Python multiprocessing fork issues cause CUDA context conflicts and worker process deadlocks.

• Python Path Conflict

  Python path conflicts cause wrong module versions to be loaded, leading to subtle bugs and errors.

• Python Version Mismatch

  Python version mismatches between training and inference cause subtle bugs from library incompatibilities.

• PyTorch / CUDA / cuDNN Version Mismatch

  Version mismatches between PyTorch, CUDA, and cuDNN cause cryptic errors.

• PyTorch CUDA Error (Generic)

  Generic PyTorch CUDA errors indicate various GPU-related issues that prevent training.

• PyTorch CUDA Mismatch

  PyTorch CUDA version mismatch with installed CUDA toolkit/driver causes import errors or runtime failures.

• PyTorch DataLoader Hangs Indefinitely with OpenCV due to Fork

  OpenCV's internal multi-threading uses OpenMP/pthreads. When PyTorch DataLoader uses `num_workers > 0` with the `fork` start method, the child processes inherit a corrupted state of locks held by threads in the parent process, leading to a deadlock.

• PyTorch Hub Error

  PyTorch Hub errors prevent loading pretrained models or datasets from torch.hub.

• PyTorch Not Compiled With CUDA

  PyTorch installed without CUDA support prevents GPU training entirely.

• PyTorch SDPA FlashAttention Version Incompatibility

  PyTorch introduced `scaled_dot_product_attention` (SDPA) with a built-in FlashAttention backend in version 2.0. However, the specific features supported by the built-in backend depend on the PyTorch version (e.g., PyTorch 2.0 only supports FlashAttention v1, PyTorch 2.2+ integrates FlashAttention v2). If the input tensors require features only available in newer FlashAttention versions (like specific causal masks, sliding window attention, or head dimensions > 64/128), an older PyTorch version will fail to dispatch to the flash backend and silently fallback (or crash if forced).

• PyTorch vs System CUDA Compilation Mismatch

  FlashAttention compiles custom CUDA kernels during installation via `setup.py`. It queries PyTorch for its compiled CUDA version (`torch.version.cuda`) and compares it against the system's `nvcc` compiler version. If they do not match exactly, the build script intentionally raises an error to prevent generating incompatible binaries that would segfault at runtime.

• pyxis enroot import failed to extract squashfs image disk quota exceeded

  Enroot exhausted the user or filesystem quota while importing and extracting the SquashFS image. Free space elsewhere on the node does not help when the configured cache or data path has a separate quota. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent containers failures.

• rclpy RCLError Failed to publish sensor_msgs Image shared memory ring buffer overwrite

  A ROS 2 publisher overwrote shared-memory ring buffer entries a subscriber had not yet consumed. The consumer is slower than the producer, so frames are being lost, and with zero-copy transport the subscriber can read a partially overwritten message rather than simply missing one. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent cuda-graphs failures.

• Safetensors HeaderTooLarge: File Integrity Checks

  Check the safetensors length prefix, artifact identity and download integrity before reloading a file rejected with HeaderTooLarge.

• Segmentation fault in custom CUDA kernel .so during backprop

  A native custom CUDA extension crashed the process during backward execution. The fault is inside the extension or its ABI boundary, not an NCCL collective simply because distributed ranks exit afterward. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent cuda-graphs failures.

• SSH Key Authentication Issue

  SSH key authentication issues prevent passwordless login for distributed training and remote execution.

• SSL Certificate Error

  SSL certificate errors prevent downloading models, datasets, or connecting to APIs in ML training pipelines.

• Tensor Parallel RNG State Context Divergence Deadlock

  In Tensor Parallelism (TP), ranks within the same TP group must execute identical control flows for collective operations. If the `CudaRNGStatesTracker` is misconfigured or activation offloading fails to perfectly restore the RNG state, dropout masks or stochastic operations will differ across TP ranks. This causes divergent control flow (e.g., dropping different tokens in sequence parallel), leading to ranks submitting different sequences of NCCL calls and deadlocking.

• TensorRT-LLM conversion and build scripts fail after NumPy is upgraded inside a pinned container

  NGC containers ship a set of extensions compiled against a specific NumPy major version. Installing anything that pulls a newer NumPy replaces the runtime those extensions were built for, and the next conversion or build script fails inside a TensorRT-LLM module that has nothing to do with the package that was installed.

• TensorRT-LLM fails at model init because the FlashInfer FMHA kernel has no build for the GPU architecture

  TensorRT-LLM delegates fused attention to FlashInfer, whose FMHA runner is compiled for a specific set of architectures. On a device outside that set the runner raises an unsupported-architecture error from its own header, so the model finishes loading and then fails to serve.

• TensorRT-LLM FP8 support depends on GPU architecture, backend and model

  An FP8 architecture rejection requires checking the selected TensorRT-LLM release, backend, quantization recipe and GPU. NVIDIA's reviewed support matrix includes Ada SM89 and Hopper SM90 FP8, but does not list native FP8 for Ampere SM80 or SM86.

• TensorRT-LLM serving fails with an AttributeError from the FlashInfer attention backend

  The FlashInfer attention backend inside TensorRT-LLM reads fields from an attention metadata object that the release bundled beside it does not define. Serving fails with a plain Python AttributeError naming the missing field, so a version-skew problem between two components arrives looking like ordinary application code being wrong.

• Thermal-Induced ECC Throttling causing NCCL timeouts

  Poor airflow or a dried-out thermal pad on the HBM stack causes the memory to overheat. High temperatures exponentially increase the rate of transient single-bit flips. The hardware ECC engine corrects these on the fly, but the correction logic adds severe latency to memory transactions. This memory starvation delays collective communication, ultimately causing a NCCL timeout.

• Timezone / UTC Mismatch

  Timezone mismatches cause confusion in distributed training logs, scheduled jobs, and time-based events.

• Topology Graph Override Mismatch

  An environment variable forces NCCL to use a static topology XML file that does not match the actual hardware layout of the dynamically allocated compute node.

• torch._dynamo.exc.BackendCompilerFailed backend inductor LoweringException

  TorchDynamo captured the graph, but TorchInductor failed while lowering an operation to generated code. The useful cause is the nested LoweringException and its target operator, not the outer BackendCompilerFailed wrapper. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent torch-compile failures.

• torch._dynamo.exc.RestartAnalysis Exceeded maximum graph break limit

  PyTorch: TorchDynamo encountered a graph break due to unsupported Python construct. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent torch-compile failures.

• torchao quantization CalibrationError observed activation tensor contains NaN during INT8 observer

  The INT8 observer saw NaN or Inf activations while calibrating. Quantization scales are derived from observed ranges, so a single non-finite activation poisons the scale for that tensor and every value quantized with it. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent quantization failures.

• Torchvision Error

  Torchvision errors prevent loading pretrained vision models or using vision transforms.

• Transformers Version Mismatch

  Transformers version mismatches between training and inference cause subtle bugs and performance regressions.

• Triton Compiler FlashAttention Fallback Failure

  The FlashAttention kernel hits a hardware capability block (e.g., unsupported SM version) and initiates a fallback path. The fallback Python code attempts to parse metadata that was never instantiated, triggering a standard Python AttributeError.

• Triton JIT compilation failed ptxas fatal Value sm_90a is not defined for option gpu-name

  NVIDIA PTX assembler failed on Triton output. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent torch-compile failures.

• triton OutOfResources Out of resources during kernel launch shared memory request exceeds device maximum

  Triton kernel requested too much shared memory. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent quant failures.

• trtllm-build fails because an optimization profile has a minimum dimension above its optimum

  TensorRT requires min to be less than or equal to opt, and opt less than or equal to max, for every dynamic dimension in an optimization profile. The engine build aborts when TensorRT-LLM derives a profile in which some dimension violates that ordering, usually because build flags imply a minimum larger than the optimum that was set.

• Type Error (Python)

  Python type errors crash training when incompatible types are used in operations.

• Ulimit Too Low

  Low ulimit values (open files, max processes) cause data loading failures and parallel training issues.

• UnicodeDecodeError Importing Unsloth on Windows (missing encoding=utf-8)

  Importing Unsloth on Windows fails with a UnicodeDecodeError because an internal open() omitted encoding='utf-8' and Windows defaults to cp1252. Upgrade Unsloth (now passes the encoding), or set PYTHONUTF8=1 / enable the system UTF-8 locale.

• Unsloth Streaming Cancel Scope Error on Python 3.13

  Check the reported Python 3.13 and anyio 4.14.0 combination against Unsloth issue 6483, then verify streaming completion after a compatible change.

• Value Error (Python)

  Python value errors crash training when functions receive values of correct type but inappropriate value.

• vLLM crashes initialising an NVFP4 model because the emulation lookup table stays on the CPU

  When no hardware NVFP4 backend is available, vLLM falls back to an emulation path that unpacks four-bit values through a small constant lookup table. The table is indexed by a tensor that lives on the GPU while the table itself was built on the CPU, so engine initialisation fails before a single token is generated.

• Watchdog Timeout False Positive during Long Checkpoint Saves

  When Rank 0 writes a massive checkpoint to slow storage (like an overloaded NFS or object store), it takes longer than the default NCCL timeout (30 minutes). The other ranks wait at a `dist.barrier()` or the next collective operation. The watchdog on the waiting ranks assumes Rank 0 crashed and kills the job.

• Watchdog Timeout from Inconsistent Tensor Shapes

  Ranks attempt to perform an all_gather or all_reduce on tensors that have different shapes across different ranks (e.g., dynamic sequence lengths in NLP). NCCL expects the byte count of the transferred buffers to match perfectly. When they don't, the internal state machine of NCCL gets deadlocked waiting for data that will never arrive.

• Xid 13: Graphics Engine Exception due to Kernel Out-of-Bounds Access

  A custom PyTorch C++ / CUDA kernel (or a bug in a framework like TensorRT/DeepSpeed) calculates an incorrect thread index or memory offset, resulting in a read or write operation past the allocated bounds of a tensor. The NVIDIA hardware traps this memory violation (Warp Illegal Address) and halts the GPU context, logging an Xid 13 error.

• Xid 43 (GPU Stopped Processing)

  Emitted when a user-space CUDA application violates memory bounds or instructions, forcing the GPU to reset the channel to protect itself.