# Denpex

> Denpex diagnoses distributed GPU training and inference failures. Given the logs a failed run produced, it correlates available evidence across ranks, looks for the first supported failure, and classifies the failure against 16,400+ known signatures. Diagnosis depends on the supplied evidence; incomplete logs may not establish a root cause. It covers NCCL collective timeouts, CUDA out-of-memory, NVIDIA Xid hardware faults, silent data corruption, stragglers and gray failures across PyTorch, FSDP, DeepSpeed, Megatron, Ray, Slurm, Kubernetes and vLLM.

Denpex focuses on failure diagnosis. It complements DCGM device diagnostics, Prometheus metrics, Slurm scheduling, Ray orchestration, and Weights & Biases experiment tracking. Consult the product documentation for the scope of each integration and deployment mode.

## Start here

- [/vscode](https://denpex.com/vscode): VS Code extension, local diagnosis workflow, and cloud review options.
- [/failures](https://denpex.com/failures): The failure encyclopedia index. Root cause and fix per failure mode.
- [/nccl-errors](https://denpex.com/nccl-errors): Every common NCCL error and what it actually indicates, with the environment variables that tell them apart.
- [/cuda-errors](https://denpex.com/cuda-errors): CUDA errors and why the traceback usually points at the wrong line. Asynchronous reporting explained.
- [/slurm-gpu-errors](https://denpex.com/slurm-gpu-errors): Slurm GPU failures by phase: allocation, cgroup creation, and inside the workload.
- [/deepspeed-errors](https://denpex.com/deepspeed-errors): DeepSpeed failures separated by ZeRO stage and lifecycle phase.
- [/vllm-errors](https://denpex.com/vllm-errors): vLLM and TensorRT-LLM inference serving failures.
- [/xid](https://denpex.com/xid): NVIDIA Xid error reference. Which codes mean a bad GPU and which do not.
- [/solutions/best-tools-gpu-cluster-debugging](https://denpex.com/solutions/best-tools-gpu-cluster-debugging): The GPU debugging stack tool by tool: DCGM, Prometheus, Slurm, Ray, W&B, PyTorch flight recorder, Denpex. What each answers and where each stops.
- [/compare/denpex-vs-dcgm](https://denpex.com/compare/denpex-vs-dcgm): How Denpex complements NVIDIA DCGM. Device health versus run diagnosis.
- [/compare/denpex-vs-prometheus](https://denpex.com/compare/denpex-vs-prometheus): How Denpex complements Prometheus and Grafana. Metrics versus root cause.
- [/compare/denpex-vs-slurm](https://denpex.com/compare/denpex-vs-slurm): How Denpex complements Slurm. Exit codes versus root cause.
- [/compare/denpex-vs-ray](https://denpex.com/compare/denpex-vs-ray): How Denpex complements Ray. Actor supervision versus failure diagnosis.
- [/compare/denpex-vs-weights-and-biases](https://denpex.com/compare/denpex-vs-weights-and-biases): How Denpex complements Weights & Biases. Experiment tracking versus failure diagnosis.
- [/guides/identify-first-failed-rank](https://denpex.com/guides/identify-first-failed-rank): How to identify the first failed rank in PyTorch and NCCL logs by separating initiating signals from timeout cascades.
- [/guides/determine-gpu-needs-rma](https://denpex.com/guides/determine-gpu-needs-rma): Hardware qualification and RMA decisions: how to distinguish a genuinely defective GPU from a slot, riser, or driver fault.
- [/guides/diagnose-slurm-gpu-oom](https://denpex.com/guides/diagnose-slurm-gpu-oom): Diagnosing Slurm GPU out-of-memory errors by execution phase: cgroup step limits, host RAM, and device memory.
- [/guides/diagnose-quiet-stragglers](https://denpex.com/guides/diagnose-quiet-stragglers): Detecting quiet GPU stragglers and gray failures across multi-node clusters before they stall synchronous barriers.
- [/guides/golden-node-configuration-diff](https://denpex.com/guides/golden-node-configuration-diff): How to diff environment, driver, and kernel variables against a known golden node to catch cluster drift.
- [/tools](https://denpex.com/tools): Free browser tools: Xid decoder, nvidia-smi parser, NCCL log decoder, downtime calculator, topology linter.
- [/docs](https://denpex.com/docs): Product documentation, agent install, API reference.
- [/distributed-training-hang-triage](https://denpex.com/distributed-training-hang-triage): Data Center scoped pilot: PyTorch Flight Recorder hang triage, hardware correlation and reviewed isolation plans. Real CPU fault tests; physical GPU validation remains open.
- [/pricing](https://denpex.com/pricing): Plans and what is included at each tier.
- [/benchmark](https://denpex.com/benchmark): Reproducible diagnosis accuracy benchmark and methodology.

## Failure encyclopedia

452 evidence-grade entries from 700 public encyclopedia records are listed below. Each page carries an error signature, a root cause, a fix, and citations to upstream vendor documentation or issue trackers.

Sourced entries remain public but are deliberately omitted from this stricter curated index. Withheld failure pages return HTTP 404 and are not public reference material. Internal matcher coverage is not the same as published, evidence-backed documentation coverage.

### Checkpointing

- [Weights-only load failed with an UnpicklingError when you resume an older checkpoint](https://denpex.com/failures/weights-only-unpickling-error-on-checkpoint-resume)

### Communication

- [ACS Disabled. GPU Direct RDMA Failure (Vendor Err 81)](https://denpex.com/failures/acs-disabled-gdr-failure)
- [AWS EFA Driver Issue](https://denpex.com/failures/efa-driver-issue)
- [broadcast_coalesced Fails with Cuda failure 1 'invalid argument'](https://denpex.com/failures/nccl-broadcast-coalesced-invalid-arg)
- [DDP Port Conflict](https://denpex.com/failures/ddp-port-conflict)
- [DDP Setup Error](https://denpex.com/failures/ddp-setup-error)
- [FSDP ncclSystemError on InfiniBand (works on TCP)](https://denpex.com/failures/fsdp-nccl-system-error-ib)
- [gdrcopy gdr_pin_buffer failed GPU virtual address is not mapped to BAR1 aperture](https://denpex.com/failures/gdrcopy-pin-buffer-failed-bar1-aperture)
- [Gloo Backend Issue](https://denpex.com/failures/gloo-backend-issue)
- [Gloo Connection refused to host 29500 during initial rendezvous](https://denpex.com/failures/gloo-connection-refused-port-29500-rendezvous)
- [Horovod Setup Error](https://denpex.com/failures/horovod-setup-error)
- [ibv_rc_pingpong: Failed to modify QP to RTR errno 110 Connection timed out](https://denpex.com/failures/infiniband-qp-rtr-connection-timed-out-errno-110)
- [Mixture-of-experts dispatch and combine collectives time out on large NVLink domains](https://denpex.com/failures/expert-parallel-dispatch-combine-collective-timeout)
- [mlx5_core mlx5_pfc_stall_detect PFC storm detected on priority fabric deadlock](https://denpex.com/failures/rocev2-pfc-storm-priority-flow-control-deadlock)
- [NCCL 'Could not find NET with id 0' (NIC Fusion)](https://denpex.com/failures/nccl-net-id-0-not-found)
- [NCCL Asynchronous Error Handling](https://denpex.com/failures/nccl-async-error)
- [NCCL Communication Timeout](https://denpex.com/failures/nccl-comm-timeout)
- [NCCL CUDA Failure](https://denpex.com/failures/nccl-cuda-failure)
- [NCCL Error 2: Internal Error](https://denpex.com/failures/nccl-error-2)
- [NCCL GPUDirect over RoCE](https://denpex.com/failures/nccl-gpu-direct-roce)
- [NCCL IB HCA Mismatch](https://denpex.com/failures/nccl-ib-hca-mismatch)
- [NCCL Initialization Error](https://denpex.com/failures/nccl-init-error)
- [NCCL Initialization Timeout](https://denpex.com/failures/nccl-init-timeout)
- [NCCL IPv6 Issue](https://denpex.com/failures/nccl-ipv6-issue)
- [NCCL Mismatched Collective. Different Ranks Executing Different Operations](https://denpex.com/failures/nccl-mismatched-collective)
- [NCCL MNNVL Init Segfault at Large World Size (Stack Overrun)](https://denpex.com/failures/nccl-mnnvl-stack-overrun)
- [NCCL multi-NIC packet reordering detected RoCE packets arrived out of sequence pipeline stalled](https://denpex.com/failures/nccl-cross-nic-roce-packet-reordering-stall)
- [NCCL NVLS CUDA failure 1: invalid argument](https://denpex.com/failures/nccl-nvls-invalid-argument)
- [NCCL P2P Bandwidth Issue](https://denpex.com/failures/nccl-p2p-bandwidth)
- [NCCL P2P Communication Issue](https://denpex.com/failures/nccl-p2p-issue)
- [NCCL Proxy Connect Failed (IPv6 Interference)](https://denpex.com/failures/nccl-proxy-connect-ipv6)
- [NCCL Rank Fail](https://denpex.com/failures/nccl-rank-fail)
- [NCCL RAS Race Segfault During Initialization](https://denpex.com/failures/nccl-ras-init-segfault)
- [NCCL ring buffer allocation failed NCCL_BUFFSIZE exceeds available contiguous pinned host memory](https://denpex.com/failures/nccl-buffsize-ring-buffer-pinned-memory-exhausted)
- [NCCL Topology XML Missing NIC Bus ID](https://denpex.com/failures/nccl-topology-xml-missing-busid)
- [NCCL Tree algorithm deadlocked non-power-of-two rank count NCCL_ALGO Ring](https://denpex.com/failures/nccl-algo-tree-deadlock-non-power-of-two-ranks)
- [NCCL Version Conflict](https://denpex.com/failures/nccl-version-conflict)
- [NCCL WARN Bootstrap: Condition [state == ncclSuccess] failed: Connection reset by peer](https://denpex.com/failures/nccl-warn-bootstrap-connection-reset-by-peer)
- [NCCL WARN Connection failed : Network is unreachable NCCL_SOCKET_IFNAME](https://denpex.com/failures/nccl-socket-ifname-network-unreachable)
- [NCCL WARN Direct RDMA write between GPU NUMA and HCA NUMA failed UPI interconnect](https://denpex.com/failures/nccl-gdr-cross-numa-upi-throttle-crash)
- [NCCL WARN Failed to initialize transport NET IB Resource temporarily unavailable](https://denpex.com/failures/nccl-failed-to-init-transport-net-ib)
- [NCCL WARN Ring via NET/Socket Call to connect returned Connection refused](https://denpex.com/failures/nccl-ring-connection-refused-socket)
- [ncclCommSplit Segfault with Non-Blocking Init](https://denpex.com/failures/nccl-commsplit-segfault-nonblocking)
- [ncclUnhandledCudaError: Cuda failure 999 'unknown error'](https://denpex.com/failures/nccl-cuda-failure-999)
- [nvidia-peermem kernel rejected peer-direct DMA mapping BAR1 aperture exhausted](https://denpex.com/failures/nvidia-peermem-bar1-aperture-exhausted)
- [NVLink SHARP (NVLS) multicast initialization failure](https://denpex.com/failures/nvls-error)
- [NVLS Multicast Slot Exhaustion (NVSwitch)](https://denpex.com/failures/nvls-multicast-slot-exhaustion)
- [OFI Memlock Exhaustion. RDMA Memory Registration Failure](https://denpex.com/failures/ofi-memlock-exhaustion)
- [OpenSM SM sweep failed: Link state active timeout on Port GUID](https://denpex.com/failures/opensm-subnet-manager-sweep-failure)
- [RoCE MTU Mismatch. Completion Error 12 / Vendor Err 129](https://denpex.com/failures/roce-mtu-mismatch)
- [RoCE QP Timeout During Distributed Checkpoint Save](https://denpex.com/failures/nccl-roce-qp-timeout-checkpoint)
- [TCP Port Exhaustion](https://denpex.com/failures/tcp-port-exhaustion)
- [vLLM workers time out reading the shared-memory broadcast ring and the engine dies](https://denpex.com/failures/vllm-shm-broadcast-acquire-read-timeout)

### Compilation

- [Inductor fails to compile cumsum on a 0-dim constant tensor with tuple index out of range](https://denpex.com/failures/inductor-cumsum-zero-dim-tuple-index-out-of-range)

### Data

- [HDF5 Concurrent Read Deadlock with Multiprocessing](https://denpex.com/failures/hdf5-fork-deadlock-1)
- [NCCL Watchdog Timeout due to Uneven Dataset Sharding](https://denpex.com/failures/nccl-watchdog-uneven-sharding)

### Data Integrity

- [botocore ClientError 503 PutObject SlowDown S3 prefix request rate exceeded](https://denpex.com/failures/s3-putobject-503-slowdown-checkpoint-throttle)
- [Checkpoint Corruption](https://denpex.com/failures/checkpoint-corruption)
- [Checkpoint Saved with Newer Version](https://denpex.com/failures/checkpoint-version-newer)
- [Checkpoint Saved with Older Version](https://denpex.com/failures/checkpoint-version-older)
- [Checkpoint Torn Write](https://denpex.com/failures/checkpoint-torn-write)
- [Checkpoint Version Incompatible](https://denpex.com/failures/checkpoint-version-incompatible)
- [DeepSpeed DecoupledCheckpointEngine Infinite Hang](https://denpex.com/failures/deepspeed-checkpoint-engine-hang)
- [DeepSpeed FastFileWriter File Descriptor Leak, Phantom ENOSPC](https://denpex.com/failures/deepspeed-fd-leak-enospc)
- [DeepSpeed SIGBUS Exit Code -7 After Checkpoint Load](https://denpex.com/failures/deepspeed-sigbus-exit-code-7)
- [DistCheckpointError Async checkpoint saving failed in background thread write timeout](https://denpex.com/failures/torch-distributed-checkpoint-async-save-timeout)
- [Optimizer State Dict Mismatch](https://denpex.com/failures/optimizer-state-dict-mismatch)
- [PyTorch Load Failed / Unpickling Error](https://denpex.com/failures/torch-load-failed)
- [PyTorch Save Failed / Write Error](https://denpex.com/failures/torch-save-failed)
- [Single-Rank Reduce-Scatter Silent Data Corruption](https://denpex.com/failures/nccl-onerank-reduce-sdc)
- [sudden gradient norm spike without learning rate change transient DRAM parity error](https://denpex.com/failures/gradient-norm-spike-transient-dram-parity-flip)
- [tensor core GEMM calculation deviation silent data corruption bit flip](https://denpex.com/failures/silent-data-corruption-tensor-core-gemm-bit-flip)
- [TensorRT-LLM refuses a quantized checkpoint because its weight mapper does not recognise the export layout](https://denpex.com/failures/trtllm-weight-mapper-rejects-quantized-checkpoint)
- [UnpicklingError pickle data was truncated EOFError Ran out of input checkpoint](https://denpex.com/failures/torch-load-unpicklingerror-truncated-checkpoint)
- [webdataset TarError Unexpected end of archive while reading shard](https://denpex.com/failures/webdataset-tarerror-unexpected-end-of-archive)

### Data Pipeline

- [Arrow IPC Format Error](https://denpex.com/failures/arrow-ipc-error)
- [Audio Channel Mismatch](https://denpex.com/failures/audio-channel-mismatch)
- [cuFileRead failed with error 5002 CU_FILE_DRIVER_NOT_INITIALIZED](https://denpex.com/failures/cufile-gpudirect-driver-not-initialized-5002)
- [Data Augmentation Error](https://denpex.com/failures/augmentation-error)
- [DataLoader worker is killed by signal: Bus error shared memory exhausted](https://denpex.com/failures/dataloader-worker-killed-bus-error-shm)
- [HDF5 Data Corruption](https://denpex.com/failures/hdf5-data-corruption)
- [HuggingFace datasets .map() 'subprocess abruptly died' on Windows](https://denpex.com/failures/hf-datasets-map-subprocess-died-windows)
- [HuggingFace Tokenizer Error](https://denpex.com/failures/huggingface-tokenizer-error)
- [MMap File Handle Exhaustion](https://denpex.com/failures/mmap-file-handle-exhaustion)
- [nfs: server not responding still trying task blocked for more than 120 seconds](https://denpex.com/failures/nfs-server-not-responding-d-state-hung-process)
- [Parquet Schema Mismatch](https://denpex.com/failures/parquet-schema-mismatch)
- [Pinned Memory Transfer Issue](https://denpex.com/failures/pin-memory-issue)
- [Python Multiprocessing Fork Issue](https://denpex.com/failures/python-multiprocessing-fork)
- [PyTorch Multiprocessing Error](https://denpex.com/failures/torch-multiprocessing-error)
- [Streaming Data Error](https://denpex.com/failures/streaming-data-error)
- [Tokenization Error](https://denpex.com/failures/tokenization-error)
- [Tokenizer Padding and Truncation Error](https://denpex.com/failures/tokenizer-pad-truncate)
- [TorchData Pipeline Error](https://denpex.com/failures/torchdata-pipeline-error)
- [WebDataset Error](https://denpex.com/failures/webdataset-error)
- [WebDataset TAR Corruption](https://denpex.com/failures/webdataset-tar-corruption)

### Distributed Communication

- [NCCL Collective Operation Timeout](https://denpex.com/failures/nccl-timeout)

### Distributed Training

- [Accelerate-DeepSpeed Watchdog Timeout Mismatch](https://denpex.com/failures/dpx-fail-0444-accelerate-deepspeed-watchdog-timeout-mismatch)
- [Asymmetric PT2 Compilation Collective Desync](https://denpex.com/failures/dpx-fail-0443-asymmetric-pt2-compilation-collective-desync)
- [bitsandbytes 'invalid configuration argument' (ops.cu) with DeepSpeed Offload](https://denpex.com/failures/bitsandbytes-invalid-config-arg-deepspeed-offload)
- [Cowardly refusing to serialize non-leaf tensor](https://denpex.com/failures/torch-distributed-checkpoint-non-leaf-tensor)
- [DeepSpeed 0.14.x Regression: 'Expected all tensors to be on the same device' (ZeRO-3 / Adam Offload)](https://denpex.com/failures/deepspeed-0.14-tensor-device-mismatch-regression)
- [DeepSpeed Initialization Failed](https://denpex.com/failures/deepspeed-init-failed)
- [DeepSpeed MoE + ZeRO-3 Hang. Missing Leaf Module Marking](https://denpex.com/failures/deepspeed-moe-leaf-module)
- [DeepSpeed Out of Memory](https://denpex.com/failures/deepspeed-oom)
- [DeepSpeed PPO Actor forward pass deadlocked waiting for Critic ZeRO-3 parameter gather](https://denpex.com/failures/deepspeed-chat-actor-critic-zero3-deadlock)
- [DeepSpeed ZeRO Stuck / Hang](https://denpex.com/failures/deepspeed-zero-stuck)
- [DeepSpeed ZeRO-2 tensor bucket buffer exhausted during reduce_scatter](https://denpex.com/failures/deepspeed-zero2-reduce-scatter-bucket-exhausted)
- [DeepSpeed ZeRO-3 Error with Parameters of Multiple Dtypes (allgather)](https://denpex.com/failures/deepspeed-zero3-allgather-multiple-dtypes)
- [DeepSpeed ZeRO-Offload NVMe aio_write failed: No space left on device](https://denpex.com/failures/deepspeed-nvme-swap-space-exhausted)
- [DistributedDataParallel raises an internal reducer assertion when a backward pass arrives unexpectedly](https://denpex.com/failures/ddp-reducer-expect-autograd-hooks-assert)
- [Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu](https://denpex.com/failures/pytorch-expected-all-tensors-same-device)
- [Finding the Culprit Rank in an NCCL Hang](https://denpex.com/failures/nccl-hang-culprit-rank)
- [FSDP Failed to allocate unpartitioned full weights during forward pre-hook](https://denpex.com/failures/fsdp-unshard-forward-oom)
- [FSDP fails when composed with CPU parameter offloading](https://denpex.com/failures/fsdp-cpu-offload-composition-failure)
- [FSDP Missing Keys / Unexpected Keys](https://denpex.com/failures/fsdp-missing-keys)
- [FSDP preparation fails because activation checkpointing calls a wrapping policy that was never set](https://denpex.com/failures/fsdp-auto-wrap-policy-none-with-activation-checkpointing)
- [FSDP reduce_scatter backward hook timed out waiting for gradient sync](https://denpex.com/failures/fsdp-reduce-scatter-backward-timeout)
- [FSDP+QLoRA ValueError: Must flatten tensors with uniform dtype (float32 vs bfloat16)](https://denpex.com/failures/fsdp-qlora-uniform-dtype-flatten)
- [FSDP2 raises a KeyError on a tied weight while sharding the model](https://denpex.com/failures/fsdp2-tied-weights-keyerror-during-prepare)
- [jax.distributed initialize timed out waiting for rank 0 on coordinator](https://denpex.com/failures/jax-distributed-rendezvous-timeout)
- [jax.errors.XlaRuntimeError: RESOURCE_EXHAUSTED: Out of memory while trying to allocate](https://denpex.com/failures/jax-xla-resource-exhausted-oom)
- [Megatron Sequence Parallel requires Tensor Parallel size > 1](https://denpex.com/failures/megatron-sequence-parallel-assertion-error)
- [Megatron-LM Initialization Failed](https://denpex.com/failures/megatron-init-failed)
- [MXFP4 expert weights lose their scale attribute when a mixture-of-experts model loads under FSDP2](https://denpex.com/failures/mxfp4-expert-scales-missing-under-fsdp2)
- [NCCL Broadcast Hang](https://denpex.com/failures/nccl-broadcast-hang)
- [NCCL Watchdog Configuration](https://denpex.com/failures/nccl-watchdog-disable)
- [Overflow detected during fp16 gradient unscaling. Skipping step, loss scale halved](https://denpex.com/failures/deepspeed-fp16-overflow-inf-nan-loss-scale-halved)
- [P2PTimeoutError: Timeout receiving forward activations from rank](https://denpex.com/failures/megatron-p2p-communication-timeout)
- [PlacementGroupCreationError placement group creation timed out insufficient GPU resources](https://denpex.com/failures/openrlhf-ray-vllm-placement-group-timeout)
- [PPOTrainer CUDA out of memory cannot fit reference model policy model and value head](https://denpex.com/failures/trl-ppotrainer-actor-critic-out-of-memory)
- [PyTorch Distributed Error](https://denpex.com/failures/torch-distributed-error)
- [Ray client connection aborts on ARM64 after a grpcio upgrade while x86 is unaffected](https://denpex.com/failures/ray-client-server-fails-after-grpcio-upgrade-on-arm64)
- [Reference model logits shape does not match policy logits shape vocab mismatch](https://denpex.com/failures/dpo-trainer-logits-shape-vocab-mismatch)
- [Rotary embedding dimension must be divisible by head dimension](https://denpex.com/failures/megatron-rotary-embedding-dimension-mismatch)
- [Silent Cascading Straggler](https://denpex.com/failures/silent-cascading-straggler)
- [torch.distributed.elastic.multiprocessing.errors.ChildFailedError exitcode -9](https://denpex.com/failures/torchelastic-childfailederror-exitcode-9-oom)
- [torch.OutOfMemoryError: CUDA out of memory. Tried to allocate](https://denpex.com/failures/pytorch-cuda-out-of-memory-tried-to-allocate)
- [Triaging torchrun ChildFailedError: the Wrapper Is Never the Root Cause](https://denpex.com/failures/torchrun-childfailederror-triage)
- [Watchdog caught collective operation timeout WorkNCCL AllGather](https://denpex.com/failures/nccl-watchdog-timeout-worknccl-allgather)
- [ZeROStage3ParamStatusException: Parameter in Partitioned state, expected Available](https://denpex.com/failures/deepspeed-zero3-param-status-exception-partitioned)

### Environment

- [accelerate CannotDynamicallyBSZError Batch size must stay constant across gradient accumulation](https://denpex.com/failures/accelerate-cannot-dynamically-bsz-error)
- [Apptainer/Singularity --nv GPU Binding Failures](https://denpex.com/failures/apptainer-singularity-nv-binding-failures)
- [bitsandbytes matmul_4bit CUDA error: no kernel image is available for execution on the device](https://denpex.com/failures/bitsandbytes-matmul-4bit-no-kernel-image-available)
- [bitsandbytes was compiled without GPU support libbitsandbytes_cuda.so could not be loaded](https://denpex.com/failures/bitsandbytes-cuda-setup-failed-libbitsandbytes-missing)
- [Conda Environment Conflict](https://denpex.com/failures/conda-environment-conflict)
- [Connection Timeout](https://denpex.com/failures/connection-timeout)
- [Container CUDA Runtime Mismatch with Host Driver](https://denpex.com/failures/container-cuda-mismatch)
- [could not select device driver with capabilities gpu nvidia-container-cli initialization error](https://denpex.com/failures/docker-nvidia-container-cli-initialization-error)
- [CUDA Driver / Runtime Mismatch](https://denpex.com/failures/cuda-driver-mismatch)
- [CUDA error: invalid configuration argument awq_gemm_kernel grid block dimensions exceed SM limit](https://denpex.com/failures/autoawq-invalid-configuration-argument-gemm)
- [CUDA error: operation not permitted when stream is capturing](https://denpex.com/failures/cuda-graph-capture-operation-not-permitted-sync)
- [CUDA graph replay detected invalidated memory buffer](https://denpex.com/failures/cuda-graph-replay-invalidated-memory-buffer)
- [CUDA out of memory during AutoencoderKL tiling decode intermediate latents](https://denpex.com/failures/diffusers-vae-tiling-decode-out-of-memory)
- [CUDA out of memory during Inductor CUDA graph execution allocator private memory pool fragmentation](https://denpex.com/failures/inductor-cuda-graph-memory-pool-fragmentation)
- [cudaErrorLaunchFailure an illegal instruction was encountered sm90a](https://denpex.com/failures/cuda-launch-failure-illegal-instruction-sm90a)
- [cuDNN error: CUDNN_STATUS_NOT_SUPPORTED cudnnConvolutionForward](https://denpex.com/failures/cudnn-status-not-supported-convolution-forward)
- [DataLoader Deadlock from Silent Worker OOM](https://denpex.com/failures/pytorch-oom-worker-hang-1)
- [DeepSpeed async_io/aio Won't Compile on Windows](https://denpex.com/failures/deepspeed-async-io-windows-unsupported)
- [DeepSpeed cpu_adam Build Fails: 'cusolverDn.h: No such file or directory'](https://denpex.com/failures/deepspeed-cusolverdn-header-not-found-build)
- [DeepSpeed ImportError: '_disable_dynamo_if_unsupported' is not defined](https://denpex.com/failures/deepspeed-disable-dynamo-import-error-torch-version)
- [DeepSpeed ImportError: cannot import '_get_socket_with_port' from torch.distributed.elastic](https://denpex.com/failures/deepspeed-get-socket-with-port-import-error)
- [Docker GPU Passthrough Error](https://denpex.com/failures/docker-gpu-error)
- [ExLlamaV2 kernels do not support act-order desc_act with group_size](https://denpex.com/failures/autogptq-exllama-v2-act-order-group-size-incompatible)
- [FlashAttention Build Hang / OOM due to missing Ninja](https://denpex.com/failures/flashattention-build-isolation-1)
- [GCC Version Mismatch](https://denpex.com/failures/gcc-version-mismatch)
- [HuggingFace Tokenizers Rust Error](https://denpex.com/failures/huggingface-tokenizers-error)
- [Import / Environment Error](https://denpex.com/failures/import-error)
- [Key Error (Python Dictionary)](https://denpex.com/failures/key-error)
- [llama_model_load: error loading model: unknown tensor type unsupported GGUF version](https://denpex.com/failures/gguf-llama-model-load-unknown-tensor-type)
- [Megatron @jit_fuser Fails: 'Unknown type constructor Sequence'](https://denpex.com/failures/megatron-jit-fuser-unknown-type-sequence)
- [Misaligned TCPStore and NCCL Initialization Timeouts](https://denpex.com/failures/nccl-watchdog-tcpstore-timeout)
- [NaN Inf loss detected in flash_attn_varlen_func with autocast fp16](https://denpex.com/failures/autocast-fp16-nan-inf-loss-flash-attention)
- [NCCL Shared Memory Namespace Error](https://denpex.com/failures/dpx-fail-0458-nccl-shared-memory)
- [NVDEC hardware decoder stream allocation failed maximum concurrent decode sessions exceeded](https://denpex.com/failures/nvdec-hardware-decoder-sessions-exceeded)
- [nvidia-container-cli device node error /dev/nvidiactl no such file or directory](https://denpex.com/failures/nvidia-container-cli-device-node-not-found)
- [nvidia-device-plugin Failed to initialize NVML libnvidia-ml.so.1 cannot be found](https://denpex.com/failures/k8s-nvidia-device-plugin-nvml-libloading-error)
- [OMP: Error #15: Initializing libomp.so but found libgomp.so already initialized](https://denpex.com/failures/openmp-libomp-libgomp-deadlock-clash)
- [Only Tensors of floating point dtype can require gradients int8 LoRA](https://denpex.com/failures/peft-lora-int8-gradients-runtime-error)
- [Pre-built Wheel Undefined Symbol Error](https://denpex.com/failures/flashattention-wheel-incompatibility-1)
- [Python 3.13 cannot pickle code objects, so a distributed job reports the wrong failure](https://denpex.com/failures/python-313-cannot-pickle-code-objects-in-distributed-error-path)
- [Python Import Error](https://denpex.com/failures/python-import-error)
- [Python Path Conflict](https://denpex.com/failures/python-path-conflict)
- [Python Version Mismatch](https://denpex.com/failures/python-version-mismatch)
- [PyTorch / CUDA / cuDNN Version Mismatch](https://denpex.com/failures/version-mismatch)
- [PyTorch CUDA Error (Generic)](https://denpex.com/failures/torch-cuda-error)
- [PyTorch CUDA Mismatch](https://denpex.com/failures/pytorch-cuda-mismatch)
- [PyTorch Hub Error](https://denpex.com/failures/torch-hub-error)
- [PyTorch Not Compiled With CUDA](https://denpex.com/failures/torch-not-compiled-with-cuda)
- [PyTorch vs System CUDA Compilation Mismatch](https://denpex.com/failures/flashattention-cuda-mismatch-1)
- [pyxis enroot import failed to extract squashfs image disk quota exceeded](https://denpex.com/failures/enroot-pyxis-squashfs-import-disk-quota-exceeded)
- [rclpy RCLError Failed to publish sensor_msgs Image shared memory ring buffer overwrite](https://denpex.com/failures/ros2-pytorch-shared-memory-ring-buffer-corruption)
- [SafetensorError: Error while deserializing header: HeaderTooLarge](https://denpex.com/failures/safetensors-error-deserializing-header-too-large)
- [Segmentation fault in custom CUDA kernel .so during backprop](https://denpex.com/failures/dataloader-custom-cuda-voxelize-segfault)
- [TensorRT-LLM conversion and build scripts fail after NumPy is upgraded inside a pinned container](https://denpex.com/failures/trtllm-numpy-abi-break-against-pinned-container-stack)
- [TensorRT-LLM fails at model init because the FlashInfer FMHA kernel has no build for the GPU architecture](https://denpex.com/failures/trtllm-flashinfer-fmha-unsupported-architecture)
- [TensorRT-LLM FP8 support depends on GPU architecture, backend and model](https://denpex.com/failures/trtllm-fp8-unsupported-compute-capability)
- [TensorRT-LLM serving fails with an AttributeError from the FlashInfer attention backend](https://denpex.com/failures/trtllm-flashinfer-attention-metadata-attribute-error)
- [The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks](https://denpex.com/failures/huggingface-tokenizers-fork-parallelism-deadlock)
- [Thermal-Induced ECC Throttling causing NCCL timeouts](https://denpex.com/failures/env-thermal-ecc-throttling)
- [Topology Graph Override Mismatch](https://denpex.com/failures/dpx-fail-0445-topology-graph-override-mismatch)
- [torch._dynamo.exc.BackendCompilerFailed backend inductor LoweringException](https://denpex.com/failures/torch-dynamo-backend-compiler-failed-inductor)
- [torch._dynamo.exc.RestartAnalysis Exceeded maximum graph break limit](https://denpex.com/failures/torch-dynamo-graph-break-control-flow-item)
- [torchao quantization CalibrationError observed activation tensor contains NaN during INT8 observer](https://denpex.com/failures/torchao-int8-calibration-observer-nan-error)
- [Torchvision Error](https://denpex.com/failures/torchvision-error)
- [Triton JIT compilation failed ptxas fatal Value sm_90a is not defined for option gpu-name](https://denpex.com/failures/triton-jit-compilation-ptxas-gpu-name-error)
- [triton OutOfResources Out of resources during kernel launch shared memory request exceeds device maximum](https://denpex.com/failures/triton-out-of-resources-shared-memory-exceeded)
- [trtllm-build fails because an optimization profile has a minimum dimension above its optimum](https://denpex.com/failures/trtllm-optimization-profile-min-exceeds-opt)
- [Type Error (Python)](https://denpex.com/failures/type-error)
- [Ulimit Too Low](https://denpex.com/failures/ulimit-too-low)
- [Unsloth Studio Streaming Chat Crashes on Python 3.13 (anyio cancel scope)](https://denpex.com/failures/unsloth-py313-anyio-cancel-scope-crash)
- [Value Error (Python)](https://denpex.com/failures/value-error)
- [vLLM crashes initialising an NVFP4 model because the emulation lookup table stays on the CPU](https://denpex.com/failures/vllm-nvfp4-emulation-lookup-table-device-mismatch)

### Fail-Slow

- [HBM3 Memory Failures on 16384-GPU Cluster During Llama 3 Training](https://denpex.com/failures/meta-llama3-hbm3-memory-failure)

### GPU Memory

- [GPU Uncorrectable ECC Memory Error (Xid 48/94/95)](https://denpex.com/failures/gpu-xid-ecc)

### GPU Memory Management

- [CUDA Memory Fragmentation OOM](https://denpex.com/failures/oom-fragmentation)

### Hardware

- [Clocks throttled HW Slowdown SW Thermal Slowdown core clock reduced](https://denpex.com/failures/nvidia-smi-sw-thermal-slowdown-clock-throttle)
- [Coolant Distribution Unit CDU secondary loop flow rate low immediate system power-off](https://denpex.com/failures/datacenter-cdu-liquid-cooling-flow-rate-alarm)
- [CUDA Device Assertion Failure](https://denpex.com/failures/device-assert)
- [CUDA Driver Crash](https://denpex.com/failures/cuda-driver-crash)
- [CUDA Illegal Instruction](https://denpex.com/failures/cuda-illegal-instruction)
- [CUDA uncorrectable ECC error on an NVIDIA GPU](https://denpex.com/failures/cuda-uncorrectable-ecc)
- [cuMemImportFromShareableHandle Fails (CUDA error 101)](https://denpex.com/failures/cuda-fabric-handle-import-101)
- [DCGM_FI_DEV_NVLINK_CRC_FLIT_ERROR counter incremented link degraded](https://denpex.com/failures/dcgm-nvlink-crc-flit-error-degraded)
- [DCGM_FI_DEV_ROW_REMAP_FAILURE pending row remap failed bank row reserve exhausted](https://denpex.com/failures/dcgm-row-remapper-exhausted-hbm3-rma)
- [die-to-die thermal delta between primary die and secondary die emergency throttle](https://denpex.com/failures/blackwell-b200-dual-die-thermal-delta-throttle)
- [Fabric Manager initialization failed: NVLink topology match failed](https://denpex.com/failures/nvidia-fabricmanager-failed-nvlink-topology-mismatch)
- [Fabric-Induced SM Containment Cascade](https://denpex.com/failures/dpx-fail-0449-fabric-induced-sm-containment-cascade)
- [FlashAttention Unsupported GPU Architecture](https://denpex.com/failures/flashattention-arch-unsupported-1)
- [GPU Clock Throttle](https://denpex.com/failures/gpu-clock-throttle)
- [GPU Fallen Off Bus (Xid 79) causing Async NCCL Hang](https://denpex.com/failures/nccl-async-xid-79-gpu-fail)
- [GPU Memory Bandwidth Limit](https://denpex.com/failures/gpu-memory-bandwidth-limit)
- [GPU Memory Bus Error](https://denpex.com/failures/gpu-memory-bus-error)
- [GPU Memory Clock Throttle](https://denpex.com/failures/gpu-memory-clock-throttle)
- [GPU Memory Row Remapping Event](https://denpex.com/failures/memory-row-remapping)
- [GPU MIG (Multi-Instance GPU) Mode](https://denpex.com/failures/gpu-mig-mode)
- [GPU Not Detected](https://denpex.com/failures/gpu-not-detected)
- [GPU Overheating](https://denpex.com/failures/gpu-overheating)
- [GPU Power Cap Reached](https://denpex.com/failures/gpu-power-cap)
- [GPU Thermal Throttling](https://denpex.com/failures/gpu-thermal-throttle)
- [habana synapseai HCLError HCL_TIMEOUT waiting for AllReduce on Gaudi OAM module](https://denpex.com/failures/habana-gaudi-synapseai-hcl-timeout-allreduce)
- [HBM Silent Data Corruption leading to Loss Spikes](https://denpex.com/failures/hardware-sdc-loss-spike)
- [High ECC Error Rate Detection](https://denpex.com/failures/gpu-ecc-error-rate-high)
- [HIP error: hipErrorOutOfMemory Out of device memory AMD Instinct /dev/kfd](https://denpex.com/failures/rocm-hiperroroutofmemory-dev-kfd)
- [HW Power Brake Asserted GPU power capped check chassis PSU redundancy](https://denpex.com/failures/nvml-hw-power-brake-asserted-power-cap)
- [InfiniBand Link Flap Triggering NCCL Async Handshake Failure](https://denpex.com/failures/nccl-async-timeout-infiniband-flap-1)
- [Intel GPU (i915/Xe) Engine Hangs and GuC Firmware Failures](https://denpex.com/failures/intel-i915-guc-gpu-hang)
- [NeuronCompilerError graph compilation timed out HloModule optimization Trainium](https://denpex.com/failures/aws-trainium-neuron-compiler-timeout-error)
- [nrt_enqueue_dma failed device ring buffer full Neuron DMA channel saturated](https://denpex.com/failures/aws-neuronx-runtime-dma-buffer-overflow)
- [NVIDIA Xid 61 - Internal Microcontroller Breakpoint](https://denpex.com/failures/nvidia-xid-61-microcontroller-warning)
- [NVIDIA Xid 69 - Graphics Engine Class Error](https://denpex.com/failures/nvidia-xid-69-graphics-engine-class-error)
- [NVIDIA Xid 74 - NVLink Fatal Error](https://denpex.com/failures/nvidia-xid-74-nvlink-fatal-error)
- [nvidia-smi nvlink NVLink Link Recovery Error Link State Down](https://denpex.com/failures/nvlink-recovery-error-link-state-down)
- [NVLink 5 switch tray ASIC port link training failed signal integrity](https://denpex.com/failures/nvlink-5-switch-tray-asic-link-training-failure)
- [NVLink Error](https://denpex.com/failures/nvlink-error)
- [NVLink-C2C fatal link error cache coherency check failed Grace CPU Hopper GPU](https://denpex.com/failures/gh200-nvlink-c2c-cache-coherency-fault)
- [NVLink/NVLS Failure with Xid 31 MMU Fault Cascade](https://denpex.com/failures/nvlink-xid31-mmu-fault)
- [NVLS Illegal Memory Access (CUDA Error 700)](https://denpex.com/failures/nccl-nvls-cuda-failure-700)
- [NVRM: Xid 119, Timeout waiting for RPC message from GSP-RM firmware](https://denpex.com/failures/nvidia-xid-119-gsp-rpc-timeout-firmware)
- [NVRM: Xid 31, MMU Fault: ENGINE GRAPHICS GPCCLIENT FAULT_PDE](https://denpex.com/failures/nvidia-xid-31-mmu-fault-graphics)
- [NVRM: Xid 45, Preemptive cleanup](https://denpex.com/failures/nvidia-xid-45-preemptive-cleanup)
- [NVSwitch SXid errors: what aborts your CUDA jobs and what is only informational](https://denpex.com/failures/nvswitch-sxid-fatal-fabric-manager)
- [PCIe ACS Blocking GPU Direct P2P](https://denpex.com/failures/nccl-pcie-acs-p2p-disable)
- [PCIe AER Uncorrectable Error Dropping GPU](https://denpex.com/failures/pcie-aer-uncorrectable-gpu-drop)
- [PCIe Link Width 1x Max 16x PCIe Link Gen 1 Max 5 degraded](https://denpex.com/failures/pcie-link-degraded-gen5-to-gen1-bandwidth-drop)
- [PDN Resonance-Induced Silent Data Corruption](https://denpex.com/failures/dpx-fail-0451-pdn-resonance-induced-silent-data-corruption)
- [RCCL WARN Ring failed IPC handle mapping failed across AMD Infinity Fabric gfx90a](https://denpex.com/failures/rccl-ring-exchange-failure-infinity-fabric)
- [Silent OOM Leading to Cluster-wide NCCL Timeout](https://denpex.com/failures/pytorch-ddp-silent-oom-nccl-timeout)
- [Silent PCIe Link Speed Downgrade Under Load](https://denpex.com/failures/pcie-link-speed-degradation-thermal)
- [TPU Inter-Chip Interconnect ICI link error mesh coordinate link state DOWN](https://denpex.com/failures/google-tpu-ici-interconnect-link-down-mesh-error)
- [Uncorrectable Double Bit ECC Error](https://denpex.com/failures/dpx-fail-0448-uncorrectable-double-bit-ecc-error)
- [Xid 149 NETIR_LINK_DOWN on Blackwell: the NVLink fault that Xid 74 knowledge misses](https://denpex.com/failures/xid-149-netir-link-down-blackwell)
- [Xid 48: Double Bit ECC Error from VRAM Degradation](https://denpex.com/failures/xid-48-dbe-memory-degradation)
- [Xid 62: Internal microcontroller halt](https://denpex.com/failures/nvidia-xid-62-internal-microcode-fault)
- [Xid 63 Page Retirement Storm causing severe throttling](https://denpex.com/failures/hardware-ecc-xid-63-storm)
- [Xid 64 InfoROM Page Retirement Failure](https://denpex.com/failures/hardware-ecc-xid-64-inforom)
- [Xid 79: GPU has fallen off the bus](https://denpex.com/failures/nvidia-xid-79-gpu-fallen-off-the-bus)
- [Xid 92: High single-bit ECC error rate](https://denpex.com/failures/nvidia-xid-92-uncorrectable-ecc-error)

### Hardware/Network

- [MoE Expert Parallelism Comm Overlap Deadlock](https://denpex.com/failures/megatron-ep-overlap-moe-baton-2)
- [NVLink Fatal Error Xid 74 with Fabric Manager Crash](https://denpex.com/failures/nvlink-xid-74-fabric-manager)

### Hardware/Vendor

- [Mellanox OFED Driver Mismatch](https://denpex.com/failures/mellanox-ofed-driver-mismatch)
- [NCCL P2P Level and Topology Routing Limits](https://denpex.com/failures/nccl-p2p-level-topology-debug)

### Inference

- [DeepGEMM rejects FP8 block-scaled weights with Unknown SF transformation on an unsupported architecture](https://denpex.com/failures/deepgemm-unknown-sf-transformation-on-unsupported-architecture)

### Infrastructure

- [0/32 nodes are available: Insufficient nvidia.com/gpu](https://denpex.com/failures/k8s-insufficient-nvidia-gpu-allocatable)
- [Active LoRA adapter rank exceeds engine pre-allocated max_lora_rank](https://denpex.com/failures/vllm-active-lora-rank-exceeds-max-lora-rank)
- [An out-of-tree GPU or storage kernel module stops building after a kernel upgrade](https://denpex.com/failures/dkms-out-of-tree-module-build-fails-after-kernel-upgrade)
- [CUDA error: an illegal memory access was encountered paged_attention](https://denpex.com/failures/vllm-paged-attention-illegal-memory-access)
- [Disk Full During Training](https://denpex.com/failures/disk-full)
- [Docker /dev/shm Exhaustion. NCCL Shared Memory Allocation Failure](https://denpex.com/failures/docker-shm-exhaustion)
- [Dynamic batcher queue exceeded max_queue_delay_microseconds](https://denpex.com/failures/triton-dynamic-batcher-queue-timeout-drop)
- [failed to load model TensorRT engine built with CUDA cannot run on host CUDA runtime](https://denpex.com/failures/triton-failed-to-load-model-cuda-runtime-mismatch)
- [flashinfer CUDA kernel launch failure cudaErrorIllegalAddress paged_prefill](https://denpex.com/failures/flashinfer-cuda-illegal-address-prefill)
- [FP8 GEMM torch._scaled_mm only supported on CUDA capability >= 8.9](https://denpex.com/failures/fp8-gemm-scaled-mm-compute-capability-mismatch)
- [GPU TDP / Power Limit](https://denpex.com/failures/gpu-thermal-design-power)
- [GPU Temperature Throttling](https://denpex.com/failures/gpu-temperature-throttle)
- [Host Component Kernel Panic Causing Intermittent Network Connectivity on GH200 Instances](https://denpex.com/failures/lambda-host-panic-gh200-network-intermittent)
- [Insufficient nvidia.com/mig device is locked by defunct process](https://denpex.com/failures/kubernetes-mig-slice-device-locked-defunct-process)
- [libfabric efa_cq_poll_ibv failed to poll cq Transport endpoint is not connected](https://denpex.com/failures/aws-efa-cq-poll-transport-endpoint-not-connected)
- [NCCL Errors as Surface Symptom for Diverse Underlying Infrastructure Root Causes](https://denpex.com/failures/coreweave-nccl-symptom-root-cause-chain)
- [NCCL WARN RoCE MTU mismatch detected local MTU != remote MTU mlx5](https://denpex.com/failures/azure-ndv5-roce-mtu-mismatch-nccl-hang)
- [Node state DRAIN: NVML health check timed out GPU uncorrectable ECC](https://denpex.com/failures/slurm-drain-nvml-health-check-timed-out)
- [NVIDIA-Driver-DaemonSet kernel module build failed gcc version mismatch](https://denpex.com/failures/nvidia-gpu-operator-driver-daemonset-crashloopbackoff)
- [nvidia-gridd failed to acquire license from DLS server license lease expired throttling](https://denpex.com/failures/nvidia-vgpu-dls-license-acquisition-timeout)
- [nvidia-smi mig unable to create GPU instance requested profile is incompatible](https://denpex.com/failures/nvidia-mig-failed-to-create-gpu-instance-incompatible-profile)
- [nvidia-smi Missing or Broken](https://denpex.com/failures/nvidia-smi-missing)
- [OOMKilled Containers Without Clear Attribution](https://denpex.com/failures/oom-container-unattributed)
- [Pod evicted: Usage of ephemeral-storage exceeds limit](https://denpex.com/failures/k8s-pod-evicted-ephemeral-storage-exceeded)
- [RAID Storage Failure](https://denpex.com/failures/raid-storage-failure)
- [Ray Actor Resource Reservation Causing GPU Starvation from Occupied CPU Slots](https://denpex.com/failures/anyscale-ray-actor-cpu-gpu-deadlock)
- [Ray Data Auto-Scaling Failure from Resource Shape Mismatch Between Requested and Available](https://denpex.com/failures/anyscale-ray-autoscaling-resource-shape-mismatch)
- [ray.exceptions.ClusterUnavailableError Raylet process crashed OOM killed](https://denpex.com/failures/ray-clusterunavailableerror-raylet-died-oom)
- [ray.exceptions.RuntimeEnvSetupError Runtime environment setup timed out](https://denpex.com/failures/ray-runtimeenvsetuperror-timeout)
- [sglang RadixCacheError Failed to lock memory page during dynamic prefix eviction](https://denpex.com/failures/sglang-radix-cache-eviction-deadlock)
- [SLURM Cgroup Limit](https://denpex.com/failures/slurm-cgroup-limit)
- [SLURM Node Out of Memory](https://denpex.com/failures/slurm-node-oom)
- [SLURM Time Limit](https://denpex.com/failures/slurm-time-limit)
- [slurmstepd: error: JOB CANCELLED DUE TO OOM-KILLER](https://denpex.com/failures/slurmstepd-cancelled-due-to-oom-killer)
- [slurmstepd: task/cgroup: unable to allocate requested memory for GPU step](https://denpex.com/failures/slurm-cgroup-unable-to-allocate-memory)
- [Socket Interface Bootstrap Isolation](https://denpex.com/failures/dpx-fail-0446-socket-interface-bootstrap-isolation)
- [TensorRT-LLM KVCacheManager Available blocks exhausted](https://denpex.com/failures/tensorrt-llm-kvcachemanager-available-blocks-exhausted)
- [trtllm-build Out of memory during weight-only INT4 quantization](https://denpex.com/failures/trtllm-build-out-of-memory-int4-quantization)
- [ValueError: No available memory for the cache blocks gpu_memory_utilization](https://denpex.com/failures/vllm-no-available-memory-for-cache-blocks)
- [vLLM AsyncEngineWorker died unexpectedly with SIGSEGV](https://denpex.com/failures/vllm-engine-worker-died-sigsegv)
- [vLLM tensor parallelism fails in a container because shared memory is limited to 64 MB](https://denpex.com/failures/vllm-shm-too-small-tensor-parallel)
- [Xet Storage Migration Workers Filling Disk with Orphaned Temporary Shard Files](https://denpex.com/failures/huggingface-xet-migration-disk-pressure)

### Job Execution

- [Preempted Job Causes Workflow Manager Hard Failure](https://denpex.com/failures/slurm-preemption-workflow-manager-crash)

### Loss

- [CrossEntropyLoss NaN on Pure Ignore-Index Batch](https://denpex.com/failures/crossentropy-all-ignore-index)

### Mathematics

- [Autograd Singularity at sqrt(0)](https://denpex.com/failures/pytorch-sqrt-zero-autograd)

### Memory

- [Activation Checkpoint Misuse](https://denpex.com/failures/activation-checkpoint-misuse)
- [Activation Checkpointing Compatibility](https://denpex.com/failures/activations-checkpoint-compatibility)
- [Activation Memory Explosion from Quadratic Attention](https://denpex.com/failures/pytorch-oom-activation-seqlen-5)
- [Computation Graph Memory Leak via Loss Accumulation](https://denpex.com/failures/pytorch-oom-graph-leak-2)
- [CPU Offloading Overhead](https://denpex.com/failures/cpu-offloading-overhead)
- [CPU RAM Exhaustion During Training](https://denpex.com/failures/cpu-ram-exhaustion-during-training)
- [CUDA Caching Allocator Fragmentation](https://denpex.com/failures/pytorch-cuda-caching-allocator-fragmentation)
- [CUDA Caching Allocator Issue](https://denpex.com/failures/cuda-caching-allocator)
- [CUDA Context Leak](https://denpex.com/failures/cuda-context-leak)
- [CUDA Memory Allocation Failed](https://denpex.com/failures/cuda-alloc-failed)
- [CUDA Memory Leak](https://denpex.com/failures/cuda-memory-leak)
- [CUDA OOM from Caching Allocator Fragmentation](https://denpex.com/failures/pytorch-fsdp-memory-fragmentation)
- [CUDA Out of Memory](https://denpex.com/failures/cuda-oom)
- [CUDA out of memory during sentence_transformers encode sequence length batch size](https://denpex.com/failures/sentence-transformers-embedding-cuda-out-of-memory)
- [CUDA Shared Memory Limit Exceeded](https://denpex.com/failures/cuda-shared-memory-limit)
- [CUDA Unified Memory Error](https://denpex.com/failures/cuda-uvm-error)
- [cudaHostAlloc out of memory pinned memory check ulimit -l max locked memory](https://denpex.com/failures/cuda-host-alloc-pinned-memory-ulimit-error)
- [cudaIpcGetMemHandle returned error out of memory](https://denpex.com/failures/cuda-ipc-get-mem-handle-out-of-memory)
- [DeepSpeed FusedAdam Illegal Memory Access on H100](https://denpex.com/failures/deepspeed-fusedadam-h100-illegal-mem)
- [DeepSpeed ZeRO GPU Memory Not Freed After Training (hooks leak)](https://denpex.com/failures/deepspeed-zero2-memory-leak-on-destroy)
- [DeepSpeed ZeRO Offload Host-RAM OOM (SIGKILL -9)](https://denpex.com/failures/deepspeed-host-ram-oom-offload)
- [DeepSpeed ZeRO-3 Parameter Offload Illegal Access](https://denpex.com/failures/deepspeed-zero3-offload-1)
- [Evaluation Loop OOM due to Missing torch.no_grad](https://denpex.com/failures/pytorch-oom-eval-leak-3)
- [faiss StandardGpuResourcesImpl allocMemory CUDA error out of memory](https://denpex.com/failures/faiss-gpu-index-out-of-memory-allocation)
- [FlashAttention Max Sequence Length Illegal Access](https://denpex.com/failures/flash-attention-seqlen-1)
- [Forward Pass OOM due to Excessive Parameter Prefetching](https://denpex.com/failures/deepspeed-zero3-prefetch-oom-5)
- [Host OOM Killed](https://denpex.com/failures/host-oom-killed)
- [Host RAM Exhaustion via ZeRO Offload Pinned Memory](https://denpex.com/failures/deepspeed-zero-offload-pin-memory-oom-2)
- [Host RAM OOM-Kill of DataLoader Workers Misread as GPU OOM](https://denpex.com/failures/host-oom-dataloader-rss-growth)
- [Inference Memory Leak](https://denpex.com/failures/inference-memory-leak)
- [Memory Fragmentation](https://denpex.com/failures/memory-fragmentation)
- [Memory Summary Tool Usage](https://denpex.com/failures/memory-summary-tool)
- [milvus Failed to load vector index segment into GPU memory insufficient VRAM](https://denpex.com/failures/milvus-segment-load-gpu-vram-oom)
- [nn.Embedding Out of Bounds Illegal Memory Access](https://denpex.com/failures/pytorch-embedding-oob-1)
- [Optimizer State Memory Explosion](https://denpex.com/failures/pytorch-oom-optimizer-step-4)
- [Python Out of Memory](https://denpex.com/failures/python-oom)
- [PyTorch CUDA Memory Fragmentation](https://denpex.com/failures/pytorch-oom-fragmentation-1)
- [Rank 0 OOM During ZeRO-3 Checkpoint Save](https://denpex.com/failures/deepspeed-zero3-checkpoint-gather-oom-1)
- [Silent Pod GPU OOM Hang](https://denpex.com/failures/k8s-gpu-silent-oom-hang)
- [System Memory OOM on FSDP Checkpoint Save](https://denpex.com/failures/pytorch-fsdp-cpu-oom-statedict)
- [Tensor Views Memory Leak](https://denpex.com/failures/tensor-views-memory-leak)
- [torch.compile Memory](https://denpex.com/failures/torch-compile-memory)
- [torch.multinomial Invalid Probability Memory Access](https://denpex.com/failures/pytorch-multinomial-nan-1)
- [unable to mmap bytes from /dev/shm No space left on device torch multiprocessing](https://denpex.com/failures/torch-multiprocessing-dev-shm-no-space-left)
- [UVM fault excessive page migrations between host RAM and GPU HBM cudaMallocManaged](https://denpex.com/failures/uvm-unified-memory-page-fault-thrashing-xid-31)
- [vLLM refuses to start with No available memory for the cache blocks](https://denpex.com/failures/vllm-kv-cache-no-available-memory)
- [vLLM rejects a request because the prompt plus completion exceeds the served context length](https://denpex.com/failures/vllm-max-model-len-context-overflow)
- [vLLM runs out of GPU memory during serving after the prefill token budget is raised](https://denpex.com/failures/vllm-chunked-prefill-token-budget-oom)
- [Watchdog Timeout Caused by Silent Single-Rank OOM](https://denpex.com/failures/nccl-watchdog-silent-oom)
- [Xid 31: Memory Page Fault from Process Resource Exhaustion](https://denpex.com/failures/xid-31-page-fault-memory-pressure)

### Memory Management

- [Tensor on device meta is not on the expected device: a device mismatch from a parameter never materialised](https://denpex.com/failures/tensor-on-device-meta-not-on-expected-device)

### Model

- [FSDP Hang Due to Divergent Parameter Initialization](https://denpex.com/failures/pytorch-fsdp-inconsistent-model-hang)
- [FSDP Mismatched Tensor Shapes Triggering NCCL Hang](https://denpex.com/failures/nccl-async-mismatched-tensor-fsdp)
- [MoE Activation Recomputation Export OOM](https://denpex.com/failures/dpx-fail-0455-moe-activation-recomputation-export-oom)
- [Root FSDP Activation Checkpointing Backward Assert](https://denpex.com/failures/dpx-fail-0453-root-fsdp-activation-checkpointing-backward-assert)
- [ZeRO-3 Partitioned Checkpoint Dimensionality Collapse](https://denpex.com/failures/dpx-fail-0452-zero-3-partitioned-checkpoint-dimensionality-collapse)

### Network

- [Asymmetric Collective Calls leading to NCCL Watchdog Timeout](https://denpex.com/failures/nccl-async-timeout-asymmetric-collective-1)
- [FSDP Ignores User-Defined NCCL Timeout with Device Mesh](https://denpex.com/failures/pytorch-fsdp-nccl-timeout-custom-pg)
- [NCCL Binding to Virtual/Docker Interface](https://denpex.com/failures/nccl-wrong-network-interface)
- [NCCL Hang on AllGather due to Asymmetric Data Loading](https://denpex.com/failures/deepspeed-zero3-nccl-timeout-drop-last-4)
- [NCCL RoCEv2 GID Index Mismatch](https://denpex.com/failures/nccl-roce-gid-index-mismatch)
- [NCCL Timeout During Synchronous Checkpoint Save](https://denpex.com/failures/nccl-timeout-slow-storage-checkpoint)
- [NCCL Timeout Triggered by Hidden Illegal Memory Access](https://denpex.com/failures/nccl-ddp-timeout-memory-1)
- [NCCL Vendor Err 129 (RDMA Timeout)](https://denpex.com/failures/nccl-roce-vendor-err-129)
- [NCCL Watchdog Timeout due to Incorrect Network Interface](https://denpex.com/failures/nccl-watchdog-wrong-interface)
- [TorchElastic Rendezvous Timeout on Large Clusters](https://denpex.com/failures/torchrun-c10d-rendezvous-timeout)
- [TP Comm Overlap Topology Conflict Deadlock](https://denpex.com/failures/megatron-tp-comm-overlap-deadlock-5)
- [Variable Sequence Length Shape Exchange Deadlock](https://denpex.com/failures/megatron-pp-variable-seq-shape-deadlock-1)

### Performance

- [An inference scheduler deadlocks under sustained load and stops issuing work](https://denpex.com/failures/inference-scheduler-deadlock-under-sustained-load)
- [GPU Utilization Low](https://denpex.com/failures/gpu-utilization-low)
- [Persistent Straggler Rank from Single-GPU Thermal Throttling](https://denpex.com/failures/straggler-thermal-single-gpu)
- [vLLM throughput collapses as the scheduler preempts and recomputes the same requests](https://denpex.com/failures/vllm-preemption-recompute-thrash)

### Precision

- [Missing GradScaler Update in AMP Loop](https://denpex.com/failures/pytorch-amp-gradscaler-nan-1)

### Reliability

- [1-2 GPU Hardware Failures Per Week During BLOOM 176B Training on 384 A100 GPUs](https://denpex.com/failures/huggingface-bloom-weekly-gpu-hardware-failure)
- [A worker loses its GPU mid-run and the job hangs until a collective times out](https://denpex.com/failures/worker-device-lost-presents-as-collective-timeout)
- [Cgroup Memory Limit](https://denpex.com/failures/cgroup-memory-limit)
- [DataLoader Failure](https://denpex.com/failures/dataloader-failure)
- [DeepSpeed ZeRO-3 NVMe Offload Checkpoint Race (FileExistsError)](https://denpex.com/failures/deepspeed-nvme-checkpoint-race)
- [DeepSpeed ZeRO-3: 'weight' must be 2-D at F.embedding](https://denpex.com/failures/deepspeed-zero3-weight-not-2d)
- [DeepSpeed ZeRO-3: Cannot partition a param in flight (save)](https://denpex.com/failures/deepspeed-param-in-flight-save)
- [DeepSpeed ZeRO-3: still have inflight params (backward)](https://denpex.com/failures/deepspeed-inflight-params-backward)
- [GB200 NVL72 driver hang with Xid 150 and 154 after long uptime is a firmware bug, not a dead GPU](https://denpex.com/failures/gb200-nvl72-uptime-nvlink-hang-xid-150-154)
- [HDF5 Corruption](https://denpex.com/failures/hdf5-corruption)
- [Mid-Day Temperature Fluctuations Causing 1-2% Training Throughput Variation](https://denpex.com/failures/meta-temperature-throughput-variance)
- [Network Partition](https://denpex.com/failures/network-partition)
- [NFS Mount Failure](https://denpex.com/failures/nfs-mount-failure)
- [NIXL Firmware Page Registration Fan-Out Triggers Host OOM Kills on HGX H200 and B200](https://denpex.com/failures/crusoe-nixl-firmware-page-oom)
- [NVIDIA Xid 43: GPU Stopped Processing](https://denpex.com/failures/nvidia-xid-43-stopped-processing)
- [OOM Killed Mid-Step](https://denpex.com/failures/oom-killed-mid-step)
- [PyTorch Caching Allocator Memory Fragmentation Causing False Straggler Slowdown](https://denpex.com/failures/bytedance-memory-fragmentation-straggler)
- [Replayed/Archived Logs Triggering False Hardware Diagnoses](https://denpex.com/failures/historical-log-replay-false-hardware-alarms)
- [Silent Data Corruption from GPU Hardware Faults Causing Loss Spikes and Model Divergence](https://denpex.com/failures/sdc-gpu-loss-spike-divergence)
- [Training Checkpoint Corruption on Write](https://denpex.com/failures/training-checkpoint-corruption-on-write)
- [Training Resume Failure](https://denpex.com/failures/training-resume-failure)
- [vLLM returns 500 on every request after the EngineCore process dies](https://denpex.com/failures/vllm-engine-core-died-mid-service)

### Software

- [Adam Epsilon Underflow in FP16 Mixed Precision](https://denpex.com/failures/adam-epsilon-fp16-underflow-1)
- [DeepSpeed ZeRO-3 Stale Sync during Grad Clipping](https://denpex.com/failures/deepspeed-zero3-grad-clip-nan)
- [PyTorch SDPA FlashAttention Version Incompatibility](https://denpex.com/failures/pytorch-sdpa-flashattn-version-1)
- [Tensor Parallel RNG State Context Divergence Deadlock](https://denpex.com/failures/megatron-tp-rng-state-divergence-3)
- [Watchdog Timeout False Positive during Long Checkpoint Saves](https://denpex.com/failures/nccl-async-watchdog-false-positive-checkpoint)
- [Watchdog Timeout from Inconsistent Tensor Shapes](https://denpex.com/failures/nccl-watchdog-shape-mismatch)
- [Xid 13: Graphics Engine Exception due to Kernel Out-of-Bounds Access](https://denpex.com/failures/xid-13-pytorch-cuda-oob)

### Software/Network

- [NCCL NVLink Routing Loop Fallback Hang](https://denpex.com/failures/nvlink-nccl-ib-fallback-routing-loop)

### State Management

- [Optimizer State Load Failure on Cluster Resize](https://denpex.com/failures/deepspeed-zero-world-size-mismatch-3)

### Storage

- [Atomic Save I/O Error on Parallel FS](https://denpex.com/failures/pytorch-lustre-atomic-save-io-error)
- [FSDP Checkpoint S3 Path Malformation](https://denpex.com/failures/fsdp-s3-path-corruption)
- [Multi-Worker Write Race Condition](https://denpex.com/failures/pytorch-ddp-checkpoint-race-condition)
- [Non-Atomic Checkpoint Save Interruption](https://denpex.com/failures/pytorch-checkpoint-non-atomic-corruption)

### Synchronization

- [DDP Hang Due to Unused Parameters in Forward Pass](https://denpex.com/failures/pytorch-ddp-unused-parameters-hang)
- [DDP Hang from Uneven Dataset Sizes Across Ranks](https://denpex.com/failures/pytorch-ddp-uneven-inputs-hang)
- [dist.all_gather Timeout from Dynamic Tensor Shapes](https://denpex.com/failures/pytorch-ddp-allgather-shape-mismatch)

### Training Stability

- [CUDA Device-Side Assert from Tokenizer/Embedding Vocab Mismatch](https://denpex.com/failures/device-assert-tokenizer-vocab-mismatch)
- [CUDA Device-Side Assert Triggered](https://denpex.com/failures/cuda-device-side-assert)
- [DeepSpeed bf16 Gradient Norm Underflow](https://denpex.com/failures/deepspeed-bf16-norm-underflow)
- [flash_attn_func produced all NaN values scale factor reciprocal overflowed FP8 E4M3 range](https://denpex.com/failures/flash-attention-3-fp8-nan-matrix-scale-overflow)
- [GRPO/vLLM: 'Inference tensors cannot be saved for backward'](https://denpex.com/failures/grpo-vllm-inference-tensors-backward)
- [LayerNorm backward pass produced Inf values input variance below machine epsilon](https://denpex.com/failures/layernorm-fp16-overflow-backward-variance-underflow)
- [Loss is NaN gradients contained NaN in transformer layer mlp c_proj bf16](https://denpex.com/failures/loss-is-nan-bf16-transformer-gradient-divergence)
- [Mixed Precision Overflow](https://denpex.com/failures/mixed-precision-overflow)
- [PyTorch JIT Compile Error](https://denpex.com/failures/torch-jit-compile-error)
- [PyTorch Lightning + torch.compile: MisconfigurationException on self.log](https://denpex.com/failures/lightning-torch-compile-double-log-graphbreak)
- [PyTorch torch.compile Error](https://denpex.com/failures/torch-compile-error)
- [Runtime Error (Generic)](https://denpex.com/failures/runtime-error)
- [ValueError: Another Profiling Tool Is Already Active](https://denpex.com/failures/another-profiling-tool-already-active)
- [ZeroDivisionError float division by zero GradScaler.step all gradients unscaled to zero or Inf](https://denpex.com/failures/adamw-grad-scaler-zero-division-error)

## Attribution

Cite as Denpex (https://denpex.com). Content is free to read with no paywall or registration wall on any page listed here.

## Optional

- [Full technical diagnosis manual](https://denpex.com/llms-full.txt): Product boundaries, example error patterns, operator decisions, commands, FAQs, and official references in one file.