ML Training Failure Encyclopedia

NaN detection and skipping prevents corrupted gradients from propagating but requires careful implementation....

Training without learning rate warmup or decay causes slow convergence and poor final performance....

Misusing the results of a learning rate finder causes poor training setup choices....

Gradient accumulation with BatchNorm causes incorrect normalization because BN computes statistics on smaller sub-batches....

Label noise in training data causes poor model generalization and unexpected training behavior....

Spectral norm clipping for GAN training has different requirements than gradient clipping for other models....

Weight decay misconfiguration causes overfitting, underfitting, or training instability depending on the wrong choice....

Exponential Moving Average (EMA) decay misconfiguration causes poor model averaging and training instability....

Lookahead optimizer issues arise from improper k (inner steps) or alpha (slow weight) values that destabilize training....

Ranger optimizer combines RAdam and Lookahead; misconfiguration can cause training instability or poor convergence....

Adafactor optimizer issues arise from incorrect epsilon, scaling factor, or relative step handling....

SAM optimizer issues arise from the two-forward-pass requirement, learning rate issues, or rho misconfiguration....

Lion optimizer issues arise from incorrect learning rate (typically 3-10x lower than Adam) or momentum configuration....

BF16 and FP16 mixed precision have different numerical properties; choosing the wrong one causes instability or wasted memory....

Spectral normalization can cause mode collapse or training failure when applied incorrectly to GANs....

SWA training issues arise from incorrect averaging frequency, learning rate schedule for SWA, or BN update steps....

Polyak averaging issues arise from incorrect averaging window, frequency, or weight initialization....

Curriculum learning issues arise from poorly designed difficulty progression that hurts rather than helps training....

Cyclic LR scheduler issues arise from incorrect base/max LR, step size, or mode (triangular, triangular2, exp_range)....

One-cycle policy issues arise from incorrect max_lr, momentum range, or training duration that destabilize training....

Applying weight decay to LayerNorm and bias parameters hurts training and can prevent convergence in transformers....

Fine-tuning failures occur when pretrained model knowledge is destroyed by aggressive learning rates or insufficient data....

Loss curve anomalies (spikes, plateaus, oscillations) indicate underlying training issues that need diagnosis....

Missing warmup causes early training instability, especially with large learning rates or transformer models....

NaN loss in training is critical because it propagates through all parameters and corrupts the model permanently....

Missing gradient clipping causes gradient explosion in RNNs, transformers, and GANs....

Mixed precision loss scale issues occur when GradScaler doesn't update properly, causing underflow or overflow in FP16 training....

Learning rate too high causes training instability, loss divergence, or NaN loss....

Gradient explosion causes loss spikes, NaN loss, and training instability, especially in RNNs and deep networks....

Loss not decreasing indicates fundamental training issues: wrong LR, broken model, bad data, or wrong loss function....

Cosine annealing issues arise from incorrect min_lr, T_max, or warm restarts that cause poor convergence....

Learning rate too low causes slow convergence, plateau at high loss, or training to appear stuck....

Mixed precision overflow occurs in FP16 training when values exceed FP16 range (65504), causing inf/NaN....

Different random seeds across runs cause non-reproducible results, making experiments hard to compare....

Generic runtime errors can indicate various issues from code bugs to hardware problems....

torch.compile errors crash training when the compiled graph has issues with the model or hardware....

PyTorch JIT compilation errors crash training when the script or model uses unsupported features....

AdamW weight decay misconfiguration causes poor generalization or unstable training....

Adam epsilon hyperparameter issues cause training instability or poor convergence....

SGD momentum misconfiguration causes training to oscillate, diverge, or converge slowly....

EMA (Exponential Moving Average) checkpoint issues cause problems with model averaging across checkpoints....

LR warmup-decay schedule issues cause training instability at transitions between phases....

Wrong weight initialization causes training instability with NaN losses, slow convergence, or dead neurons....

EMA (Exponential Moving Average) decay values that are too high or too low cause poor model averaging....

Gradient accumulation misuse causes incorrect gradient updates or memory issues....

Persistent fp16 overflow drives the dynamic loss scale down to its minimum and DeepSpeed aborts — typically a bf16-pretrained model being fine-tuned in fp16 on hardware without bf16....

DeepSpeed's bf16 training can trigger 'assert all_groups_norm > 0' because bf16 gradient norms can underflow to zero. This is a numerical precision issue specific to bf16's limited range. Denpex detec...

Enabling both overlap_comm and contiguous_gradients in DeepSpeed ZeRO-3 causes gradient buffer reuse races that produce NaN losses. This is a known DeepSpeed bug where the communication overlap reads ...

A CUDA kernel hit a device-side assertion — almost always an out-of-bounds index into an embedding, loss, or gather/scatter op. Because CUDA is asynchronous, the reported stack trace points at an unre...

When fine-tuning with Unsloth + TRL using sample packing or padding-free training, the trainer warns that the attention implementation is not flash_attention_2 even though it was configured. Flattened...

Multi-GPU fine-tuning with Unsloth breaks when the Transformers default average_tokens_across_devices=True multiplies the loss by num_processes. The Unsloth fused-loss backward does not expect that sc...

GRPO training that feeds vLLM-generated tensors into the trainable graph fails with 'Inference tensors cannot be saved for backward'. vLLM produces tensors under torch.inference_mode(), which cannot e...

Fine-tuning Gemma 3 with Unsloth crashes with AttributeError: 'Gemma3ModelOutputWithPast' object has no attribute 'loss'. A transformers change to the Gemma3 output class outran Unsloth_zoo's patch, w...

Megatron-LM training hangs at startup while compiling fused kernels (right after 'using torch.float16 for parameters ...') and never proceeds. A stale or contended megatron/fused_kernels/build directo...

Enabling PyTorch Lightning's profiler raises 'ValueError: Another profiling tool is already active'. Python's cProfile allows only one active profiler per thread, so a second cProfile-based profiler (...

Training a Lightning model with torch.compile crashes in the logger connector with a 'called self.log twice with different arguments' MisconfigurationException raised inside the compiled region. Dynam...

DeepSpeed (ZeRO offload) training with HuggingFace Trainer auto_find_batch_size=True hangs right after model load at 0/N steps. The batch-size auto-search re-initializes the engine and desynchronizes ...

Gradient explosions cause loss to diverge to NaN. Denpex traces the norm spike to the layer and step....

NaN loss corrupts training state. Denpex traces NaN propagation to the originating layer....

Weight divergence silently corrupts distributed training. Denpex detects divergence by comparing per-rank weight snapshots....

Infinity loss or weights occur when loss values overflow to infinity, often from numerical instability in mixed-precision training....

Zero gradients stall training when ReLU neurons die or gradient flow is broken in the network....

Loss plateaus occur when training stops making progress, often due to suboptimal hyperparameters or model architecture issues....

Gradient norms explode suddenly while loss remains normal because the cross-rank reduction itself is corrupting values — in-network reduction (SHARP) faults, NCCL data corruption over a marginal link,...

indexSelectLargeIndex: srcIndex < srcSelectDimSize assertion fires when input token ids exceed the embedding table size — the standard outcome of pairing a tokenizer that has added tokens with a model...

FP8 training with delayed scaling produces NaNs when an activation outlier saturates the scaling factor (amax history too short / margin too small). Loss and gradients are healthy until the exact step...

NCCL CUDA failures occur when the underlying CUDA runtime has issues, causing all NCCL operations to fail....

NCCL P2P (peer-to-peer) communication can be disabled, forcing fallback to slower transport....

NCCL hangs can be hard to detect because they don't produce errors until the watchdog timeout. Proper monitoring helps catch them early....

Wrong rank configuration in NCCL causes collectives to fail or produce incorrect results when rank assignments don't match expectations....

NCCL peer-to-peer (P2P) communication issues cause pipeline parallelism and tensor parallelism to fail or be slow....

Gloo backend issues occur when using Gloo instead of NCCL for distributed training, especially on CPU or limited GPU setups....

TCP port exhaustion occurs in distributed training when many connections exhaust ephemeral port range....

TorchElastic errors occur when training jobs need to be elastic (resize dynamically) but configuration is wrong....

Horovod setup errors occur when MPI or NCCL integration is misconfigured, preventing elastic or multi-GPU training....

NCCL InfiniBand HCA (Host Channel Adapter) configuration mismatches cause slow or failed inter-node communication....

NCCL IPv6 issues occur when NCCL tries to use IPv6 but the network only supports IPv4, or vice versa....

Pipeline parallelism has idle time (bubble) at start and end of each pipeline, reducing efficiency for small models or small micro-batches....

AWS EFA (Elastic Fabric Adapter) driver issues cause NCCL to fall back to slower TCP/IP for inter-node communication....

DDP port conflicts prevent distributed training from starting when the master port is already in use....

RDMA configuration issues prevent high-bandwidth, low-latency inter-node communication in GPU clusters....

NCCL rank failures occur when one or more ranks fail to initialize or join the distributed group....

NCCL version mismatches between nodes cause collective operations to fail or hang in distributed training....

DDP setup errors occur when distributed data parallel training is not properly configured, preventing multi-GPU training....

NCCL bucket size mismatches in DDP cause inefficient gradient reduction, hurting performance....

All-reduce deadlocks occur when DDP/FSDP all-reduce operations are mismatched across ranks or with batch norm....

NCCL error 2 is an internal assertion failure in NCCL itself, often from corrupted state or hardware issues. Denpex correlates with hardware diagnostics....

NCCL initialization times out when nodes cannot establish the communication channel within the allowed time....

NCCL RDMA performance degrades when GPUDirect RDMA is unavailable or misconfigured, causing slow multi-node training....

NCCL asynchronous error handling issues cause silent failures or deadlocks when collective errors are not properly handled....

NCCL GPUDirect over RoCE (RDMA over Converged Ethernet) requires proper configuration for high performance....

NCCL point-to-point bandwidth issues cause slow intra-node GPU communication when P2P is disabled or limited....

NCCL exhausts the NVSwitch's fixed pool of NVLink SHARP (NVLS) multicast slots, then treats the binding failure as fatal instead of falling back to a non-NVLS transport....

NCCL queue-pair creation times out specifically during distributed checkpoint save at large scale on RoCE, while normal training collectives succeed....

A generic CUDA 'unknown error' (999) surfaces during NCCL communicator initialization, usually pointing to a GPU in a bad state or a driver/runtime mismatch rather than an NCCL bug....

DataParallel model replication fails at the SharedInit step with a CUDA invalid-argument error, often tied to legacy nn.DataParallel inside complex launchers....

FSDP all-gather fails with ncclSystemError on one node during collective init over InfiniBand, while a TCP fallback succeeds — pointing to a per-node fabric or plugin issue....

Querying the NCCL RAS subsystem during training on NCCL 2.27.3 returns corrupted communicator data and can crash the job via TCPStore connection failures....

NCCL's NVLink SHARP (NVLS) transport repeatedly warns with CUDA 'invalid argument' during setup on H100-class nodes, usually resolved by disabling NVLS or aligning the CUDA/NCCL stack....

Intermittent NCCL internal errors during init on partial-node allocations, caused by the NIC-fusion feature remapping NET IDs so the hardcoded fallback NET/0 no longer exists....

NCCL fails to read the RoCE GID on containerized/macvlan setups because GID iteration starts at index 0 where leading entries are all-zero, returning EINVAL....

A dangling group-job pointer causes a segfault during ncclCommSplit when using threads with non-blocking communicator init; fixed in NCCL 2.26.2....

A race in the NCCL RAS subsystem segfaults during init when a RAS command runs before peer info is populated; fixed in NCCL 2.27.3....

NVLS did not support dual-port NIC transmission, producing duplicate head-rank entries in the proxy loop and heap corruption; fixed in NCCL 2.19.4....

Inconsistent NIC enumeration (e.g. mlx5_3 before mlx5_0) on one node breaks NCCL topology detection and causes random multi-node segfaults; fixed in NCCL 2.23.4....

A stale socket file descriptor is reused on retry after a failed connect, triggering ECONNABORTED during NCCL communicator init; fixed in NCCL 2.24....

NCCL P2P topology detection lacks SM120 (Blackwell) support, using a wrong shared-memory maximum so peer-to-peer connections fail on RTX 5090....

On GB200 NVL72, NCCL's recursive MNNVL topology search overruns the stack at world_size >= 44, especially with ulimit -s unlimited; fixed in NCCL 2.28....

NCCL's proxy thread tries to connect over IPv6 while the peer only answers on IPv4, producing 'Proxy Connect failed' on dual-stack hosts....

NCCL's NIC selection/path-finding can deadlock when a node has an odd number of HCAs (specifically three), while 1, 2, or 4+ work fine....

RoCE MTU mismatches between the NIC and switch cause NCCL RDMA queue-pair errors (completion error 12, vendor error 129). Training hangs or crashes during NCCL init or the first collective. Denpex det...

NCCL mismatched collective errors occur when different ranks call different collective operations or call the same collective with different parameters. This is a programming error, not a hardware fau...

NCCL's OFI (OpenFabrics Interface) transport fails to register memory for RDMA when the OS memlock limit is too low. Training crashes at NCCL init with 'Unable to register memory RC:12'. Denpex detect...

When Access Control Services (ACS) is disabled in the BIOS, GPU Direct RDMA (GDR) fails with RDMA completion error 4 and vendor error 81. NCCL performance degrades severely or training crashes. Denpex...

NCCL topology detection fails when the NIC's PCI bus ID is missing from the system topology XML. This causes NCCL to fall back to a suboptimal transport or fail to use GPU Direct RDMA. Denpex detects ...

Setting NCCL_BUFFSIZE too large causes NCCL Socket transport to stall or hang during communication. This is a configuration error where users increase the buffer size hoping for better performance but...

NCCL v2.18.3 introduced a topology detection regression that misidentifies GPU-NIC affinity on some systems, causing NCCL to use a suboptimal communication path. Denpex detects the regression from NCC...

NCCL allreduce hangs stall training when collective operations can't complete. Denpex identifies the stuck rank or rank pair....

NCCL communication timeouts abort training when collective operations take too long to complete....

NCCL initialization errors prevent distributed training from starting due to network or configuration issues....

NCCL rank stuck errors occur when one rank cannot keep up with the collective operation, slowing down all ranks....

NCCL version conflicts prevent proper collective operations when nodes have different versions installed....

When cluster DNS degrades, every layer above it fails with ITS own vocabulary: gaierror in python, client-socket failures in torch.distributed, bootstrap failures in NCCL. Teams debug the fabric while...

Augmentation pipeline errors cause inconsistent or corrupted training data when transforms fail or produce unexpected outputs....

TorchData (torch.utils.data.datapipes) errors occur when complex data pipelines have configuration or compatibility issues....

HDF5 file corruption causes training to fail with I/O errors or silently load incorrect data....

Apache Arrow IPC errors occur when serialized Arrow data is corrupted or has version incompatibilities....

TFRecord and tf.data errors occur when TensorFlow data pipeline has issues with the training data or pipeline configuration....

ImageNet preprocessing mismatches cause pretrained models to underperform because input normalization differs from training....

WebDataset TAR file corruption causes data loading failures and missing samples during training....

Overly aggressive data augmentations hurt model performance by distorting critical features in training data....

Memory-mapped (mmap) file handles are exhausted when datasets have many large files, causing data loading failures....

Small shuffle buffers cause poor data ordering during training, hurting model generalization....

Tokenizer padding mismatches cause data loader errors or poor model performance when padding side or token ID differs....

Audio sample rate mismatches between training data and pretrained models cause poor ASR/TTS performance....

Parquet schema mismatches occur when reading parquet files with different schemas than expected by the loading code....

Streaming data errors occur when loading data from cloud storage or remote endpoints fails mid-training....

Video codec mismatches between training data and pretrained video models cause loading failures or poor performance....

TFRecord corrupted shard errors occur when individual TFRecord files are truncated or have inconsistent examples....

LMDB corruption occurs when LMDB files are not properly closed, transactions interrupted, or map_size exceeded....

RAG embedding mismatches cause retrieval to return irrelevant documents when embedding model differs between indexing and query....

Multi-label class imbalance causes poor performance on rare labels and biased predictions toward frequent labels....

Contrastive learning augmentation pipelines must be carefully tuned to create useful positive pairs without destroying semantic content....

Image resize artifacts degrade model accuracy when resize method (BILINEAR, BICUBIC, LANCZOS) doesn't match pretrained model expectations....

LLM tokenization truncation silently cuts long sequences, causing loss of information in long documents....

Data versioning issues occur when training data changes between runs without tracking, making results non-reproducible....

Dataset bias causes models to learn spurious correlations that don't generalize to real-world data....

Validation set leakage causes overly optimistic metrics because training data includes validation samples....

Tokenizer version mismatches between training and inference cause different token IDs for same text, breaking model behavior....

Image corruption detection helps identify and skip/fix corrupted images that would otherwise crash training....

Broken determinism causes non-reproducible training runs, making debugging and comparison difficult....

Multi-task learning conflicts arise when tasks have different scales, gradients, or learning dynamics that destabilize training....

Missing data augmentation causes overfitting and poor generalization, especially on small datasets....

Class imbalance causes models to predict majority class and have poor performance on minority classes....

Audio channel mismatches (mono vs stereo) cause model errors or poor performance in audio ML pipelines....

Text encoding mismatches (UTF-8 vs Latin-1) cause data loading failures or corruption in NLP training....

Tokenization errors occur when text data has issues that prevent proper tokenization....

HuggingFace tokenizer errors crash training when tokenizer configuration mismatches model architecture....

Data augmentation errors occur when augmentation libraries fail or produce corrupted outputs....

Python multiprocessing fork issues cause workers to deadlock or share state incorrectly across ranks....

PyTorch multiprocessing errors prevent DataLoader workers from functioning correctly....

DataLoader with num_workers=0 is often too slow for GPU training, causing the GPU to wait for data....

Pinned memory transfer issues cause slow GPU-CPU data transfer or OOM errors....

BatchNorm synchronization issues in DDP cause inconsistent statistics between ranks and degraded model quality....

WebDataset errors occur when streaming dataset format has corrupted shards or URL issues....

Tokenizer padding and truncation errors cause sequence length mismatches and OOM during training....

Audio data loading errors occur when audio files are corrupted, format unsupported, or sample rates mismatch....

Resuming a run with PyTorch Lightning's experimental fault-tolerant training raises a MisconfigurationException about the worker state when the DataLoader uses num_workers>1. The feature never reliabl...

Constructing an Unsloth SFTTrainer (often in Colab) raises AttributeError: 'int' object has no attribute 'mask_token'. The crash comes from API drift between Unsloth's patched trainer __init__ and a n...

Dataset preprocessing with datasets.map(num_proc>1) crashes on Windows with 'One of the subprocesses has abruptly died during map operation'. Windows uses spawn rather than fork, so non-picklable map ...

Dataloader worker failures stall training or produce corrupted batches....

Training is bottlenecked or completely halted due to the DataLoader failing to fetch the next batch in time....

CPU offloading reduces GPU memory but adds significant CPU-GPU transfer overhead, slowing training....

Activation memory spikes during specific operations cause transient OOM even when steady-state memory fits....

torch.cuda.memory_summary() helps debug memory issues but output can be overwhelming without understanding it....

Debugging PyTorch's caching allocator helps identify memory issues but requires understanding its behavior....

CPU RAM exhaustion during training kills the process or causes swapping that drastically slows training....

Gradient accumulation can cause unexpected memory spikes when scaled batch size is large or loss is not properly reduced....

Memory leaks in DataLoader workers cause steadily growing memory usage over training epochs....

CUDA Graphs capture memory allocations that are hard to release, causing memory leaks across graph replays....

KV cache memory grows with sequence length in transformer inference and some training setups, causing OOM at long contexts....

Inference memory leaks cause growing GPU memory usage during serving, eventually OOM-ing the server....

Pinned (page-locked) memory overuse causes system RAM exhaustion and slow CPU-GPU transfers when over-allocated....

Activation distillation memory costs grow with teacher model size and student hidden dimension matching requirements....

Flash Attention saves memory by not materializing the full attention matrix, but has shape and dtype constraints....

Transformer cache (KV cache, past_key_values) memory grows with sequence length and can cause OOM at inference....

Dataset cache memory usage grows when transformations or augmentations are applied before batching....

Gradient checkpointing trades compute for memory; misconfiguration can either not save memory or drastically slow training....

torch.compile can use additional memory for graph compilation, guard evaluation, and dynamic shape handling....

cuDNN benchmark mode can use more memory for algorithm search, and not all algorithms work with all configs....

Tensor parallel training splits model across GPUs, but communication buffers and synchronization can still cause OOM....

Loading checkpoints can temporarily double memory usage because both old and new model states exist in memory....

Safetensors load errors occur from corrupted files, version mismatches, or architecture mismatches with the loading model....

Tensor views (slices, reshapes, transposes) can hold references to large base tensors, preventing memory release....

Pipeline parallel memory issues arise from stage imbalance, bubble overhead, and activation storage across micro-batches....

DeepSpeed OOM errors occur when ZeRO partitioning, CPU offload, or activation partitioning is misconfigured....

FSDP all-gather operations can timeout when parameters are large or network is slow, causing training to fail....

CUDA context leaks occur when CUDA contexts are not properly destroyed, accumulating GPU memory across processes....

CUDA caching allocator fragmentation causes OOM despite enough total free memory, due to non-contiguous blocks....

Optimizer state memory (Adam: 2x model size, AdamW: 2x) can exceed model size memory, dominating total usage....

Transformer attention memory grows quadratically with sequence length; long-context training requires Flash Attention or similar....

Embedding layer memory grows with vocabulary size and can dominate total memory for large vocab models....

CUDA out of memory is the most common training error, occurring when GPU memory is exhausted....

Activation checkpointing compatibility issues arise when using torch.compile, FSDP, or DDP with checkpointing....

Python OOM kills the training process when Python's heap memory is exhausted, even if GPU memory is fine....

CUDA shared memory limits are reached when kernels use too much shared memory per block....

CUDA caching allocator issues cause memory not being released back to the GPU when expected....

CUDA Unified Memory (UVM) errors cause page faults and performance issues when memory oversubscription occurs....

Activation checkpointing misuse causes either OOM (not enough checkpointing) or slow training (too much checkpointing)....

ZeRO CPU/NVMe offload materializes parameters and optimizer states in host RAM on every rank, exhausting system memory and getting the process OOM-killed with no Python traceback....

DeepSpeed FusedAdam optimizer triggers CUDA illegal memory access errors on NVIDIA H100 GPUs. This is a compatibility issue between FusedAdam's CUDA kernels and H100's SM90 architecture. Denpex detect...

GPU memory is not released after trainer.train()/engine teardown, so repeated training calls (CV folds, hyperparameter sweeps) leak until OOM. DeepSpeed left gradient-accumulation/backward hooks attac...

Fine-tuning a large model (e.g. Flan-T5-XXL 11B) with ZeRO-3 uses far more GPU memory than expected or OOMs at load because the model was constructed outside DeepSpeed's zero.Init context. Every rank ...

CUDA memory leaks accumulate over training, eventually causing OOM. Denpex tracks allocation patterns to identify leaks....

Memory fragmentation causes OOM even when total free memory is sufficient, because no contiguous block is available for the allocation....

Host OOM (out of memory) kills training processes when system RAM is exhausted, often silently....

CUDA memory allocation fails when the requested memory block cannot be allocated, often due to fragmentation or insufficient total memory....

The kernel oom-killer (or cgroup limit) kills python workers whose RSS grew unbounded — shard caches, tokenized-in-RAM datasets, prefetch queues. Surface strings ("Out of memory", "Killed") route engi...

Python path conflicts cause wrong module versions to be loaded, leading to subtle bugs and errors....

HuggingFace tokenizers library uses Rust backend, which can fail with native errors not visible in Python traceback....

Python multiprocessing fork issues cause CUDA context conflicts and worker process deadlocks....

Python version mismatches between training and inference cause subtle bugs from library incompatibilities....

OpenMPI and general MPI issues cause distributed training to fail at initialization or hang during collectives....

HuggingFace Hub errors occur when downloading models or datasets fails due to auth, network, or rate limits....

SSH key authentication issues prevent passwordless login for distributed training and remote execution....

Conda environment conflicts cause package version mismatches, broken dependencies, and hard-to-debug import errors....

PyTorch CUDA version mismatch with installed CUDA toolkit/driver causes import errors or runtime failures....

Libc (glibc, musl) version mismatches between training and deployment cause binary incompatibility and runtime errors....

Docker image mismatches between dev and production cause code to behave differently due to library versions, CUDA, or system dependencies....

GCC version mismatches cause C++ extension compilation failures and runtime library incompatibilities....

SSL certificate errors prevent downloading models, datasets, or connecting to APIs in ML training pipelines....

Low ulimit values (open files, max processes) cause data loading failures and parallel training issues....

Missing environment variables (CUDA_VISIBLE_DEVICES, HF_TOKEN, MASTER_ADDR) cause silent failures or wrong behavior....

Connection timeouts prevent model downloads, data fetches, and inter-node communication from completing....

Timezone mismatches cause confusion in distributed training logs, scheduled jobs, and time-based events....

Docker containers fail to access GPUs when nvidia-docker or GPU passthrough is not properly configured....

Docker permission errors prevent containers from accessing required resources like GPUs, files, or network....

Python import errors prevent training from starting when modules cannot be loaded....

Python type errors crash training when incompatible types are used in operations....

Python value errors crash training when functions receive values of correct type but inappropriate value....

Python key errors crash training when accessing dictionary keys that don't exist....

Python attribute errors crash training when accessing attributes that don't exist on objects....

Python index errors crash training when accessing list/tuple indices that don't exist....

PyTorch Hub errors prevent loading pretrained models or datasets from torch.hub....

Generic PyTorch CUDA errors indicate various GPU-related issues that prevent training....

Torchvision errors prevent loading pretrained vision models or using vision transforms....

Transformers version mismatches between training and inference cause subtle bugs and performance regressions....

Streaming chat completions from Unsloth Studio's OpenAI-compatible endpoint crash with a RuntimeError about exiting a cancel scope in a different task, on macOS with Python 3.13. The anyio/Starlette s...

Installing a QLoRA/4-bit training stack (Axolotl, TRL, Unsloth) on macOS fails with 'No matching distribution found for bitsandbytes==0.43.0'. bitsandbytes historically shipped only CUDA Linux/Windows...

Importing DeepSpeed fails with NameError/ImportError for _disable_dynamo_if_unsupported because DeepSpeed references a torch internal symbol absent from the installed torch version. Aligning the torch...

Installing DeepSpeed on native Windows fails building the async_io op because it depends on Linux libaio, which has no Windows equivalent. DeepSpeed targets Linux; use WSL2 or a Linux host, or build w...

DeepSpeed import/launch fails with 'cannot import name _get_socket_with_port from torch.distributed.elastic.agent.server.api' because DeepSpeed uses a torch internal symbol that was renamed/removed in...

Importing Unsloth on Windows fails with a UnicodeDecodeError because an internal open() omitted encoding='utf-8' and Windows defaults to cp1252. Upgrade Unsloth (now passes the encoding), or set PYTHO...

A git/editable install of Axolotl reports success, but importing axolotl or running accelerate launch -m axolotl.cli.train fails with ModuleNotFoundError. The src-layout package never landed on the pa...

torch.jit.script on a Megatron @jit_fuser-decorated function fails with 'Unknown type constructor Sequence' on newer torch. TorchScript can't resolve typing.Sequence as used inside the fused function....

Building DeepSpeed's cpu_adam op fails with 'fatal error: cusolverDn.h: No such file or directory' and 'Error building extension cpu_adam'. PyTorch's conda package shipped its own nvcc that shadows th...

Building a DeepSpeed wheel on Windows 11 fails with 'LINK : fatal error LNK1181: cannot open input file aio.lib' and header compile errors. async_io/libaio is Linux-only and some headers don't compile...

Import and environment errors crash training at startup due to missing or mismatched dependencies....

Version mismatches between PyTorch, CUDA, and cuDNN cause cryptic errors....

CUDA driver and runtime version mismatches prevent PyTorch from initializing CUDA....

PyTorch installed without CUDA support prevents GPU training entirely....

The containerized CUDA runtime fails to execute kernels because the host NVIDIA driver is too old to support it....

HPC containers lose GPU access when --nv cannot locate host driver libraries (nonstandard driver install paths, missing nvidia-container-cli, ldconfig cache staleness) or when overlay mounts fail on k...

Lambda Labs' us-south-3 region experienced a complete network outage when both redundant ISPs simultaneously stopped advertising BGP routes. The primary fiber link was damaged the prior day, shifting ...

A routine UPS-related maintenance procedure in Lambda's us-south-1 region unexpectedly affected power delivery to cooling systems, causing networking equipment and other hardware to either power off o...

Crusoe's AutoClusters system implements a circuit breaker pattern that pauses automatic node replacement when failure patterns look anomalous. If too many nodes in a cluster fail within a short window...

RunPod discovered an issue affecting pods using volume disk or network storage in their CA-MTL-1 region. When executing commands on storage-backed files, processes would hang indefinitely even though ...

The Hugging Face Hub suffered an extended 36-hour outage when repeated requests to space_info_cache and datasets_info_cache MongoDB collections overloaded the database cluster. A script repeatedly que...

Anyscale users experienced GPU starvation when CPU-heavy Ray actors consumed all available CPU slots on GPU nodes, preventing GPU-dependent actors from ever being scheduled. A SpectrogramExtractor wit...

Lambda's us-east-3 region, serving exclusively GH200 instances, experienced intermittent network connectivity caused by a host component kernel panic. Nodes would crash and become unreachable until po...

Ray Data auto-scaling fails when operator resource requirements exceed available cluster resources, throwing ActorUnschedulableError. The specific case documented by Anyscale involved a MapBatches ope...

An AWS us-east-1 outage degraded RunPod's control plane, making the UI, API, and serverless coordination unavailable for GPU users. However, running pod workloads remained fully operational since GPU ...

During HuggingFace's migration from Git LFS to Xet storage backend, migration worker pods filled their ephemeral disks with orphaned temporary shard files. The shard files were first written to /tmp a...

In large-scale distributed training, NCCL errors are almost always the surface symptom rather than the root cause. The actual failure originates at a lower infrastructure layer: GPU hardware fault, st...

RAID storage failures cause data loss and training interruption when storage media degrades....

SLURM cgroup limits restrict resources (CPU, memory, GPU) for jobs, causing OOM kills or throttling when exceeded....

Kubernetes GPU pods can stay in Pending state when GPU resources are not available or configured incorrectly....

DCGM Exporter errors prevent GPU metrics from being collected for monitoring, alerting, and observability....

nvidia-smi is missing or broken when NVIDIA driver is not properly installed, blocking GPU access entirely....

GPU temperature throttling reduces clock speed and performance when GPUs overheat, sometimes causing training failures....

Cluster shared storage (NFS, Lustre, GPFS) can be a major bottleneck for distributed training data loading....

MIG (Multi-Instance GPU) and MPS (Multi-Process Service) both partition GPU resources, causing conflicts when used together....

GPU TDP and power limit configuration affects performance, thermals, and energy efficiency of training workloads....

SLURM time limit causes training jobs to be killed when they exceed their requested walltime, losing progress....

Container time drift occurs when container time differs from host time, causing TLS failures and timestamp issues....

DNS resolution failures prevent distributed training from finding nodes, downloading from HF Hub, or connecting to services....

Ray task timeouts occur when tasks take longer than the configured timeout....

Ray Dataset OOM occurs when the dataset pipeline uses more memory than available in Ray object store or worker memory....

NFS stalls freeze training when the storage server becomes unresponsive....

Cloud storage throttling slows or fails training when too many requests hit the storage service....

SLURM node OOM kills training when the requested memory exceeds the SLURM memory limit....

Cloud GPU quota limits prevent spinning up new training instances when the account has exhausted its allocated GPU quota....

GPU scheduling delays cause jobs to wait in queue for GPU resources to become available....

A physical link on the parameter (backend) network plane goes down, isolating a node's RDMA path and stalling distributed collectives — one of the most common hardware faults in production traces....

A network interface degrades — negotiating a lower link speed, disappearing, or reporting GID errors — silently throttling RDMA collective throughput across the job....

A node power supply fails or loses redundancy, risking a hard node-down event mid-training; sensor telemetry gives early warning before the node drops....

A cooling fan hits a critical speed/fault or loses redundancy, leading to rising GPU temperatures and thermal throttling if not addressed....

Checkpoint writes stall because the NFS RPC layer saturates its slot table, producing a 'bandwidth paradox' where the network is barely utilized yet I/O is the bottleneck....

Docker containers default to 64MB of /dev/shm, which is far too small for NCCL shared memory inter-process communication. Training crashes with 'posix_fallocate failed: No space left on device'. Denpe...

Full disks crash training by preventing checkpoint writes....

Kubernetes OOM kills report the process name (e.g., 'python' or 'java') rather than the pod name, making it highly difficult to trace which specific distributed training job caused a cluster-wide memo...

When mmfsd degrades — token-manager overload, quorum loss, or internal deadlock — mounts stall cluster-wide and training processes block in D-state on I/O. Jobs neither progress nor die, mimicking sil...

Azure Blob returns 503 "server busy" and operation timeouts when per-account/partition limits are exceeded by dataset streaming or parallel checkpoint uploads. SDK surfaces vary (ServiceResponseError,...

Sequence length imbalance across micro-batches within the same global batch causes certain pipeline stages to compute significantly longer than others. ByteDance's production trace analysis found that...

Silent data corruption arising from latent GPU hardware defects bypasses ECC and other hardware detection mechanisms, causing incorrect computation results without any error signal. In LLM training, t...

NVIDIA Inference Xfer Library workloads on HGX H200 and HGX B200 instances triggered host OOM kills because UCX registered GPU memory as firmware pages across all 8 NICs simultaneously, consuming ~34 ...

Meta FAIR's analysis of 11 months of operational data across two Research SuperClusters confirmed that Mean Time to Failure for training jobs scales inversely with GPU count, dropping from 47.7 days a...

ByteDance's production trace analysis found that Python's automatic garbage collection causes periodic, transient straggler behavior in distributed LLM training. Full GC passes on long-lived objects (...

During Llama 3 training on 16,384 H100 GPUs, Meta observed that mid-day temperature fluctuations of 5-10 degrees Celsius caused 1-2% throughput variation across the training cluster. The effect was at...

In BF16 distributed training, a single-bit flip in the exponent field of a gradient tensor can propagate silently through NCCL all-reduce without detection, causing one rank's gradient update to diver...

During the 4-month training of the 176B-parameter BLOOM model on 384 NVIDIA A100 GPUs, the team experienced 1-2 GPU hardware failures per week on average. Each failure required a 30-60 minute node rep...

ByteDance's production straggler analysis identified PyTorch's CUDA caching allocator memory fragmentation as a previously unreported cause of training stragglers. Over long-running jobs, the allocato...

CoreWeave's six-week benchmark training a 30B-parameter Llama-3-style model on 1,024 H100 GPUs demonstrated that purpose-built infrastructure architecture can improve Mean Time to Failure from the ind...

MosaicML's training platform automatically detected hardware failures and resumed training without human intervention using its Node Doctor and Watchdog services. During Stable Diffusion training on 1...

Training restarts after failure can get stuck if cleanup didn't complete properly, blocking new training jobs....

Training checkpoint corruption on write happens when storage failures, power loss, or process kills interrupt the write operation....

NFS mount failures during training cause silent data corruption, hangs, or training failures when accessing shared storage....

Spot instance preemption can kill training jobs with little notice, causing data loss and incomplete training runs....

Missing graceful shutdown handlers cause training jobs to be killed without saving final checkpoint or metrics....

Missing disaster recovery planning means training progress can be lost due to hardware failure, corruption, or natural disaster....

Network partitions split distributed training into isolated groups, causing hangs, deadlocks, or wrong gradients....

HDF5 corruption occurs when files are not properly closed, written from multiple processes, or have incompatible versions....

OOM kills during training are abrupt and lose progress, often happening mid-step when peak memory is reached....

Training appears stuck with no progress, often caused by deadlocks, infinite loops, or network hangs in distributed training....

Silent data corruption during training causes wrong results without error messages, often from hardware issues....

DataLoader failures cause training to crash or hang when data loading is misconfigured or data is corrupted....

Zombie processes occur when child processes aren't properly reaped, exhausting process table and file descriptors....

Weights & Biases or TensorBoard failures cause loss of experiment tracking, making debugging and comparison hard....

Partial checkpoint save happens when training crashes or is killed before all model state is saved....

Training resume failures occur when checkpoints can't be loaded properly to continue from a previous run....

Missing signal handlers prevent graceful shutdown of training, causing checkpoint loss and resource leaks....

Cgroup memory limits cause OOM kills when training exceeds the container or pod's memory limit....

save_16bit_model() under ZeRO-3 asserts because parameters are still in flight when the save interval is not a multiple of the gradient-accumulation steps....

ZeRO-3 backward fails because parameters from a previous forward path remain INFLIGHT, common with dynamic forward graphs such as RLHF, NAS, or conditional branching....

All ranks race to write into the same consolidation directory during ZeRO-3 NVMe-offload checkpoint save, so only rank 0 succeeds and the rest hit FileExistsError....

After ZeRO-3 training, accessing a partitioned parameter directly (e.g. an embedding weight) during inference sees a flattened 1-D tensor and raises a shape error....

One rank runs slower than the rest, so every collective waits on it. There is no crash and no error — just falling throughput and idle GPUs. Stragglers are the dominant cause of silent efficiency loss...

GPT-NeoX/Pythia models produce degraded or garbled text under DeepSpeed Inference with kernel injection, while the same model generates correctly without DeepSpeed. The injected fused kernels for the ...

Silent hangs stall training without error messages. Denpex heartbeat detection identifies stuck ranks....

Zombie processes hold GPU memory after the main job exits. Denpex auto-detects and kills them....

Preemption kills cloud spot instances mid-training, losing progress if checkpoints aren't saved frequently enough....

Postmortem archives, log-replay pipelines, and documentation snippets containing old Xid/ECC lines get concatenated into live streams and pasted into diagnostic tools, producing confident "replace the...

NCCL broadcast operations can hang when one rank fails to participate in the collective....

DDP training hangs at the end of an epoch when one rank exhausts its data partition before others....

One DDP rank becomes unresponsive and stalls all ranks waiting for gradient sync....

FSDP flat parameter errors occur when the flattened parameter management fails during sharding or unsharding operations....

FSDP mixed precision errors arise when parameter precision settings conflict between FSDP wrapping and autocast....

Tensor parallel errors occur when model layer splitting across GPUs fails due to dimension mismatches or communication issues....

DeepSpeed initialization fails when configuration is invalid or incompatible with the model....

PyTorch distributed errors prevent DDP and FSDP from functioning correctly....

NCCL watchdog settings control how long collectives can stall before timing out. Wrong settings cause premature or delayed failure detection....

When one rank stalls, NCCL's synchronous collectives freeze every rank, masking which GPU actually failed — localizing the culprit needs divergence signals, not the NCCL stack trace....

The configured world size does not match the number of ranks that actually connect, so NCCL initialization hangs or errors during the communication-setup stage....

DeepSpeed ZeRO-3 with Mixture-of-Experts (MoE) models hangs during training because MoE blocks need to be marked as leaf modules for ZeRO-3 parameter gathering. Without this marking, ZeRO-3 attempts t...

DeepSpeed ZeRO-3 crashes with 'dict object has no attribute _in_forward' when used with PyTorch 2.5+, which changed the internal _parameters attribute from a dict to a different type. Denpex detects t...

DeepSpeed ZeRO-3 crashes with UnboundLocalError in partition_parameters.py when the model has very small parameters (fewer elements than the number of GPUs). The partition math divides a small paramet...

After unwrapping an FSDP2 (fully_shard) model the parameters are still DTensors, so save_pretrained or a plain state_dict produces invalid storage or sharded tensors. You must gather a full, unsharded...

Combining a bitsandbytes 8-bit optimizer with DeepSpeed ZeRO offload crashes with 'Error invalid configuration argument at line 216 in file .../bitsandbytes/csrc/ops.cu'. The two optimizer-state manag...

FSDP combined with QLoRA fails to build its flat parameter: 'Must flatten tensors with uniform dtype but got torch.float32 and torch.bfloat16'. QLoRA keeps some parameters in float32 while the base is...

After upgrading to DeepSpeed >0.14.0 (e.g. 0.14.2), ZeRO-3 / optimizer-offload training crashes with 'Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!'. A...

Model weights are not written by accelerator.save_state when the model was set up with accelerator.prepare_model(model) instead of accelerator.prepare(...). prepare_model only wraps the model for dist...

DeepSpeed ZeRO-3 errors during parameter allgather when the model holds parameters in more than one dtype (e.g. bf16 base with some fp32 modules). Older ZeRO-3 assumed a single dtype for coalesced all...

DeepSpeed pipeline-parallel training (PipelineModule) hangs mid-run when microbatch input shapes vary. The PP engine caches the first microbatch's tensor shapes for point-to-point send/recv buffers, s...

DDP setup errors prevent distributed training from initializing correctly....

DDP port conflicts occur when the chosen master port is already in use by another process....

DDP NaN detection on a single rank often indicates inconsistent data or hardware between ranks....

DDP OOM during allreduce occurs when NCCL communication buffers exceed available GPU memory....

FSDP state_dict missing keys errors prevent checkpoint saving or loading in sharded models. Denpex identifies the source of key mismatches....

FSDP all-gather timeouts stall training when sharded parameters cannot be collected from distributed ranks....

DeepSpeed OOM errors crash ZeRO-optimized training when memory optimization settings don't match model architecture....

DeepSpeed ZeRO hangs stall training when gradient synchronization stalls or parameter offload produces deadlocks....

DeepSpeed initialization fails when configuration is invalid or incompatible with the model....

Megatron-LM initialization errors prevent large model training from starting due to configuration mismatches....

Torchrun's approach to spawning jobs conflicts with SLURM's process management, resulting in hung NCCL initialization, rank mismapping, and job allocation failures....

A single GPU drops in performance without throwing an error, causing all other GPUs in the collective to wait. This silently degrades cluster-wide throughput....

NCCL timeout occurs while gathering the state dictionary or broadcasting during a checkpoint save or restore operation....

ChildFailedError is torchelastic's envelope around a dead worker. The exitcode/signal in its table is the actual signal: -9 host OOM-kill, -11 segfault, -6 abort (often CUDA), 1 python exception whose...

GPU memory clock throttling reduces memory bandwidth and training performance when memory is under-utilized....

NVLink errors cause intra-node GPU communication failures and degraded multi-GPU performance....

GPU thermal throttling reduces clock speeds to prevent overheating, causing training to be slower than expected....

GPU power cap limits GPU power consumption, reducing performance for power-constrained deployments....

CUDA driver crash terminates all GPU processes on the node, losing unsaved training progress....

GPU overheating causes thermal throttling, performance degradation, and potential hardware damage....

GPU not detected errors prevent training from starting when the system cannot see the GPU....

Uncorrectable ECC errors indicate GPU memory hardware faults that corrupt training data and require GPU replacement....

CUDA illegal instruction errors crash training when the GPU encounters unsupported instructions, often from binary mismatches....

GPU clock throttle reduces performance when power or thermal limits are reached....

GPU memory bus errors corrupt data transfer between GPU cores and memory, causing silent data corruption or crashes....

GPU MIG mode splits a GPU into multiple instances, which can cause issues with PyTorch and NCCL if not configured correctly....

High GPU ECC error rates indicate degrading memory hardware. Correctable errors accumulating signal impending failure....

GPU memory bandwidth limits reduce training performance when compute exceeds memory access speed....

NVLS collective failures on Hopper/NVLINK4 nodes cascade from Fabric Manager restarts, producing Xid 31 MMU page faults and illegal memory accesses across the node....

P2P fabric-handle import between containers on a multi-node NVLink (MNNVL) GPU fails with CUDA 'invalid device ordinal' due to a CUDA driver bug, not NCCL....

NVLink bandwidth degradation can be a single failing link or a whole-node NVSwitch fault — comparing per-GPU NVLink bandwidth across all 8 GPUs tells them apart....

The GPU System Processor (GSP) stops responding to RPCs, logged as Xid 119 — the GPU becomes unresponsive and usually needs a reset....

A GPU drops off the PCIe bus (Xid 79) and disappears from the system — a critical hardware/connectivity fault that ends training on that node until the GPU is recovered....

The GPU remaps a failing memory row after ECC errors — a recovery mechanism, but a rising remap count signals degrading memory that may need RMA....

Xid 48 is NVIDIA's code for uncorrectable GPU memory errors. It crashes training and corrupts model weights. Denpex correlates Xid 48 events with NCCL failures to identify the originating GPU and trig...

Xid 79 means the GPU lost PCIe connectivity entirely. Training dies immediately. Denpex identifies which GPU, node, and the chain of events that led to the GPU fall. Root caused by bad PCIe, power eve...

CUDA device assertion failures indicate GPU hardware problems that crash training with ambiguous error messages. Denpex maps the device assertion to the specific GPU and operation, distinguishing hard...

KV cache corruption or NaNs in key_value tensors occurring during decoding on hardware topologies that do not fully support PCIe P2P memory sharing....

The display driver crashes and recovers (or fails to recover) due to a long-running CUDA kernel triggering the OS watchdog....

On Intel data-center GPUs, engine hangs surface as i915 "GPU HANG"/engine-reset dmesg lines and GuC (scheduling microcontroller) load or crash messages. Compute contexts die on reset; repeated hangs i...

Checkpoints saved with older PyTorch versions may not load with newer versions due to format changes....

Torn writes produce incomplete checkpoint files when training is interrupted mid-save....

PII in training data can leak through model outputs, causing privacy and compliance issues....

Dataset license issues prevent commercial use or distribution of models trained on the data....

Checkpoint version incompatibility prevents loading checkpoints saved with different PyTorch versions....

Optimizer state doesn't match model parameters when loading checkpoints across different architectures or configs....

Checkpoints saved with newer PyTorch versions may not load with older versions, causing compatibility issues....

Model state dict corruption produces degraded model quality after loading....

On world_size=1 with non-aligned tensor sizes, NCCL's reduce-scatter/AVG left the last elements unprocessed (zero) due to a floor-division bug — silent corruption with no error....

A faulty GPU corrupts computation without ever producing a NaN — loss settles slightly above baseline and parameters drift, making it the most deceptive SDC mode....

A permanent hardware fault causes monotonically increasing divergence of parameters from a baseline once injection begins, even while loss curves look identical to a healthy run....

DeepSpeed's DecoupledCheckpointEngine can hang indefinitely during checkpoint saving, blocking training progress without producing an error. The hang occurs when the async checkpoint writer thread dea...

DeepSpeed's FastFileWriter leaks file descriptors during checkpoint saves, eventually causing 'No space left on device' (ENOSPC) errors even when disk space is available. The leaked file descriptors e...

DeepSpeed universal checkpoint loading sorts rank directories lexicographically instead of numerically, causing rank_10 to be loaded before rank_2. This silently corrupts model weights because each ra...

DeepSpeed training crashes with SIGBUS (exit code -7) after loading a checkpoint, with no Python traceback. This typically indicates a memory-mapped file access failure — the checkpoint file was trunc...

Corrupted checkpoints silently poison resumed training with bad weights. Denpex validates checkpoint integrity before resume....

Partial checkpoint saves write incomplete data when training is interrupted, leaving a file that appears valid but is truncated....

Safetensors loading fails when files are truncated or checksums mismatch....

Checkpoint save fails when disk writes are interrupted....

Checkpoint load fails when files are corrupted or format is mismatched....

Optimizer state doesn't match model parameters when loading checkpoints across different architectures or configs....

CPU affinity misconfiguration prevents DataLoader workers and training threads from using optimal CPU cores....

Low GPU utilization indicates training is bottlenecked by data loading, CPU work, or communication....

ZeRO-3 throughput collapses versus ZeRO-2 because the parameter prefetch behaves synchronously, so transfer time fails to overlap with compute....

Fine-tuning the Qwen3-30B-A3B MoE model with Unsloth is far slower (200-300 s/step) at 10-20% GPU utilization than the comparable dense Qwen3-32B, which runs at full utilization. Missing/unoptimized M...

One rank consistently 2-4x slower drags every synchronous collective. When the cause is a failed fan/blocked airflow on exactly one GPU, the fleet-wide symptom (slow allreduce everywhere) hides a sing...

During the 54-day pre-training of Llama 3 405B on 16,384 NVIDIA H100 GPUs, 419 unexpected interruptions occurred at an average rate of one every 3 hours. HBM3 memory failures alone accounted for 17.2%...

In a five-month trace analysis of ByteDance's LLM training cluster, uneven pipeline stage partitioning was identified as the most prevalent cause of training stragglers, affecting 39.3% of jobs. The l...

During AOTAutograd phase of torch.compile, complex dynamic control flow or deeply nested autograd graphs can cause the Inductor compiler to stall indefinitely. This often manifests as a single node ha...

A NCCL collective operation failed to complete within the watchdog timeout period because one or more ranks stopped participating. This is the most common distributed training failure and is almost al...

A single rank failure cascades into mass NCCL timeouts across most or all ranks as they wait at a collective barrier. The cascade signature is one rank failing first, followed by a wave of timeouts on...

NCCL cannot find a valid path for the network pattern in the discovered PCIe topology. This is typically caused by NIC fusion (NIC split) remapping NIC device IDs during partial-node InfiniBand alloca...

NVIDIA's CUTLASS kernels for SM120 (and some Hopper architectures) suffer from a silent regression that halves throughput on Mixture of Experts (MoE) inference, commonly seen when utilizing certain Py...

Improper NCCL P2P (Peer-to-Peer) configuration and topology routing lead to PCIe bandwidth bottlenecks, silently degrading multi-GPU training performance....

New InfiniBand adapters (like ConnectX-7) require specific OFED versions. Mismatches with the host kernel or existing driver stack break the network fabric, making nodes unreachable or failing RDMA....

GPU memory fragmentation causes CUDA OOM even when total free memory exceeds the allocation request. PyTorch caching allocator cannot find a single contiguous block large enough, leading to a false OO...

NVLink SHARP (NVLS) multicast setup failed because the NVSwitch multicast slot table is exhausted or a CUDA invalid argument bug prevents memory registration. This prevents hardware-accelerated all-re...

GPU HBM memory experienced an uncorrectable ECC error (double-bit error) that the hardware cannot fix. This indicates physical HBM memory degradation requiring GPU drain and potential RMA....