Infrastructure error references

Browse 77 published references for infrastructure failures in GPU training and inference. Match the message and context from your run, then open the reference for possible causes, evidence checks and sources. A matching message alone does not confirm the cause.

• 0/32 nodes are available: Insufficient nvidia.com/gpu

  Insufficient nvidia.com/gpu resources, scheduler cannot fit pod on any node. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent k8s failures.

• Active LoRA adapter rank exceeds engine pre-allocated max_lora_rank

  A LoRA adapter was requested whose rank is larger than the engine reserved at startup. vLLM preallocates LoRA slots sized by max_lora_rank, so a higher-rank adapter cannot be loaded into the running engine. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent fp8-serving failures.

• An out-of-tree GPU or storage kernel module stops building after a kernel upgrade

  DKMS rebuilds out-of-tree modules against each new kernel, but it can only recompile the source it was given. When the kernel removes or renames an internal function the module calls, the rebuild fails with an implicit declaration error and the node comes back from its upgrade without GPUDirect, without its parallel filesystem client, or in some cases without a working driver at all.

• Atomic Save I/O Error on Parallel FS

  PyTorch Lightning uses atomic saves by default, saving to a temporary directory (`/tmp`) and then renaming. If `/tmp` is on a different filesystem than the destination (Lustre), the rename triggers a cross-device move or heavy metadata load, causing Lustre to throttle or error out.

• AWS us-east-1 Outage Degrading GPU Cloud Control Plane with Multi-Region Failover Response

  An AWS us-east-1 outage degraded RunPod's control plane, making the UI, API, and serverless coordination unavailable for GPU users. However, running pod workloads remained fully operational since GPU compute resources are independent of the control plane. RunPod responded by deploying core services across multiple AWS regions within 72 hours and planning a partitioned multi-region deployment on their own provider network.

• Azure Blob Storage Throttling and Timeouts in Training I/O Paths

  Azure Blob returns 503 "server busy" and operation timeouts when per-account/partition limits are exceeded by dataset streaming or parallel checkpoint uploads. SDK surfaces vary (ServiceResponseError, HttpResponseError), hiding the common throttling cause.

• Checkpoint I/O Stall (NFS RPC Saturation)

  Checkpoint writes stall because the NFS RPC layer saturates its slot table, producing a 'bandwidth paradox' where the network is barely utilized yet I/O is the bottleneck.

• Cloud GPU Quota Exceeded

  Cloud GPU quota limits prevent spinning up new training instances when the account has exhausted its allocated GPU quota.

• Cloud Storage Throttling

  Cloud storage throttling slows or fails training when too many requests hit the storage service.

• Cluster Shared Storage Slow

  Cluster shared storage (NFS, Lustre, GPFS) can be a major bottleneck for distributed training data loading.

• Container Time Drift

  Container time drift occurs when container time differs from host time, causing TLS failures and timestamp issues.

• CUDA error: an illegal memory access was encountered paged_attention

  A vLLM paged-attention kernel accessed an invalid GPU address. The immediate owner can be corrupted block-table metadata, an unsupported shape or dtype, or a binary compatibility defect in the selected attention backend. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent inference failures.

• DeepGEMM rejects FP8 block-scaled weights with Unknown SF transformation on an unsupported architecture

  A block-scaled FP8 checkpoint fails while its weights are being prepared, before the server ever accepts a request. The runtime selected a specialised matrix-multiply library for the detected architecture, and that library does not implement the scale-factor layout this class of device requires, so it refuses the transformation outright.

• Disk Full During Training

  Full disks crash training by preventing checkpoint writes.

• DNS Resolution Failure

  DNS resolution failures prevent distributed training from finding nodes, downloading from HF Hub, or connecting to services.

• Docker /dev/shm Exhaustion. NCCL Shared Memory Allocation Failure

  Docker containers default to 64MB of /dev/shm, which is far too small for NCCL shared memory inter-process communication. Training crashes with 'posix_fallocate failed: No space left on device'. Denpex detects the /dev/shm exhaustion and recommends the correct Docker configuration.

• Dual ISP BGP Route Withdrawal Causing Complete GPU Cloud Region Outage

  Lambda Labs' us-south-3 region experienced a complete network outage when both redundant ISPs simultaneously stopped advertising BGP routes. The primary fiber link was damaged the prior day, shifting traffic to the secondary ISP. When the secondary ISP also stopped advertising routes, all network connectivity to the region was lost, making cloud instances inaccessible for an extended period.

• Dynamic batcher queue exceeded max_queue_delay_microseconds

  Triton's dynamic batcher held requests longer than max_queue_delay_microseconds and dropped them. The server is saturated: arrival rate exceeds what the model instances can clear, so the queue never drains within the delay budget. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent inference failures.

• Ephemeral Storage Exhaustion by Model Weights

  Machine learning workloads often download massive pre-trained model weights (e.g., from Hugging Face) into the default cache directory (like ~/.cache/huggingface) located on the container's root filesystem. This consumes the node's underlying root partition (ephemeral storage). Without explicit limits, a single pod can fill the disk, triggering a kubelet eviction.

• failed to load model TensorRT engine built with CUDA cannot run on host CUDA runtime

  A TensorRT engine (a .plan/.engine file) was built against a different CUDA/TensorRT version than the one the server is running. TensorRT engines are not portable across versions, they are compiled artifacts tied to the exact runtime that produced them. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent inference failures.

• Fan Speed Critical / Cooling Redundancy Lost

  A cooling fan hits a critical speed/fault or loses redundancy, leading to rising GPU temperatures and thermal throttling if not addressed.

• flashinfer CUDA kernel launch failure cudaErrorIllegalAddress paged_prefill

  A FlashInfer paged-prefill kernel accessed an invalid GPU address. Common owners are malformed page metadata, a shape or dtype unsupported by the selected kernel, or a binary built for an incompatible CUDA stack. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent fp8-serving failures.

• FP8 GEMM torch._scaled_mm only supported on CUDA capability >= 8.9

  FP8 matrix multiply was requested on a GPU without hardware FP8 support. torch._scaled_mm requires compute capability 8.9 or newer (Ada, Hopper, Blackwell); Ampere (sm_80/sm_86) has no FP8 tensor cores and cannot emulate them at useful speed. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent fp8-serving failures.

• FSDP Checkpoint S3 Path Malformation

  FSDP checkpointing often relies on utilities like `pathlib.Path` which are designed for local filesystems. When provided an S3 URI (e.g., `s3://bucket`), `pathlib` collapses the double slashes into a single slash (`s3:/bucket`), rendering the path invalid for S3 client libraries.

• GPFS/Spectrum Scale Daemon Stalls: Token Contention, Quorum Loss, Deadlocks

  When mmfsd degrades. Token-manager overload, quorum loss, or internal deadlock. Mounts stall cluster-wide and training processes block in D-state on I/O. Jobs neither progress nor die, mimicking silent hangs.

• GPU Scheduling Delay

  GPU scheduling delays cause jobs to wait in queue for GPU resources to become available.

• GPU TDP / Power Limit

  GPU TDP and power limit configuration affects performance, thermals, and energy efficiency of training workloads.

• GPU Thermal Throttling: Check Temperature and Clock Reasons

  GPU thermal throttling: compare temperature, clock event reasons and workload timing. Distinguish thermal slowdown from a power cap or external power brake.

• Host Component Kernel Panic Causing Intermittent Network Connectivity on GH200 Instances

  Lambda's us-east-3 region, serving exclusively GH200 instances, experienced intermittent network connectivity caused by a host component kernel panic. Nodes would crash and become unreachable until power cycled. The root cause was identified as a known kernel bug with an available but not yet applied patch. Rolling power cycles of affected nodes restored service while the patch was being deployed.

• Insufficient nvidia.com/mig device is locked by defunct process

  MIG partition not accessible. MIG mode off or GPU instance misconfigured. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent mig-vgpu failures.

• Interactive Allocation PENDING indefinitely (Nodes Drained)

  The specific nodes requested (or all nodes satisfying the constraints in the partition) are in a DRAINED or DOWN state due to hardware issues, maintenance, or failed health checks, making them unallocatable.

• Job Submission Fails due to QOS Limits (MaxTRESPerUser)

  The user is exceeding the maximum Trackable Resources (TRES) like GPUs, CPUs, or total running jobs allowed for their specific Quality of Service (QOS) tier or account. The partition itself has the hardware available, but the user's specific policy restricts them.

• Kubernetes GPU Pod Pending

  Kubernetes GPU pods can stay in Pending state when GPU resources are not available or configured incorrectly.

• libfabric efa_cq_poll_ibv failed to poll cq Transport endpoint is not connected

  The EFA provider reported a disconnected endpoint while polling the completion queue. A peer process exit, security-group mismatch, EFA attachment problem, or libfabric transport fault interrupted the fabric path. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent cloud failures.

• MIG and MPS Conflict

  MIG (Multi-Instance GPU) and MPS (Multi-Process Service) both partition GPU resources, causing conflicts when used together.

• MongoDB Info Cache Collection Queries Overloading Database During HuggingFace Hub Outage

  The Hugging Face Hub suffered an extended 36-hour outage when repeated requests to space_info_cache and datasets_info_cache MongoDB collections overloaded the database cluster. A script repeatedly querying a specific Space listing URL triggered a cascade: increased MongoDB network I/O, replica lag, and ultimately complete database cluster failure. The outage required multiple waves of mitigation over two days.

• Multi-Worker Write Race Condition

  In a multi-GPU/multi-node environment (e.g., DDP), multiple processes (ranks) execute the `torch.save()` command simultaneously, attempting to write to the exact same file path. This causes a race condition where writes interleave, corrupting the file.

• NCCL Errors as Surface Symptom for Diverse Underlying Infrastructure Root Causes

  In large-scale distributed training, NCCL errors are almost always the surface symptom rather than the root cause. The actual failure originates at a lower infrastructure layer: GPU hardware fault, straggler, memory corruption, network misconfiguration, cable fault, or storage latency. CoreWeave documented that NCCL errors obscure the true failure source, leading teams to misdiagnose and waste time on communication layer fixes while the underlying hardware issue persists.

• NCCL WARN RoCE MTU mismatch detected local MTU != remote MTU mlx5

  RoCE MTU Mismatch - Packets are being dropped because the node MTU exceeds the switch MTU. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent cloud failures.

• Network Storage Volume Causing Process Hangs on H100 GPU Nodes

  RunPod discovered an issue affecting pods using volume disk or network storage in their CA-MTL-1 region. When executing commands on storage-backed files, processes would hang indefinitely even though files were created successfully. The issue primarily impacted H100 GPUs and some A40 GPUs, requiring targeted filesystem updates on affected machines.

• NFS / Network Filesystem Stall

  NFS stalls freeze training when the storage server becomes unresponsive.

• NIC Degradation (Link Speed Low / NIC Lost / GID Error)

  A network interface degrades. Negotiating a lower link speed, disappearing, or reporting GID errors. Silently throttling RDMA collective throughput across the job.

• Node state DRAIN: NVML health check timed out GPU uncorrectable ECC

  Slurm drained the node because its NVML health check timed out while the GPU also reported uncorrectable ECC evidence. The timeout is a node-health failure, not a reason to return the GPU to service after a simple job retry. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent slurm failures.

• Non-Atomic Checkpoint Save Interruption

  The training process was interrupted (e.g. by OOM, preemption, or crash) while PyTorch was executing `torch.save()`. Because the save operation writes directly to the destination file, the interruption leaves the file in a partially written, corrupted state.

• NVIDIA-Driver-DaemonSet kernel module build failed gcc version mismatch

  The GPU Operator driver daemonset could not build the NVIDIA kernel module for the running kernel. Until it succeeds the node has no driver, so every GPU pod stays Pending while the node otherwise looks healthy. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent k8s failures.

• nvidia-gridd failed to acquire license from DLS server license lease expired throttling

  The vGPU guest could not reach its Delegated License Service. NVIDIA vGPU throttles clocks severely when unlicensed, so the symptom users report is "the GPU got slow", not "a license expired". This entry explains how to confirm the cause, apply the fix, and separate it from adjacent mig-vgpu failures.

• nvidia-smi mig unable to create GPU instance requested profile is incompatible

  The requested MIG profile does not fit alongside the instances already configured. MIG placements are constrained to fixed slot positions, so a set of profiles that sums to the right memory can still be unplaceable. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent mig-vgpu failures.

• nvidia-smi Missing or Broken

  nvidia-smi is missing or broken when NVIDIA driver is not properly installed, blocking GPU access entirely.

• OOMKilled Containers Without Clear Attribution

  Kubernetes OOM kills report the process name (e.g., 'python' or 'java') rather than the pod name, making it highly difficult to trace which specific distributed training job caused a cluster-wide memory exhaustion.

• Parameter-Plane Cable Link Down

  A physical link on the parameter (backend) network plane goes down, isolating a node's RDMA path and stalling distributed collectives. One of the most common hardware faults in production traces.

• Pod evicted: Usage of ephemeral-storage exceeds limit

  The kubelet evicted the pod for exceeding its ephemeral-storage limit. On training pods this is almost always checkpoint or dataset cache written to the container filesystem instead of to a mounted volume. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent k8s failures.

• Power Supply Failure / Redundancy Lost

  A node power supply fails or loses redundancy, risking a hard node-down event mid-training; sensor telemetry gives early warning before the node drops.

• Preempted Job Causes Workflow Manager Hard Failure

  SLURM reclaimed the node for a higher-priority job and sent a SIGTERM. The running application did not catch the signal to gracefully exit or checkpoint within the 2-minute grace period. The workflow manager was not configured to interpret the 'PREEMPTED' (PR) state as a resumable event.

• RAID Storage Failure

  RAID storage failures cause data loss and training interruption when storage media degrades.

• Ray Actor Resource Reservation Causing GPU Starvation from Occupied CPU Slots

  Anyscale users experienced GPU starvation when CPU-heavy Ray actors consumed all available CPU slots on GPU nodes, preventing GPU-dependent actors from ever being scheduled. A SpectrogramExtractor with concurrency=8 requested 1 CPU per actor, and Ray distributed 4 of these to the GPU node's 4 CPU slots. The GPU actor was permanently blocked waiting for a CPU slot on the only node with a GPU, stalling the pipeline for nearly an hour.

• Ray Data Auto-Scaling Failure from Resource Shape Mismatch Between Requested and Available

  Ray Data auto-scaling fails when operator resource requirements exceed available cluster resources, throwing ActorUnschedulableError. The specific case documented by Anyscale involved a MapBatches operator with concurrency=2, each task requesting 1 CPU and 1 GPU, totaling 2 CPUs and 2 GPUs, but the cluster did not have enough resources to satisfy the combined GPU+CPU placement constraint.

• Ray Dataset Out of Memory

  Ray Dataset OOM occurs when the dataset pipeline uses more memory than available in Ray object store or worker memory.

• Ray Task Timeout

  Ray task timeouts occur when tasks take longer than the configured timeout.

• ray.exceptions.ClusterUnavailableError Raylet process crashed OOM killed

  The operating system or container memory limit killed the Raylet, which made the cluster unavailable. This is node memory exhaustion, not an object-store eviction or a recoverable worker-only failure. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent ray failures.

• ray.exceptions.RuntimeEnvSetupError Runtime environment setup timed out

  Ray could not prepare the job runtime environment before its setup deadline. Package download, dependency installation, or working-directory upload is stalled or too large on at least one node. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent ray failures.

• Remediation Storm Prevention via Circuit Breaker Pattern in AutoClusters

  Crusoe's AutoClusters system implements a circuit breaker pattern that pauses automatic node replacement when failure patterns look anomalous. If too many nodes in a cluster fail within a short window, rather than continuing automated remediation (which could exhaust the spare pool or compound a shared root cause), the system stops, alerts human operators, and waits for investigation before resuming replacements.

• Requested node configuration is not available (Memory/CPU Mismatch)

  The user requested a combination of resources (like --mem-per-cpu combined with a high --ntasks) that exceeds the physical hardware capacity of any single node within the specified partition. Often caused by confusing --mem (total memory per node) with --mem-per-cpu.

• Routine UPS Maintenance Triggering Cascading Power and Cooling Failure Across GPU Cloud Region

  A routine UPS-related maintenance procedure in Lambda's us-south-1 region unexpectedly affected power delivery to cooling systems, causing networking equipment and other hardware to either power off or throttle. The resulting 16-hour outage affected all instances in the region and required individual hypervisor reboots during recovery.

• sglang RadixCacheError Failed to lock memory page during dynamic prefix eviction

  SGLang failed to acquire a radix-cache page lock while dynamic prefix eviction was mutating the cache. Extreme token pressure can expose the race, but repeated failure at moderate pressure indicates a cache implementation or version defect. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent fp8-serving failures.

• SLURM Cgroup Limit

  SLURM cgroup limits restrict resources (CPU, memory, GPU) for jobs, causing OOM kills or throttling when exceeded.

• SLURM Node Out of Memory

  SLURM node OOM kills training when the requested memory exceeds the SLURM memory limit.

• SLURM Requeue Fails on Preemption in Cloud Environments

  In elastic cloud clusters (like AWS Spot instances), the compute node itself may be terminated by the cloud provider before SLURM's slurmctld can successfully execute the requeue command. Alternatively, `JobRequeue=1` is not set in `slurm.conf`.

• SLURM Time Limit

  SLURM time limit causes training jobs to be killed when they exceed their requested walltime, losing progress.

• slurmstepd: error: JOB CANCELLED DUE TO OOM-KILLER

  Slurm step memory cgroup limit hit during NCCL all-gather in FSDP due to memory spike. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent slurm failures.

• slurmstepd: task/cgroup: unable to allocate requested memory for GPU step

  slurmstepd could not create the memory cgroup for the step. The requested amount exceeds what the node can back, or the cgroup hierarchy is misconfigured, the job never starts rather than being killed later. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent slurm failures.

• Socket Interface Bootstrap Isolation

  The NCCL bootstrap routine binds to a virtual ethernet interface (veth) created by Docker/Kubernetes, which lacks routability to peer nodes. TCP connection attempts silently blackhole.

• TensorRT-LLM KVCacheManager Available blocks exhausted

  TensorRT-LLM ran out of paged KV-cache blocks. The block pool is fixed at engine build/startup, so concurrent sequences × their lengths exceeded what was reserved, commonly triggered by beam search or speculative decoding, which multiply the blocks a single request consumes. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent inference failures.

• trtllm-build Out of memory during weight-only INT4 quantization

  The TensorRT-LLM engine build ran out of HOST memory, not GPU memory. Quantization holds full-precision weights in RAM while it converts them, so a 70B model can need well over 100 GB of system memory to build even though the resulting engine is small. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent inference failures.

• ValueError: No available memory for the cache blocks gpu_memory_utilization

  vLLM fails to initialize due to pre-allocating too much memory for cache blocks when CUDA graphs are enabled or `--max-num-seqs` is set too high. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent inference failures.

• vLLM AsyncEngineWorker died unexpectedly with SIGSEGV

  vLLM async engine crashed, a worker process exited unexpectedly. This entry explains how to confirm the cause, apply the fix, and separate it from adjacent inference failures.

• vLLM tensor parallelism fails in a container because shared memory is limited to 64 MB

  A container gets 64 MB of shared memory by default. PyTorch and NCCL use that space to move data between the worker processes of a tensor-parallel deployment, so a model that runs on one GPU fails to initialise across several, with an error naming shared memory rather than the model.

• Xet Storage Migration Workers Filling Disk with Orphaned Temporary Shard Files

  During HuggingFace's migration from Git LFS to Xet storage backend, migration worker pods filled their ephemeral disks with orphaned temporary shard files. The shard files were first written to /tmp and then moved to the Xet cache, but /tmp and the cache were on different mount points. The move failed silently, orphan shard files were never cleaned up, and the disk eventually filled, triggering No space left on device errors across the migration fleet.