Collective cascades
Separate ranks that timed out while waiting from the rank or node that stopped participating first.
For foundation-model training teams
Frontier-model training reliability starts with the first failed rank
At large rank counts, one initiating failure becomes hundreds or thousands of secondary NCCL timeouts, launcher exceptions and cancelled steps. Debugging the loudest rank wastes the allocation because that rank was often waiting for the process that failed silently.
Denpex orders evidence across ranks, preserves the first distinct error, connects it to GPU and scheduler state, and tells the operator what must be true before resuming from a checkpoint. The goal is a verified recovery path, not a plausible explanation after the run is already gone.
Who this is for
- Training infrastructure engineers
- Distributed systems engineers
- Research infrastructure leads
- Foundation-model researchers
- GPU cluster engineers
- Directors of research infrastructure
The failure surface
These incidents look adjacent in the final alert, but they require different owners and different proof before recovery.
Memory across tiers
Distinguish HBM pressure, host OOM, pinned-memory limits, ZeRO offload and checkpoint materialization peaks.
Checkpoint safety
Verify the newest usable checkpoint instead of assuming the last directory written is complete and resumable.
Fail-slow and hardware evidence
Attach straggler, Xid, ECC, NVLink and node recurrence signals to the training incident.
The operating loop
- 1
Capture every rank
Retain the earliest distinct error and enough preceding context before launchers collapse the incident.
- 2
Identify patient zero
Order application, CUDA, NCCL, host and scheduler events with rank and node identity.
- 3
Protect recoverable state
Check checkpoint existence, completeness and expected contents before selecting a resume point.
- 4
Resume through a control
Canary the corrected condition at production shape before returning the full allocation.
What Denpex contributes
- NCCL cascade analysis and initiating-rank evidence
- DeepSpeed, FSDP, CUDA, GPU hardware and scheduler failure coverage
- Checkpoint evidence collection when the job exposes its checkpoint directory
- Synthetic cascade exercises at high rank counts with separate disclosure from real-fabric validation
Measure this in an evaluation
Use your incidents and timestamps. These are measurement definitions, not Denpex customer-outcome claims.
| Metric | Measurement |
|---|---|
| Time to first failed rank | Minutes from job failure to the earliest causal rank or node, checked against the final postmortem. |
| Checkpoint rewind avoided | Training steps preserved by selecting the newest checkpoint that passes integrity and resume checks. |
| GPU hours recovered | Failed allocation time avoided after verified diagnosis and recovery, derived from actual job timestamps. |
| Cascade compression | Secondary rank errors grouped under one initiating incident instead of creating duplicate investigations. |
Start with the technical evidence
NCCL errors
Decode collective, bootstrap, socket, InfiniBand and duplicate-GPU failures.
DeepSpeed errors
Trace ZeRO, offload, mixed-dtype, pipeline and checkpoint failures.
GPU cluster preflight
Check topology and production-shaped fabric performance before an expensive run.
Distributed job collection
Wrap Slurm, Ray, Kubernetes and bare-metal training commands.
Frequently asked questions
Why is the rank that reports an NCCL timeout often not the failed rank?
The reporting rank was alive and waiting long enough for its watchdog to fire. The initiating rank may have crashed, hung or been killed and therefore emitted no timeout. The earliest distinct application, CUDA, host or scheduler event across all ranks carries more causal value.
Can Denpex prove a checkpoint is safe to resume?
When the checkpoint directory is provided, the agent can inspect the newest candidates and report verification evidence. If resumability cannot be evaluated, the outcome remains inconclusive rather than guessing that the latest timestamp is safe.
Does Denpex diagnose fail-slow jobs as well as crashes?
The product includes telemetry, straggler and performance evidence paths, but every production claim should be measured against the team's workloads. A pilot can compare known healthy and degraded controls and require the causal signal to persist across repeated runs.
How should a frontier lab scope a pilot?
Choose resolved incidents across NCCL, memory, checkpoint, scheduler and hardware families, add shadow monitoring to representative runs, and score causal owner, evidence completeness, operator time, checkpoint decision and verified recovery separately.
Prove it on incidents your team already resolved
Freeze the expected owner and outcome, replay the evidence, then compare operator time and decision quality with your current process.
Build the evaluation plan