Data Center exclusive · scoped pilot
Find the ranks behind a distributed training hang.
A training job stops progressing. Other ranks wait, and a collective timeout may tell you where the failure surfaced without identifying where it began. Denpex compares PyTorch Flight Recorder evidence across ranks to narrow the investigation.
For distributed training operators and GPU infrastructure teams. Pilot scope and cluster compatibility are agreed during onboarding.
From a collective timeout to an evidence-backed decision
Use this workflow to investigate PyTorch distributed training hangs and NCCL watchdog timeouts when compatible Flight Recorder traces are available. It complements your scheduler and monitoring tools.
Step 1
Collect the evidence
Publish per-rank PyTorch Flight Recorder traces with job identity, process-group membership and optional GPU or host telemetry. Signed requests bind each upload to the tenant and cluster.
Step 2
Compare progress across ranks
Compare collective sequence progress and calls within each process group. Identify ranks that have stalled, never entered the collective, or stopped reporting. Preserve multiple candidates when the evidence is ambiguous.
Step 3
Correlate the hardware signals
Compare implicated ranks with supplied Xid events, NVLink errors, fatal PCIe AER events and host OOM records. Correlation helps investigation; it does not prove every hardware root cause.
Step 4
Review a safe isolation plan
Review the evidence and a Slurm or Kubernetes isolation plan. Execution uses the configured remediation controls and scheduler checks. Missing or stale evidence prevents a drain recommendation.
Validation recorded October 28, 2026
What we tested, and what a pilot still needs to prove
Real distributed process failures
Four real PyTorch CPU/Gloo processes produced native c10d Flight Recorder dumps. Five scenarios covered a withheld rank, multiple withheld ranks, mismatched collectives, a killed process and a stopped process. Stalled observations used more than 31 seconds of real elapsed time; recovery checks verified completed collective results.
The production API handler replay passed 14 checks, including authentication, tier restrictions, hardware-event correlation and recovery. Hardware events in these replays were explicitly simulated. A separate synthetic 4,096-rank expansion checked diagnosis at larger input size.
Physical GPU validation remains open
CPU fault injection does not reproduce a physical GPU, NCCL transport or NVLink failure. We have not induced those physical faults in this validation. A customer pilot must verify supported PyTorch versions, topology, collector coverage, alert delivery and scheduler policy on the target cluster.
No promise of zero hangs, universal root-cause detection or automatic repair is made. Incomplete evidence is reported as incomplete, and isolation remains subject to the configured controls.
Questions before you connect a cluster
Does a missing rank prove that its GPU failed?
No. A missing trace can mean a dead process, collector interruption or unavailable evidence. Denpex preserves that uncertainty and uses supplied telemetry to help distinguish causes.
Will Denpex drain nodes automatically?
The triage result can propose an isolation plan. Scheduler execution is a separate controlled workflow with configured permissions, approvals and checks. A triage upload alone does not authorize a node drain.
Which plan includes this workflow?
Cluster-wide distributed training hang triage is exclusive to the Data Center plan through a scoped pilot. Free, Team, Scale and Growth do not include this workflow. Compare Data Center pricing and features.
What should we bring to a pilot?
Your PyTorch and NCCL versions, process-group layout, scheduler policy, representative Flight Recorder dumps, telemetry sources and incident examples. Agree on detection, false-positive, recovery and operational acceptance criteria before expanding the rollout.