Operator task guide

How to identify the first failed rank

Collect logs from every rank, preserve their original order, and mark watchdog timeouts, ProcessGroupNCCL exceptions and launcher ChildFailedError messages as secondary. Those lines prove a survivor noticed the failure, not that it caused the incident.

Inspect the earliest distinct CUDA, application, host OOM, Xid, transport or process-exit evidence. Then validate the suspected owner with one controlled comparison, such as moving the workload, replacing the node, changing the request or reproducing one rank synchronously.

Step-by-step method

1
Freeze all rank and host evidence
Copy the complete logs before a retry truncates them. Preserve at least the context before the first error, the launcher output, scheduler state and host kernel events for every participating node.
```
rg -n -i 'error|exception|timeout|failed|killed|oom|xid' rank-*.log
dmesg -T | rg -i 'xid|oom|killed process|nvlink|pcie'
```
2
Build the rank-to-node-to-GPU map
A rank number without node and device identity cannot support an infrastructure decision. Capture global rank, local rank, hostname, CUDA-visible list and GPU UUID together.
```
printf 'rank=%s local_rank=%s host=%s visible=%s
' "$RANK" "$LOCAL_RANK" "$(hostname)" "$CUDA_VISIBLE_DEVICES"
nvidia-smi --query-gpu=index,uuid,pci.bus_id --format=csv,noheader
```
3
Mark the cascade wrappers
Collective watchdog timeouts, ncclRemoteError on peers, elastic ChildFailedError and job cancellation are normally downstream. Keep them, but do not let their volume outrank an earlier distinct failure.
4
Find the earliest distinct initiating signal
Prioritize application exceptions, CUDA faults, host OOM kills, NVIDIA Xids, lost processes and transport failures that appear before the timeout wave. If clocks are not synchronized, compare local sequence and causal dependency rather than trusting cross-host wall time alone.
5
Inspect the rank that went silent
A dead rank may emit no NCCL timeout. Check its host kernel, scheduler step state and last successful operation. The ranks that kept reporting often identify who they were waiting for but not why that peer disappeared.
```
sacct -j <jobid> --format=JobID,NodeList,State,ExitCode,Elapsed,MaxRSS
scontrol show job <jobid>
```
6
Prove ownership with one control
Repeat the smallest production-shaped case while changing one variable. A failure that follows the input belongs to workload or configuration; one that follows the node or device belongs to infrastructure until disproved.

Rank evidence priority

Signals, meanings and actions for how to identify the first failed rank.
Signal	What it means	Next action
CUDA illegal access, device assert or OOM on one rank	Potential initiating application or capacity fault	Open that rank first and reproduce synchronously or with a smaller controlled shape.
Host OOM kill or non-zero scheduler exit on one node	The rank disappeared below Python	Inspect cgroup, MaxRSS and kernel evidence before changing NCCL settings.
NVIDIA Xid at the same time on one GPU	Device, driver, power, thermal or link evidence	Preserve UUID and host evidence, then run the hardware ownership control.
Watchdog timeout on many surviving ranks	Ranks waited for work that never completed	Find the missing or earlier-failing participant. Do not choose the first timeout by itself.
ChildFailedError or launcher cancellation	Parent process discovered a failed child	Open the child log named by the launcher and move upward to its first distinct error.

Evidence checklist

Complete unmerged rank logs with original file and line order
Rank, local rank, hostname, GPU UUID and PCI bus mapping
Launcher command and resolved distributed environment
Scheduler job and step state with exit codes
Host kernel Xid and OOM evidence before reboot
One known-good control with the same production shape

Common mistakes

Choosing the loudest rank

A living rank can emit thousands of timeout lines while the process that failed first is already silent.

Sorting unsynchronized wall clocks

Cross-host clock skew can reverse apparent order. Use local sequence, process dependency and corrected timestamps when available.

Changing several variables

A retry with a new node, smaller batch, different image and longer timeout may succeed without identifying which condition mattered.

Tools and deeper references

NCCL log decoder

Classify covered initiating signatures and secondary wrappers in browser.

NCCL error reference

Decode common collective, bootstrap and transport messages.

CUDA error reference

Separate asynchronous kernel errors from runtime and hardware faults.

Collect distributed incidents

Wrap jobs and preserve rank, scheduler, checkpoint and host evidence.

Official references

Frequently asked questions

Is the first rank to report an NCCL timeout the failed rank?

No. That rank was alive long enough for its watchdog to fire. The initiating rank may have crashed, hung or been killed and produced no timeout. Use the earliest distinct application, CUDA, host or transport evidence across all ranks.

What if rank logs use different clocks?

Do not sort cross-host wall timestamps blindly. Preserve local file order, compare collective sequence and process dependencies, inspect scheduler and kernel events, and use corrected collection timestamps when the agent or logging platform provides them.

Can one rank fail because another rank corrupted data?

Yes. A transport or collective can surface corruption on a receiver, while the sender or an earlier kernel created the bad state. That is why the method preserves competing hypotheses until a controlled input, rank, node or backend comparison identifies ownership.

How much log context should I keep?

Keep the complete incident when possible. At minimum, preserve enough lines before the first recognized error to include the last successful operation, rank and node identity, configuration warnings, resource events and any earlier CUDA or application exception.

Apply the method to the complete incident

Paste the logs for an answer-first diagnosis, evidence request and verification step.

Diagnose the incident