← Back to reference

Nccl Debugging Guide

What Denpex Catches

RuleTrigger
NCCL_ASYNC_ERRORAsync exception from collectives
NCCL_WATCHDOGWatchdog fired
NCCL_RING_LOCALIZEDStragglers isolated to a sub-ring
NCCL_TREE_LEAFTree leaves idle for too long
NCCL_COLLNET_DEGRADEDCollNet falling back to Ring

Setup

export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=ALL
export NCCL_ASYNC_ERROR_HANDLING=1

The agent tails stderr of the wrapped process and parses all NCCL log lines.

What We Send

  • Per-rank per-op timing (from torchtrainer.profile key paths + parsed tails).
  • eBPF uprobes on ncclCommInitRank, ncclAllReduce, etc.
  • Topology XML (parsed once).

Debug Recipes

"AllReduce hangs at step 12k"

  1. Open Denpex incident.
  2. Inspect the evidence graph. Top-1 is the rank that completed last.
  3. Cross-reference the rank with the FAB graph (it's the leaf on the right side of the tree).
  4. Run DCGM Level-3 on the warm spare; if it passes, route the job there.

"CollNet falls back to Ring"

CollNet requires SHARP. Check:

  • NCCL_COLLNET_ENABLE=1.
  • Mellanox SHARP is enabled in UFM.
  • The fabric manager reports nvswitch sharp-capable for the relevant switch.

"Watchdog fires after 600s"

Cross-reference with:

  • /proc/pressure/cpu full_avg10 for the rank's host.
  • Recent NCCL log lines for "ring N closed".
  • eBPF tcp_retransmit_skb count for the host.

See Also

  • docs/user-guides/network-failure-guide.md.
  • docs/user-guides/storage-bottleneck-guide.md (when the bottleneck is OST saturation causing shadow collectives).