← Back to reference

Supported Nccl Matrix

NCCLTestedNotes
2.20YesA100 SXM4
2.19YesMixed
2.18Best effortOlder RDMA

Topology Parsing

We parse the NCCL XML emitted by NCCL_DEBUG=INFO (or higher) and store it in the TopologyGraph (NodeKind::Npu/Nic/Switch).

Transports

  • NVLink/Sharp: yes.
  • IB/RoCE: yes (with mlx5 drivers ≥ 5.x).
  • TCP socket: yes (degraded; flagged in evidence).
  • UCC: best effort.

Failures We Detect

  • NCCL WARN NET lines: surfaced as FabricEvent::NcclWarning.
  • Async errors: detected within 30s via the eBPF uprobe.
  • Watchdog: caught at the 600s mark via kernel log.

Per-rank Latency Capture

NCCL_DEBUG=INFO plus denpex_sdk.nccl records per-collective wall-clock via torch.profiler keys + parsed tails.

Unsupported

  • GPUDirect RDMA on consumer SKUs that lack pinned memory.
  • Cross-cloud setups where the IB fabric isn't visible from the CP VM.