NCCL P2P Level and Topology Routing Limits

Improper NCCL P2P (Peer-to-Peer) configuration and topology routing lead to PCIe bandwidth bottlenecks, silently degrading multi-GPU training performance.

NCCL P2P Level and Topology Routing Limits is a Hardware/Vendor failure seen during ML training runs. Improper NCCL P2P (Peer-to-Peer) configuration and topology routing lead to PCIe bandwidth bottlenecks, silently degrading multi-GPU training performance. Common tags: Nccl, P2p, Topology, Pcie.

PCIe ACS is enabled, blocking true P2P transfers between GPUs. NCCL_P2P_LEVEL is strictly set, preventing valid fallback routes. Improper container capabilities or IOMMU settings prevent direct memory mapping. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• Multi-GPU training scales poorly, often bottle-necking at PCIe bandwidth instead of NVLink
• VLLM_SKIP_P2P_CHECK allows execution but performance is severely degraded
• NCCL falls back to shared memory or slower PCIe paths instead of direct NVLink

• Multi-GPU setups (4x, 8x) without full-mesh NVLink
• Heterogeneous PCIe topologies or environments with ACS (Access Control Services) enabled

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: NCCL logs show 'P2P Disabled' or routing via SHM/SYS instead of PIX/NVL
• ✓Verified signal present: nvidia-smi topo -m shows incomplete NVLink connectivity
• ✓Verified signal present: High CPU usage during NCCL collectives due to PCIe staging
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

NCCL is where a distributed job reports failure, which is not the same as where it failed. The hub lists every common NCCL error next to what it actually indicates, and the environment variables that tell them apart.

• PCIe ACS is enabled, blocking true P2P transfers between GPUs
• NCCL_P2P_LEVEL is strictly set, preventing valid fallback routes
• Improper container capabilities or IOMMU settings prevent direct memory mapping