NCCL Topology XML Missing NIC Bus ID
NCCL topology detection fails when the NIC's PCI bus ID is missing from the system topology XML. This causes NCCL to fall back to a suboptimal transport or fail to use GPU Direct RDMA. Denpex detects the missing bus ID and recommends the configuration override.
NCCL topology detection fails when the NIC's PCI bus ID is missing from the system topology XML.
What this failure is
NCCL Topology XML Missing NIC Bus ID is a Communication failure seen during ML training runs. NCCL topology detection fails when the NIC's PCI bus ID is missing from the system topology XML. This causes NCCL to fall back to a suboptimal transport or fail to use GPU Direct RDMA. Denpex detects the missing bus ID and recommends the configuration override. Common tags: Nccl, Topology, Busid, Nic.
Is this what broke your run? Paste your log.
You're reading about NCCL Topology XML Missing NIC Bus ID. Paste your own crash log or traceback below and get the real root cause for YOUR run, not this generic entry. No account, no card. Logs are masked at ingress and never saved to account history.
Want 14 days on the Scale plan?
Request an evaluation code. A verified workplace organization activates up to 50 diagnoses a day, alerts, history, and follow-up questions. No credit card or automatic subscription.
Why it happens (the mechanism)
The NIC's PCI bus ID is not populated in sysfs or the NCCL topology XML, preventing NCCL from associating the NIC with the correct NUMA node and GPU. EFA devices on AWS don't have standard PCI bus IDs that NCCL can discover. Virtual NICs in VMs may not expose PCI topology information. After a driver upgrade, the NIC's PCI path may change, invalidating the cached topology XML. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.
What you'll observe
- NCCL falls back to Socket transport despite InfiniBand NICs being available
- NCCL_WARN: Attribute busid of node nic not found in topology
- Multi-node training performance is much worse than expected on IB clusters
Common symptoms and what they mean
| Symptom | Why it happens |
|---|---|
| NCCL WARN Attribute busid of node nic not found | The NIC's PCI bus ID is not populated in sysfs or the NCCL topology XML, preventing NCCL from associating the NIC with the correct NUMA node and GPU |
| NCCL uses Socket transport instead of IB despite ibstat showing active NICs | EFA devices on AWS don't have standard PCI bus IDs that NCCL can discover |
| NCCL_TOPO_FILE environment variable has no effect or contains invalid bus IDs | Virtual NICs in VMs may not expose PCI topology information |
| Performance regression after a driver or firmware update | After a driver upgrade, the NIC's PCI path may change, invalidating the cached topology XML |
Which systems are affected
- AWS EFA instances where the NIC doesn't expose a standard PCI bus ID
- Virtual machines with virtual NICs that lack PCI topology information
- Servers with custom NIC drivers that don't populate the PCI bus ID field
- Multi-node training after a Mellanox OFED driver upgrade
How to confirm this is the problem
Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.
- ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
- ✓Verified signal present: NCCL WARN Attribute busid of node nic not found
- ✓Verified signal present: NCCL uses Socket transport instead of IB despite ibstat showing active NICs
- ✓Verified signal present: NCCL_TOPO_FILE environment variable has no effect or contains invalid bus IDs
- ✓Verified signal present: Performance regression after a driver or firmware update
- ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.
The fix and the prevention pattern
The root cause is on this page and stays free. A free account adds the exact remediation steps, saved history, and the fix on every entry in the encyclopedia.
Sign up free. Unlock the full analysisNo credit card. Daily allowance follows verified trust tier. Instant access.
Diagnose this failure in VS Code
Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.
Install the free VS Code extensionNCCL errors in context
NCCL is where a distributed job reports failure, which is not the same as where it failed. The hub lists every common NCCL error next to what it actually indicates, and the environment variables that tell them apart.
Compare every nccl error side by sideRelated failures to investigate next
Root cause
- The NIC's PCI bus ID is not populated in sysfs or the NCCL topology XML, preventing NCCL from associating the NIC with the correct NUMA node and GPU
- EFA devices on AWS don't have standard PCI bus IDs that NCCL can discover
- Virtual NICs in VMs may not expose PCI topology information
- After a driver upgrade, the NIC's PCI path may change, invalidating the cached topology XML
The fix and how to prevent it
Evaluate Denpex on your own logs
Request a Scale evaluation code. A verified workplace organization activates 14 days with up to 50 diagnoses a day. Every account keeps its current diagnosis allowance and gets a verification path. No card or automatic subscription.
Don't just read the fix, diagnose your run
The encyclopedia tells you what went wrong. Denpex tells you what went wrong in YOUR training run. With your logs, your config, and your stack.