ACS Disabled. GPU Direct RDMA Failure (Vendor Err 81)

When Access Control Services (ACS) is disabled in the BIOS, GPU Direct RDMA (GDR) fails with RDMA completion error 4 and vendor error 81. NCCL performance degrades severely or training crashes. Denpex detects the GDR failure from NCCL logs and recommends the BIOS or NCCL configuration fix.

ACS Disabled. GPU Direct RDMA Failure (Vendor Err 81) is a Communication failure seen during ML training runs. When Access Control Services (ACS) is disabled in the BIOS, GPU Direct RDMA (GDR) fails with RDMA completion error 4 and vendor error 81. NCCL performance degrades severely or training crashes. Denpex detects the GDR failure from NCCL logs and recommends the BIOS or NCCL configuration fix. Common tags: Acs, Gdr, Gpu Direct Rdma, Rdma.

ACS (Access Control Services) is disabled in the server BIOS, which prevents peer-to-peer DMA between the GPU and NIC across the PCIe fabric. Without ACS, the IOMMU cannot validate DMA transactions between the GPU and NIC, so the RDMA completion fails with vendor error 81. ACS is sometimes disabled for PCIe passthrough (e.g., for VM device assignment), which has the side effect of breaking GPU Direct RDMA. NCCL attempts to use GDR by default on supported hardware, and the failure is not gracefully handled on all NCCL versions. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• NCCL RDMA operations fail with completion error 4 and vendor error 81
• Multi-node training with GDR enabled crashes or performs very poorly
• The same training works with GPU Direct RDMA disabled

• Bare-metal GPU servers with ACS disabled in BIOS
• Servers where ACS was disabled for PCIe passthrough or device assignment
• Multi-node training over InfiniBand or RoCE with GPU Direct RDMA
• HPE, Dell, and Supermicro servers where ACS default varies by BIOS version

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: NCCL WARN Got completion with error 4, vendor err 81
• ✓Verified signal present: NCCL_DEBUG=INFO shows GDR registration failures
• ✓Verified signal present: Performance is significantly worse than expected on RoCE/IB fabrics
• ✓Verified signal present: ibv_devinfo shows the NIC supports GDR but NCCL cannot use it
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

NCCL is where a distributed job reports failure, which is not the same as where it failed. The hub lists every common NCCL error next to what it actually indicates, and the environment variables that tell them apart.

• ACS (Access Control Services) is disabled in the server BIOS, which prevents peer-to-peer DMA between the GPU and NIC across the PCIe fabric
• Without ACS, the IOMMU cannot validate DMA transactions between the GPU and NIC, so the RDMA completion fails with vendor error 81
• ACS is sometimes disabled for PCIe passthrough (e.g., for VM device assignment), which has the side effect of breaking GPU Direct RDMA
• NCCL attempts to use GDR by default on supported hardware, and the failure is not gracefully handled on all NCCL versions