NVLink SHARP (NVLS) multicast initialization failure

NVLS initialization failures occur while NCCL is preparing NVSwitch multicast resources for collective offload. Diagnose support, allocation, software state, and fabric health separately instead of treating every NVLS message as the same defect.

Quick answer

Capture the first NVLS or multicast error, determine whether NVLS was forced or automatically selected, and replay the same collective with NVLS disabled. Inspect Fabric Manager, NVSwitch health, and stack compatibility before deciding whether the failure is resource, fabric, or software owned.

Communication#nvls#nvlink#sharp#multicast#nvswitch#nccl

What this failure is

NVLink SHARP is an NCCL transport path that offloads supported collective work into the NVSwitch domain. An NVLS initialization failure means NCCL could not complete that setup. It does not by itself prove slot exhaustion, a broken GPU, or an NCCL regression.

Live diagnosis, no signup

Is this what broke your run? Paste your log.

You're reading about NVLink SHARP (NVLS) multicast initialization failure. Paste your own crash log or traceback below and get the real root cause for YOUR run, not this generic entry. No account, no card. Logs are masked at ingress and never saved to account history.

training_logs.txt

No log to hand? Try one:

3 free diagnoses/day

Want 14 days on the Scale plan?

Request a work-email trial for up to 50 diagnoses a day, alerts, history, and follow-up questions. No credit card or automatic subscription.

Evaluate one incident

Why it happens (the mechanism)

NVIDIA documents three NVLS selection modes: disabled, forced, and automatic. Forced mode fails initialization when required resources cannot be allocated, while automatic mode can decline the path when the platform does not support it. Other documented failures can include system-not-ready or Fabric Manager state. The first emitted signature determines which branch to investigate.

What you'll observe

NCCL communicator initialization fails before a collective can run.
Training or inference may report only a later communicator failure or timeout.
A broad NVLS error label can collapse unsupported configuration, allocation failure, software incompatibility, and unhealthy NVSwitch state into one diagnosis.

Common symptoms and what they mean

Symptom	Why it happens
NCCL logs name the NVLS subsystem or multicast group setup before the final failure.	NCCL could not create, bind, or use the multicast resources required by the selected NVLS path.
The application may report CUDA_ERROR_SYSTEM_NOT_READY during multicast binding.	The causal owner can be unsupported or forced configuration, unavailable resources, inconsistent software components, or unhealthy fabric state.
System logs may report Fabric Manager absence or an NVSwitch-related error.	A later Xid, timeout, or communicator abort is not enough to choose among those causes without the first NVLS and system log evidence.
The same collective can complete after NVLS is disabled.	NCCL could not create, bind, or use the multicast resources required by the selected NVLS path.

Which systems are affected

Supported Hopper or later GPU systems with third-generation or later NVSwitch
NCCL jobs that select NVLink SHARP
Systems whose Fabric Manager and NVSwitch state participate in multicast setup

How to confirm this is the problem

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

✓Locate the first line containing NVLS, multicast, cuMulticast, or an NVLS source file and preserve preceding context.
✓Confirm the resolved value of NCCL_NVLS_ENABLE and whether the platform is expected to support NVLS.
✓Run one unchanged minimal collective with NVLS disabled.
✓If that passes, compare Fabric Manager, NVSwitch health, and software versions against a known-good system.
✓Re-enable the intended setting and require both the minimal collective and production-sized control to pass.

Searchable error signature

search key

CUDA_ERROR_SYSTEM_NOT_READY
NV_ERR_FABRIC_MANAGER_NOT_PRESENT

Use this text as a lookup key in logs and upstream issue trackers. It is not presented as a captured customer log. Confirm the cause from your own preceding events, versions, configuration and the cited references.

The fix and the prevention pattern

The root cause is on this page and stays free. A free account adds the exact remediation steps, keeps your diagnoses instead of discarding them, and unlocks the fix on every entry in the encyclopedia.

No credit card · 3 free diagnoses · Instant access

Why the recommended fix works

The sequence separates transport selection from platform health and software compatibility. It avoids a permanent performance workaround when the real correction is Fabric Manager recovery or a supported stack combination, and it avoids blaming hardware when automatic selection simply found no supported NVLS path.

Code examples

snippet

export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,GRAPH,NVLS
export NCCL_DEBUG_FILE=/tmp/nccl_%h_%p.log

# One diagnostic replay only
NCCL_NVLS_ENABLE=0 <same-nccl-test-command>

# Platform evidence
systemctl status nvidia-fabricmanager
nvidia-smi -q

Adapt the snippet to your framework. The same pattern holds for PyTorch Lightning, Hugging Face Trainer, DeepSpeed, Megatron-LM, and vLLM training wrappers. Where the wrapper exposes a config flag (for examplelr_scheduler_type in Trainer), prefer the flag over the imperative API to keep the schedule declarative and reproducible.

Best practices by model family

Model / Stack	Recommendation	Notes
Selection	Prefer automatic NVLS selection	Forced enablement is useful for qualification but is designed to fail when resources cannot be allocated.
Isolation	Replay with NVLS disabled	This isolates the path while preserving the workload, ranks, and message size.
Return to service	Verify correctness and throughput	A successful initialization alone does not prove production-sized collectives are healthy.

With the fix vs without the fix

Dimension	With the fix	Without the fix
NVLS disabled	No multicast resources are allocated	The job uses another NCCL transport path
NVLS forced	Initialization fails when NVLS resources cannot be allocated	Useful for qualification, risky as an unexplained production default
NVLS automatic	NCCL selects the path when support and resources are available	The platform can continue without NVLS when it is unavailable

Diagnostic note

“A clean run with NVLS disabled proves the failing path, not the failing part. Preserve that distinction in the incident. Otherwise a temporary transport rollback becomes a false hardware diagnosis or an unsupported permanent tuning rule.”

Visual fingerprint

NVLS initialization decision path

first NVLS signature
        |
        v
record selection mode
        |
        v
same collective, NVLS off
        |
        +-- passes --> inspect fabric, resources, stack
        +-- fails  --> find earlier non-NVLS owner
        |
        v
re-enable and verify

Transport isolation comes before a claim about resource exhaustion, hardware failure, or software regression.

Diagnose this failure in VS Code

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

Install the free VS Code extension

NCCL errors in context

NCCL is where a distributed job reports failure, which is not the same as where it failed. The hub lists every common NCCL error next to what it actually indicates, and the environment variables that tell them apart.

Compare every nccl error side by side

Related failures to investigate next

Use the exact-signature guide when transport/nvls.cc reports CUDA failure 1 and invalid argument. NCCL NVLS invalid argument →

Use the resource-specific guide only when allocation or platform evidence actually reports exhausted multicast capacity. NVLS multicast slot exhaustion →

Root cause

NCCL could not create, bind, or use the multicast resources required by the selected NVLS path.
The causal owner can be unsupported or forced configuration, unavailable resources, inconsistent software components, or unhealthy fabric state.
A later Xid, timeout, or communicator abort is not enough to choose among those causes without the first NVLS and system log evidence.

The fix and how to prevent it

Unlock the full remediation runbook

14 days on the Scale plan, up to 50 diagnoses a day. Step-by-step remediation, the RMA evidence payload, and multi-node correlation on your own logs. No card, and it does not roll into a subscription.

Frequently asked questions

Questions engineers and on-call staff commonly ask about this failure.

What does an NVLS initialization error mean?

It means NCCL could not complete NVLink SHARP multicast setup. The first error and a controlled replay are needed to distinguish support, resources, software, and fabric state.

Does disabling NVLS fix the system?

It can contain the incident by selecting another NCCL path. It does not repair or identify the underlying NVLS failure.

Should NVLS be forced on?

Forced mode is useful when a qualification must fail if NVLS is unavailable. Automatic mode is safer for general operation because unsupported or unavailable NVLS can be skipped.

What proves recovery?

The intended NVLS configuration must initialize successfully, the same collective must complete, and platform health must remain clean through the agreed observation window.

References

Don't just read the fix, diagnose your run

The encyclopedia tells you what went wrong. Denpex tells you what went wrong in YOUR training run. With your logs, your config, and your stack.

Diagnose your logs free Browse all failures

Related Communication errors

NCCL CUDA Failure

Communication · critical

NCCL P2P Disabled

Communication · medium

NCCL Hang Detection

Communication · high

NCCL Wrong Rank Configuration

Communication · critical