NCCL NVLS CUDA failure 1: invalid argument
This NCCL warning occurs while the NVLink SHARP transport is creating or registering multicast resources. The line identifies the failing NVLS path, but it does not prove whether the owner is application input, NCCL, CUDA, Fabric Manager, or the NVSwitch domain.
Capture the first NVLS warning from every rank, compare the same collective with NVLS disabled, and inspect Fabric Manager and GPU health before changing the production stack. NCCL_NVLS_ENABLE=0 is a diagnostic control, not proof of the final cause.
What this failure is
The signature transport/nvls.cc ... Cuda failure 1 'invalid argument' is emitted from NCCL's NVLink SHARP setup path. NVIDIA documents NVLS as an NVSwitch offload for supported Hopper and later systems. The surrounding warning, selected topology, rank, and fabric state are required to determine why the CUDA operation rejected an argument.
Is this what broke your run? Paste your log.
You're reading about NCCL NVLS CUDA failure 1: invalid argument. Paste your own crash log or traceback below and get the real root cause for YOUR run, not this generic entry. No account, no card. Logs are masked at ingress and never saved to account history.
Want 14 days on the Scale plan?
Request a work-email trial for up to 50 diagnoses a day, alerts, history, and follow-up questions. No credit card or automatic subscription.
Why it happens (the mechanism)
NCCL can enter the NVLS path only when the detected GPU, NVSwitch, driver, and runtime state support multicast setup. An invalid-argument warning at that point can reflect an unsupported or inconsistent resource, a software compatibility defect, or unhealthy fabric state. The warning alone is not evidence that the user's collective parameters are wrong, because it is emitted inside the NVLS transport rather than at the original collective call.
What you'll observe
- Communicator initialization or the first collective fails while NCCL is configuring NVLS.
- The same workload can succeed after NVLS is disabled, which proves path ownership but not the specific failed component.
- Aggregated framework errors can hide the first rank and first NVLS warning.
Common symptoms and what they mean
| Symptom | Why it happens |
|---|---|
| transport/nvls.cc:598 NCCL WARN Cuda failure 1 'invalid argument' | The direct failure is an invalid CUDA operation or resource passed while NCCL configures the NVLS multicast path. |
| NVLS warnings appear during communicator initialization before useful training or serving work begins. | The underlying owner remains one of several possibilities until a control separates software compatibility, NVLS support, resource state, and fabric health. |
| One or more ranks report a later NCCL failure after the first NVLS warning. | A later collective timeout or communicator abort is downstream evidence and should not replace the first NVLS warning in the incident record. |
Which systems are affected
- Hopper or later GPUs on supported NVSwitch systems
- NCCL communicators that select NVLink SHARP
- Multi-process jobs where rank-local logs are written separately
How to confirm this is the problem
Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.
- ✓Find the first NCCL WARN line containing nvls.cc and preserve at least 100 preceding lines for the same rank.
- ✓Confirm from NCCL_DEBUG_SUBSYS=GRAPH,NVLS that NCCL selected the NVLS path.
- ✓Repeat one minimal collective with the same ranks, message size, and topology while changing only NCCL_NVLS_ENABLE from automatic selection to 0.
- ✓If the failure follows NVLS, compare stack versions and Fabric Manager or NVSwitch health with a known-good node.
- ✓Require communicator initialization and the same collective to pass after the targeted correction before resuming the job.
Searchable error signature
transport/nvls.cc:598 NCCL WARN Cuda failure 1 'invalid argument'Use this text as a lookup key in logs and upstream issue trackers. It is not presented as a captured customer log. Confirm the cause from your own preceding events, versions, configuration and the cited references.
The fix and the prevention pattern
The root cause is on this page and stays free. A free account adds the exact remediation steps, keeps your diagnoses instead of discarding them, and unlocks the fix on every entry in the encyclopedia.
Sign up free. Unlock the full analysisNo credit card · 3 free diagnoses · Instant access
Why the recommended fix works
The NVLS-off control changes one transport decision while keeping the workload and topology fixed. A pass localizes the incident to multicast setup without claiming which NVLS component failed. Version and fabric comparisons then distinguish a software mismatch from unhealthy system state.
Code examples
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,GRAPH,NVLS
export NCCL_DEBUG_FILE=/tmp/nccl_%h_%p.log
# Run the same minimal collective once as the control
export NCCL_NVLS_ENABLE=0
<same-nccl-test-command>
# Preserve platform state
nvidia-smi -q
systemctl status nvidia-fabricmanagerAdapt the snippet to your framework. The same pattern holds for PyTorch Lightning, Hugging Face Trainer, DeepSpeed, Megatron-LM, and vLLM training wrappers. Where the wrapper exposes a config flag (for examplelr_scheduler_type in Trainer), prefer the flag over the imperative API to keep the schedule declarative and reproducible.
Best practices by model family
| Model / Stack | Recommendation | Notes |
|---|---|---|
| Evidence capture | Write one NCCL log per host and process | The first NVLS warning and emitting rank carry more causal value than the final framework exception. |
| Diagnostic control | Disable NVLS for one identical replay | A pass localizes the path. It does not by itself justify a permanent production setting. |
| Recovery | Re-enable only after the same collective passes | Verify both correctness and expected bandwidth before returning the node to service. |
With the fix vs without the fix
| Dimension | With the fix | Without the fix |
|---|---|---|
| Failure with NVLS automatic | The same invalid-argument warning recurs | The collective completes with NVLS disabled |
| Interpretation | NVLS path remains implicated | Application-level collective arguments are less likely to be the owner |
| Next discriminator | Compare fabric state and stack versions | Do not stop at the workaround |
Diagnostic note
“Do not label every invalid argument in an NCCL job as an application bug. The file and subsystem in the first warning matter. A warning from transport/nvls.cc should be tested as an NVLS-path failure before changing collective shapes or training code.”
Visual fingerprint
first nvls.cc warning
|
v
same collective, NVLS off
|
+-- passes --> inspect NVLS stack and fabric
+-- fails --> inspect earlier CUDA or application evidence
|
v
repeat after one targeted correctionDiagnose this failure in VS Code
Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.
Install the free VS Code extensionNCCL errors in context
NCCL is where a distributed job reports failure, which is not the same as where it failed. The hub lists every common NCCL error next to what it actually indicates, and the environment variables that tell them apart.
Compare every nccl error side by sideRelated failures to investigate next
Root cause
- The direct failure is an invalid CUDA operation or resource passed while NCCL configures the NVLS multicast path.
- The underlying owner remains one of several possibilities until a control separates software compatibility, NVLS support, resource state, and fabric health.
- A later collective timeout or communicator abort is downstream evidence and should not replace the first NVLS warning in the incident record.
The fix and how to prevent it
Unlock the full remediation runbook
14 days on the Scale plan, up to 50 diagnoses a day. Step-by-step remediation, the RMA evidence payload, and multi-node correlation on your own logs. No card, and it does not roll into a subscription.
Frequently asked questions
Questions engineers and on-call staff commonly ask about this failure.
Does Cuda failure 1 invalid argument prove my collective arguments are wrong?
Should I permanently set NCCL_NVLS_ENABLE=0?
What logs should I collect?
When is the incident resolved?
Don't just read the fix, diagnose your run
The encyclopedia tells you what went wrong. Denpex tells you what went wrong in YOUR training run. With your logs, your config, and your stack.