DistributedDataParallel raises an internal reducer assertion when a backward pass arrives unexpectedly

DDP's gradient reducer arms itself for exactly one backward pass per forward and disarms once that pass is finalised. A second backward, or a backward whose forward was run under a different synchronisation context, arrives when the reducer is not armed and trips an internal assertion whose text invites you to file a PyTorch bug.

An internal assertion in DDP's gradient reducer raised when autograd hooks fire while the reducer is not armed for a backward pass, caused by backward counting or synchronisation-context mismatches rather than by a defect in PyTorch.

DDP overlaps gradient reduction with backward computation by attaching hooks to parameters and reducing buckets as they fill. That only works if it knows when a backward begins and ends, so it arms itself before one and disarms after. Anything that produces an extra backward, or that moves a backward relative to the forward it belongs to, breaks the bookkeeping rather than the mathematics, and the assertion is the bookkeeping noticing.

• The error asks you to report a bug to PyTorch, which is almost never the right response
• The same model trains correctly on a single GPU and only fails once wrapped in DDP
• It appears after enabling gradient checkpointing or a static graph, neither of which mentions backward counting
• The traceback points into C++ reducer internals with no application frame to act on

• DDP with static_graph enabled, which assumes one unchanging backward per iteration
• Gradient checkpointing under DDP, where the forward is re-run during backward and can re-fire hooks
• Gradient accumulation using no_sync, where a forward and its backward can end up on opposite sides of the context
• Models returning several tensors that are each backwarded separately

Apply the following checklist to a small reproduction: each box below is a positive signal that you are looking at this exact failure rather than a sibling in the same taxonomy.

• ✓Count backward calls per forward in the training step. More than one against the DDP-wrapped module is the condition the assertion describes.
• ✓Disable static_graph and re-run. Success identifies the strict contract as the trigger even when the underlying pattern is what needs changing.
• ✓Disable gradient checkpointing and re-run. Working with one of the two enabled but not both is the signature of the reentrant checkpointing conflict.

[rank0]: RuntimeError: expect_autograd_hooks_ INTERNAL ASSERT FAILED at "../torch/csrc/distributed/c10d/reducer.cpp":1603, please report a bug to PyTorch.
[rank0]: No no_sync() context - works without error

Timestamps and exact values vary across runs, but the pattern. An info-level start, an early WARN, an ERROR carrying the symptom. Is the actual fingerprint you should alert on. The Denpex platform flags this combination automatically.

Collapsing several losses into one backward restores the one-per-forward assumption the reducer is built on. Non-reentrant checkpointing avoids re-running the forward in a way that re-fires hooks. Aligning no_sync boundaries makes the reducer's expectation and the actual sequence agree again. All three fix the sequence rather than suppressing the check.

// Typical pattern:
import torch.optim as optim
optimizer = optim.AdamW(model.parameters(), lr=3e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_training_steps)
for step in range(num_training_steps):
    optimizer.zero_grad()
    loss = model(batch)
    loss.backward()
    optimizer.step()
    scheduler.step()

Adapt the snippet to your framework. The same pattern holds for PyTorch Lightning, Hugging Face Trainer, DeepSpeed, Megatron-LM, and vLLM training wrappers. Where the wrapper exposes a config flag (for examplelr_scheduler_type in Trainer), prefer the flag over the imperative API to keep the schedule declarative and reproducible.