Megatron errors

Megatron-LM divides large models across 3D parallel boundaries: tensor, pipeline, and data parallelism. An error in one domain often cascades, making configuration mismatches look like catastrophic hardware failures.

The most common failures involve loading checkpoints with a different parallel topology than they were saved with, or initializing process groups with incorrect rank assignments. Always verify your parallel dimensions first.

Diagnose your log with evidence Full failure encyclopedia

Working in VS Code or Remote-SSH? Diagnose your log locally with the free VS Code extension. No account or upload required for local diagnosis.

Every common megatron error

The literal string is what you paste into a search bar, so it is the heading. “Class” is a triage category. It is not proof of ownership in your workload.

TP size mismatch

Configuration

RuntimeError: Checkpoint tensor parallel size mismatch

The tensor parallel size used to launch the job does not match the TP size of the saved checkpoint. Megatron cannot automatically reshape weights across arbitrary TP dimensions during load.

First action: Check the --tensor-model-parallel-size launch argument and compare it to the checkpoint topology. Use the provided Megatron checkpoint conversion python script to reshape.

PP rank mismatch

Configuration

AssertionError: Pipeline parallel rank mismatch during initialization

The global world size is not perfectly divisible by the product of tensor, pipeline, and data parallel sizes. This leaves stranded ranks that cannot join the 3D grid.

First action: Calculate TP * PP * DP and ensure it exactly equals the total number of GPUs launched by the --nnodes and --nproc_per_node arguments.

Sequence parallel hang

Application bug

NCCL Timeout in sequence parallel all-gather

Ranks in the tensor parallel group stalled during a sequence parallel collective. This often happens if micro-batch shapes vary or activation checkpoints drop data.

First action: Set TORCH_DISTRIBUTED_DEBUG=DETAIL and verify that sequence length is perfectly divisible by the tensor parallel size across all ranks.

Errors not listed here exist. Rather than guess at their meaning, check Megatron-LM GitHub Repository, which is the authority for the strings above.

Frequently asked questions

Why does my Megatron run hang during the first forward pass?

Usually due to a mismatch in micro-batch sizes or pipeline parallel schedules. If stage one sends a tensor shape that stage two is not expecting, the receiver blocks forever while the sender moves on.

How do I change the tensor parallel size of a Megatron checkpoint?

You cannot just change the launch argument. You must run the offline checkpoint manipulation script provided in the Megatron repository to merge the old checkpoint and split it into the new tensor parallel topology.

What causes NCCL timeouts exclusively in Megatron sequence parallelism?

Sequence parallelism relies on splitting the sequence length across the TP group. If the total sequence length is not cleanly divisible by the TP size, the all-gather operations will have mismatched expected byte counts and hang.

Is it possible to recover a corrupted Megatron distributed checkpoint?

If only one rank failed to write its shard, you may be able to resume from an older global checkpoint. Megatron requires all ranks in the parallel groups to successfully commit their shards to guarantee mathematical correctness.

Knowing the error is not knowing the cause

Which rank failed first, whether the checkpoint is safe to resume from, and whether this is your code or the hardware. Paste the log and Denpex answers all three, free and without an account.

Diagnose your logs free