Weights-only load failed with an UnpicklingError when you resume an older checkpoint

A checkpoint that loaded without complaint for months starts failing after a framework upgrade. Nothing about the file changed. The loader now refuses by default to reconstruct any Python object that is not a plain tensor, and a training checkpoint carries several — the optimizer class among them — so the load stops on the first one it meets.

A resume-time failure in which a checkpoint written under a permissive deserialization default is read under a restrictive one, so the loader refuses to reconstruct the non-tensor objects the checkpoint legitimately contains.

Deserialization of a pickle is code execution, so the safe default is to rebuild nothing but tensors. Training checkpoints have always stored more than tensors, because resuming needs the optimizer, the scheduler and the run configuration. Those two facts were compatible only while the default was permissive; once it flipped, every previously written checkpoint became a file the reader declines to fully trust.

• The file is intact and readable, so every corruption check comes back clean
• The same file loaded successfully under the previous framework release
• The message offers a way to disable the restriction, which is the fastest fix and the wrong default
• It surfaces at resume, typically hours into a queue slot, on every rank at once

• Resuming any training checkpoint written before the loader's default changed
• Distributed checkpoint formats that store a common state dictionary alongside the sharded tensors
• Optimizer state, which pickles the optimizer class itself rather than only its numbers
• Learning-rate schedulers, dataloader samplers, and argument namespaces saved beside the weights

Apply the following checklist to a small reproduction: each box below is a positive signal that you are looking at this exact failure rather than a sibling in the same taxonomy.

• ✓Read the class named after Unsupported global. If it is an optimizer, a scheduler or an argument container, this is the strict-loader change and not damage to the file.
• ✓Load the same file with the restriction disabled once, in a scratch process. Success proves the bytes are fine and the reader's default is what moved.
• ✓Check whether the checkpoint predates your current framework release. A file written under the permissive default is the whole precondition.

_pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options
WeightsUnpickler error: Unsupported global: GLOBAL torch.optim.adamw.AdamW was not an allowed global by default
Please use `torch.serialization.add_safe_globals([torch.optim.adamw.AdamW])` to allowlist this global
  File ".../megatron/training/checkpointing.py", in _load_base_checkpoint
    state_dict = dist_checkpointing.load_common_state_dict(checkpoint_name)

Timestamps and exact values vary across runs, but the pattern. An info-level start, an early WARN, an ERROR carrying the symptom. Is the actual fingerprint you should alert on. The Denpex platform flags this combination automatically.

Allowlisting states, class by class, that you know what is in your own file. The loader keeps refusing everything you did not name, so the protection still applies to any checkpoint arriving from elsewhere. Disabling the restriction wholesale also loads the file, but it discards the protection for every object in it rather than for the one you inspected.

// Typical pattern:
import torch.optim as optim
optimizer = optim.AdamW(model.parameters(), lr=3e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_training_steps)
for step in range(num_training_steps):
    optimizer.zero_grad()
    loss = model(batch)
    loss.backward()
    optimizer.step()
    scheduler.step()

Adapt the snippet to your framework. The same pattern holds for PyTorch Lightning, Hugging Face Trainer, DeepSpeed, Megatron-LM, and vLLM training wrappers. Where the wrapper exposes a config flag (for examplelr_scheduler_type in Trainer), prefer the flag over the imperative API to keep the schedule declarative and reproducible.