Python 3.13 cannot pickle code objects, so a distributed job reports the wrong failure

An interpreter upgrade changes what pickle will serialise, and traceback objects stop qualifying. Any machinery that ships an exception between ranks then fails while packing it, so the error the cluster reports is the packing failure and the original fault is never printed.

An interpreter-upgrade failure in which traceback objects can no longer be pickled, so any code path that serialises an exception to move it between processes fails and replaces the original diagnosis with its own.

Serialising compiled bytecode was never well defined across interpreter versions, and the newer release stops pretending otherwise. Distributed frameworks had come to rely on it because moving an exception whole is the most convenient way to re-raise it on another rank. The reliance is invisible until something fails, at which point the convenience becomes the failure.

• The reported error is about pickling and has nothing to do with what actually went wrong
• The real exception is destroyed by the reporting path, so there is nothing to search for
• It only appears on the newer interpreter, so it reads as a framework bug rather than an environment change
• It surfaces on the error path, which by definition is the least exercised code in the system

• Distributed error propagation, where a rank's exception is gathered to rank zero to be re-raised
• Asynchronous checkpoint save, which reports a worker's failure back to the caller
• Process-pool executors and any launcher that returns exceptions across a process boundary
• Anything that stores an extracted traceback for later inspection or logging

Apply the following checklist to a small reproduction: each box below is a positive signal that you are looking at this exact failure rather than a sibling in the same taxonomy.

• ✓Extract a traceback and attempt to pickle it in a bare interpreter session. Failing on the new version and succeeding on the old one confirms the interpreter, not the framework.
• ✓Read the per-rank log files rather than the aggregated output; the original exception was recorded there before the propagation attempt.
• ✓Check whether the failing frame is inside object gathering or a pool executor's result handling, which is where a traceback would be serialised.

TypeError: cannot pickle code objects
    pickle.Pickler(io.BytesIO()).dump(trace)
[<FrameSummary file /tmp/pickly.py, line 2 in <module>>]
AssertionError: 'fail_once policy triggered failure' not found in 'cannot pickle code objects'

Timestamps and exact values vary across runs, but the pattern. An info-level start, an early WARN, an ERROR carrying the symptom. Is the actual fingerprint you should alert on. The Denpex platform flags this combination automatically.

Formatting the traceback at capture time preserves everything a person reads from it — the frames, the lines, the final message — in a form that has no version sensitivity at all. Carrying the type name separately keeps whatever programmatic branching depended on the exception class. Nothing of diagnostic value is lost, and the transport stops depending on an interpreter implementation detail.

// Typical pattern:
import torch.optim as optim
optimizer = optim.AdamW(model.parameters(), lr=3e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_training_steps)
for step in range(num_training_steps):
    optimizer.zero_grad()
    loss = model(batch)
    loss.backward()
    optimizer.step()
    scheduler.step()

Adapt the snippet to your framework. The same pattern holds for PyTorch Lightning, Hugging Face Trainer, DeepSpeed, Megatron-LM, and vLLM training wrappers. Where the wrapper exposes a config flag (for examplelr_scheduler_type in Trainer), prefer the flag over the imperative API to keep the schedule declarative and reproducible.