Python 3.13 cannot pickle code objects, so a distributed job reports the wrong failure

An interpreter upgrade changes what pickle will serialise, and traceback objects stop qualifying. Any machinery that ships an exception between ranks then fails while packing it, so the error the cluster reports is the packing failure and the original fault is never printed.

Symptom

TypeError: cannot pickle code objects

Root cause

An extracted traceback is not a lightweight record. Its frames reference code objects, and code objects are compiled bytecode with no stable serialised representation, so pickling them was always questionable. The newer interpreter stops permitting it, which is a correctness decision rather than a regression.

Recommended fix

Convert the traceback to text at the point of capture and send the string. Formatting produces exactly what a human needs, and a string pickles everywhere.

How Denpex helps

Denpex matches Python 3.13 cannot pickle code objects, so a distributed job reports the wrong failure across every rank in a distributed run and reports which rank failed first, so you act on the initiating node instead of the loudest one.

An interpreter-upgrade failure in which traceback objects can no longer be pickled, so any code path that serialises an exception to move it between processes fails and replaces the original diagnosis with its own.

Serialising compiled bytecode was never well defined across interpreter versions, and the newer release stops pretending otherwise. Distributed frameworks had come to rely on it because moving an exception whole is the most convenient way to re-raise it on another rank. The reliance is invisible until something fails, at which point the convenience becomes the failure.

• The reported error is about pickling and has nothing to do with what actually went wrong
• The real exception is destroyed by the reporting path, so there is nothing to search for
• It only appears on the newer interpreter, so it reads as a framework bug rather than an environment change
• It surfaces on the error path, which by definition is the least exercised code in the system

• Distributed error propagation, where a rank's exception is gathered to rank zero to be re-raised
• Asynchronous checkpoint save, which reports a worker's failure back to the caller
• Process-pool executors and any launcher that returns exceptions across a process boundary
• Anything that stores an extracted traceback for later inspection or logging

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Extract a traceback and attempt to pickle it in a bare interpreter session. Failing on the new version and succeeding on the old one confirms the interpreter, not the framework.
• ✓Read the per-rank log files rather than the aggregated output; the original exception was recorded there before the propagation attempt.
• ✓Check whether the failing frame is inside object gathering or a pool executor's result handling, which is where a traceback would be serialised.

• An extracted traceback is not a lightweight record. Its frames reference code objects, and code objects are compiled bytecode with no stable serialised representation, so pickling them was always questionable. The newer interpreter stops permitting it, which is a correctness decision rather than a regression.
• The failure lands on the reporting path specifically because that is the only place a traceback is deliberately carried around. Normal operation never pickles one, so an upgrade can pass every test and every training step and still break the moment something else fails.
• The consequence is a lost diagnosis rather than a lost job. Whatever genuinely failed raised first; the attempt to describe it to the other ranks raised second; and only the second one reaches the log.

TypeError: cannot pickle code objects
    pickle.Pickler(io.BytesIO()).dump(trace)
[<FrameSummary file /tmp/pickly.py, line 2 in <module>>]
AssertionError: 'fail_once policy triggered failure' not found in 'cannot pickle code objects'

Use this text as a lookup key in logs and upstream issue trackers. It is not presented as a captured customer log. Confirm the cause from your own preceding events, versions, configuration and the cited references.

Formatting the traceback at capture time preserves everything a person reads from it, the frames, the lines, the final message, in a form that has no version sensitivity at all. Carrying the type name separately keeps whatever programmatic branching depended on the exception class. Nothing of diagnostic value is lost, and the transport stops depending on an interpreter implementation detail.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.