← Back to reference

Fsdp Guide

Overview

denpex.fsdp instruments PyTorch FSDP models with three integrations:

  1. Layer-level forward hooks (weight_l2, grad_l2).
  2. All-gather / reduce-scatter / reshard timers via hooks on the FSDP collectives.
  3. Per-shard state tracking (summon_full_params only on rank 0).

Setup

import denpex
from denpex.fsdp import FsdpInstrument

instrument = FsdpInstrument(
    track_reshard=True,
    track_prefetch=True,
    mixed_precision="bf16",
    cpu_offload=False,
    state_dict_type="FULL",
)
instrument.attach(model)

What We Capture

EventField
All-gatherrank, ns_pre, ns_post
Reduce-scatterrank, ns_pre, ns_post
Reshardafter_forward: bool
Prefetchparam_id, ns_until_used

Reshard-after-forward Toggle

Setting track_reshard_after_forward=True (default) ensures we observe the reshard boundary; common source of OOM is a missing reshard.

State-Dict Type

instrument.set_state_dict_type("FULL" | "SHARDED") records the checkpointing mode.

CPU Offload

instrument = FsdpInstrument(cpu_offload=True)

We capture the offload async copies as well.

What We Detect

  • FSDP_ALL_GATHER_OOM
  • FSDP_REDUCE_SCATTER_TIMEOUT
  • FSDP_RESHARD_MISSING

On Failure

We capture the traceback and the parameter shapes that were being gathered. The diagnosis includes which rank was slow.

Compatibility Tested

  • FSDP 2.0 (PyTorch 2.0+).
  • FSDP 2.1 (PyTorch 2.1+).
  • FSDP 2.2 (PyTorch 2.2+) with use_orig_params.