MXFP4 expert weights lose their scale attribute when a mixture-of-experts model loads under FSDP2

A quantised mixture-of-experts checkpoint loads on a single device and fails under sharded loading. Materialising a module from the placeholder device reads each parameter by name, and the quantised expert block does not expose the scale attribute the checkpoint expects, so loading stops on a missing attribute with a helpful suggestion naming a different one.

A load-time failure in which sharded materialisation resolves a checkpoint key against a live quantised module by attribute name, and the quantised expert implementation does not expose the scale tensor under the name the checkpoint uses.

Sharded construction is designed to avoid ever holding the whole model, so it builds an empty skeleton and fills each parameter in as its key is read. That requires the module to already expose every name the checkpoint will mention. A quantised implementation is free to store packed weights and scales however it likes, and when its choice differs from the checkpoint's key layout, the discrepancy surfaces here rather than in a conventional load that assigns into a dictionary first.

• The identical checkpoint loads correctly without sharding, so the file is plainly fine
• The error names a missing attribute and suggests a near neighbour, which invites editing the name rather than understanding it
• It fails during loading, before any training step, so no gradient or collective is involved
• Quantised experts and sharded loading are each supported, and only their combination fails

• Mixture-of-experts models whose expert weights are stored in a block-scaled four-bit format
• Fully sharded loading, which builds the model without data and fills parameters in by name
• Quantised module implementations that register some tensors as parameters and derive or fuse others
• Any loader that pairs a checkpoint key with an attribute lookup on the live module

Apply the following checklist to a small reproduction: each box below is a positive signal that you are looking at this exact failure rather than a sibling in the same taxonomy.

• ✓Load the same checkpoint on one device with no sharding. Succeeding there confirms the sharded materialisation path rather than the checkpoint.
• ✓List the expert module's parameters and buffers and compare the names against the checkpoint keys. The absent name is the disagreement, stated exactly.
• ✓Check the quantisation library version against the one that produced the checkpoint, since these attribute names change with the scheme.

AttributeError: 'Mxfp4GptOssExperts' object has no attribute 'down_proj_scales'. Did you mean: 'down_proj_bias'?
  File ".../transformers/modeling_utils.py", in _load_state_dict_into_meta_model
    value = getattr(module, param_type)
Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]

Timestamps and exact values vary across runs, but the pattern. An info-level start, an early WARN, an ERROR carrying the symptom. Is the actual fingerprint you should alert on. The Denpex platform flags this combination automatically.

Loading before sharding restores the ordinary path, where the checkpoint is read into a fully constructed model and the quantised module can reconcile its own tensors before anything is sharded. Aligning the quantisation library with the one that wrote the checkpoint removes the disagreement at its source. Both mean the name being looked up is one the module actually has.

// Typical pattern:
import torch.optim as optim
optimizer = optim.AdamW(model.parameters(), lr=3e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_training_steps)
for step in range(num_training_steps):
    optimizer.zero_grad()
    loss = model(batch)
    loss.backward()
    optimizer.step()
    scheduler.step()

Adapt the snippet to your framework. The same pattern holds for PyTorch Lightning, Hugging Face Trainer, DeepSpeed, Megatron-LM, and vLLM training wrappers. Where the wrapper exposes a config flag (for examplelr_scheduler_type in Trainer), prefer the flag over the imperative API to keep the schedule declarative and reproducible.