vLLM crashes initialising an NVFP4 model because the emulation lookup table stays on the CPU

When no hardware NVFP4 backend is available, vLLM falls back to an emulation path that unpacks four-bit values through a small constant lookup table. The table is indexed by a tensor that lives on the GPU while the table itself was built on the CPU, so engine initialisation fails before a single token is generated.

An initialisation-time crash in vLLM's software NVFP4 fallback, where a constant unpacking table allocated on the CPU is indexed by accelerator-resident values, violating PyTorch's requirement that both live on one device.

Backend selection is automatic and silent, and emulation is the last resort. Because it is rarely reached, its helper carries an assumption that no accelerated path makes: that the small constant table and the values indexing it are on the same device. The moment the fallback is genuinely used, that assumption meets a tensor already moved to the accelerator and PyTorch refuses the operation.

• The model loads and then the engine dies during initialisation rather than during generation
• The same checkpoint is reported working elsewhere, on hardware with a real NVFP4 backend
• The error names an indexing rule rather than the quantization format, so the connection is not obvious
• Nothing in the launch command mentions emulation, which was selected automatically

• NVFP4 and modelopt_fp4 checkpoints served by vLLM
• Hardware or builds where no FlashInfer, CUTLASS or Marlin NVFP4 backend is available, leaving emulation as the only option
• Mixture-of-experts models, where the emulated path is reached through the MoE layers
• Deployments that set the batch-invariant mode, which can exclude the faster backends

Apply the following checklist to a small reproduction: each box below is a positive signal that you are looking at this exact failure rather than a sibling in the same taxonomy.

• ✓Search the startup log for the line naming the selected NvFp4 backend. Emulation there confirms the fallback was taken and the accelerated paths were unavailable.
• ✓Check whether batch-invariant mode is set in the environment, since it can remove the accelerated backends from consideration.
• ✓Launch the identical checkpoint on hardware with a supported NVFP4 backend. Success there localises the fault to the fallback path rather than to the weights.

[nvfp4.py:283] Using 'EMULATION' NvFp4 MoE backend out of potential backends: ['FLASHINFER_TRTLLM', 'FLASHINFER_CUTEDSL', 'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'EMULATION'].
  File ".../vllm/model_executor/layers/quantization/utils/nvfp4_emulation_utils.py", line 38, in break_fp4_bytes
    values = kE2M1[abs_vals] * torch.where(signs, -1.0, 1.0)
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)

Timestamps and exact values vary across runs, but the pattern. An info-level start, an early WARN, an ERROR carrying the symptom. Is the actual fingerprint you should alert on. The Denpex platform flags this combination automatically.

Restoring an accelerated backend removes the emulated helper from the execution path altogether, so the mismatched pair is never constructed. It is a better fix than forcing the table onto the accelerator would be, because emulation is a correctness reference rather than something a served deployment should be running on at all.

// Typical pattern:
import torch.optim as optim
optimizer = optim.AdamW(model.parameters(), lr=3e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_training_steps)
for step in range(num_training_steps):
    optimizer.zero_grad()
    loss = model(batch)
    loss.backward()
    optimizer.step()
    scheduler.step()

Adapt the snippet to your framework. The same pattern holds for PyTorch Lightning, Hugging Face Trainer, DeepSpeed, Megatron-LM, and vLLM training wrappers. Where the wrapper exposes a config flag (for examplelr_scheduler_type in Trainer), prefer the flag over the imperative API to keep the schedule declarative and reproducible.