vLLM crashes initialising an NVFP4 model because the emulation lookup table stays on the CPU

When no hardware NVFP4 backend is available, vLLM falls back to an emulation path that unpacks four-bit values through a small constant lookup table. The table is indexed by a tensor that lives on the GPU while the table itself was built on the CPU, so engine initialisation fails before a single token is generated.

Symptom

[nvfp4.py:283] Using 'EMULATION' NvFp4 MoE backend out of potential backends: ['FLASHINFER_TRTLLM', 'FLASHINFER_CUTEDSL', 'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'EMULATION'].

Root cause

PyTorch requires the index tensor and the tensor being indexed to live on one device, or the index to be on the CPU. The emulation helper builds its constant table as a plain tensor, which lands on the CPU, and then indexes it with values that have already been moved to the accelerator, so the pair straddles the boundary the rule forbids. The path is only reached when backend selection has exhausted every accelerated option.

Recommended fix

Check which backend was chosen. The log states the selected NvFp4 backend and lists the alternatives, and emulation appearing there is the signal that no accelerated path was available.

How Denpex helps

Denpex matches vLLM crashes initialising an NVFP4 model because the emulation lookup table stays on the CPU across every rank in a distributed run and reports which rank failed first, so you act on the initiating node instead of the loudest one.

An initialisation-time crash in vLLM's software NVFP4 fallback, where a constant unpacking table allocated on the CPU is indexed by accelerator-resident values, violating PyTorch's requirement that both live on one device.

Backend selection is automatic and silent, and emulation is the last resort. Because it is rarely reached, its helper carries an assumption that no accelerated path makes: that the small constant table and the values indexing it are on the same device. The moment the fallback is genuinely used, that assumption meets a tensor already moved to the accelerator and PyTorch refuses the operation.

• The model loads and then the engine dies during initialisation rather than during generation
• The same checkpoint is reported working elsewhere, on hardware with a real NVFP4 backend
• The error names an indexing rule rather than the quantization format, so the connection is not obvious
• Nothing in the launch command mentions emulation, which was selected automatically

• NVFP4 and modelopt_fp4 checkpoints served by vLLM
• Hardware or builds where no FlashInfer, CUTLASS or Marlin NVFP4 backend is available, leaving emulation as the only option
• Mixture-of-experts models, where the emulated path is reached through the MoE layers
• Deployments that set the batch-invariant mode, which can exclude the faster backends

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Search the startup log for the line naming the selected NvFp4 backend. Emulation there confirms the fallback was taken and the accelerated paths were unavailable.
• ✓Check whether batch-invariant mode is set in the environment, since it can remove the accelerated backends from consideration.
• ✓Launch the identical checkpoint on hardware with a supported NVFP4 backend. Success there localises the fault to the fallback path rather than to the weights.

• PyTorch requires the index tensor and the tensor being indexed to live on one device, or the index to be on the CPU. The emulation helper builds its constant table as a plain tensor, which lands on the CPU, and then indexes it with values that have already been moved to the accelerator, so the pair straddles the boundary the rule forbids.
• The path is only reached when backend selection has exhausted every accelerated option. That makes the failure look hardware-specific when it is really a fallback that few deployments exercise, which is why the same checkpoint is reported working on machines that never enter emulation.
• It fires during initialisation because the unpacking happens while weights are being prepared rather than while tokens are being produced, so there is no partially working server to inspect.

[nvfp4.py:283] Using 'EMULATION' NvFp4 MoE backend out of potential backends: ['FLASHINFER_TRTLLM', 'FLASHINFER_CUTEDSL', 'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'EMULATION'].
  File ".../vllm/model_executor/layers/quantization/utils/nvfp4_emulation_utils.py", line 38, in break_fp4_bytes
    values = kE2M1[abs_vals] * torch.where(signs, -1.0, 1.0)
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)

Use this text as a lookup key in logs and upstream issue trackers. It is not presented as a captured customer log. Confirm the cause from your own preceding events, versions, configuration and the cited references.

Restoring an accelerated backend removes the emulated helper from the execution path altogether, so the mismatched pair is never constructed. It is a better fix than forcing the table onto the accelerator would be, because emulation is a correctness reference rather than something a served deployment should be running on at all.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

vLLM failures cross request, model, KV cache, worker, kernel and host boundaries. The hub compares common literal signatures and the first control that separates their causal owners.