TensorRT-LLM fails at model init because the FlashInfer FMHA kernel has no build for the GPU architecture

TensorRT-LLM delegates fused attention to FlashInfer, whose FMHA runner is compiled for a specific set of architectures. On a device outside that set the runner raises an unsupported-architecture error from its own header, so the model finishes loading and then fails to serve.

A runtime rejection in which the FlashInfer fused multi-head attention runner bundled with TensorRT-LLM finds no compiled kernel for the target GPU architecture and raises rather than falling back, after model initialisation has already succeeded.

Attention kernels are compiled ahead of time for an enumerated list of architectures and dispatched at runtime. Dispatch either finds an entry for the device or it does not, and there is no generic path to degrade to, so the runner raises from inside its own header. Because the enumeration belongs to the bundled kernel library rather than to the framework, nothing in the TensorRT-LLM invocation reflects it.

• The container release is current and the model is supported, yet initialisation fails on this card
• The error is raised from a FlashInfer header path, which does not appear in the TensorRT-LLM command that was run
• Both a small inference card and a new flagship can hit it, which makes the pattern look inconsistent
• Model loading reports success first, so the failure looks like a serving problem rather than a support one

• GPUs whose compute capability is outside the set the bundled FlashInfer FMHA was compiled for, in both directions
• Newer silicon released after the container image, such as SM120 Blackwell workstation parts
• Smaller inference cards such as the L4, which are sometimes omitted from a kernel's build list
• Release-candidate TensorRT-LLM containers, whose bundled kernel coverage moves between builds

Apply the following checklist to a small reproduction: each box below is a positive signal that you are looking at this exact failure rather than a sibling in the same taxonomy.

• ✓Query the device compute capability and check it against the architecture list the installed FlashInfer build advertises. A device absent from that list is the whole explanation.
• ✓Run the identical image and command on a different GPU generation. Success there separates a kernel coverage gap from a problem with the model or the configuration.
• ✓Confirm the failure occurs after the model init timing line. A rejection at that point is kernel dispatch rather than weight loading.

RuntimeError: Error in function 'TllmGenFmhaRunner' at /usr/local/lib/python3.12/dist-packages/flashinfer/data/include/flashinfer/trtllm/fmha/fmhaRunner.cuh:37: Unsupported architecture
Model init total -- 21.36s
[TRT-LLM] [W] [quantize] Inline 'config.json.quantization_config' diverges from 'hf_quant_config.json'

Timestamps and exact values vary across runs, but the pattern. An info-level start, an early WARN, an ERROR carrying the symptom. Is the actual fingerprint you should alert on. The Denpex platform flags this combination automatically.

Changing to a build that enumerates the architecture supplies the object code dispatch was looking for, which is the only thing missing. Selecting a different attention provider works for the same reason: it moves the request to a kernel set that does cover the device. Neither is a tuning change, because nothing about the model or its configuration was wrong.

// Typical pattern:
import torch.optim as optim
optimizer = optim.AdamW(model.parameters(), lr=3e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_training_steps)
for step in range(num_training_steps):
    optimizer.zero_grad()
    loss = model(batch)
    loss.backward()
    optimizer.step()
    scheduler.step()

Adapt the snippet to your framework. The same pattern holds for PyTorch Lightning, Hugging Face Trainer, DeepSpeed, Megatron-LM, and vLLM training wrappers. Where the wrapper exposes a config flag (for examplelr_scheduler_type in Trainer), prefer the flag over the imperative API to keep the schedule declarative and reproducible.