TensorRT-LLM fails at model init because the FlashInfer FMHA kernel has no build for the GPU architecture

TensorRT-LLM delegates fused attention to FlashInfer, whose FMHA runner is compiled for a specific set of architectures. On a device outside that set the runner raises an unsupported-architecture error from its own header, so the model finishes loading and then fails to serve.

Symptom

RuntimeError: Error in function 'TllmGenFmhaRunner' at /usr/local/lib/python3.12/dist-packages/flashinfer/data/include/flashinfer/trtllm/fmha/fmhaRunner.cuh:37: Unsupported architecture

Root cause

Fused attention is not written per model; it is a prebuilt kernel selected at runtime. The build carries object code for an enumerated list of architectures, and a device outside that list has nothing to dispatch to, so the runner refuses rather than falling back. The list is a property of the bundled FlashInfer build rather than of TensorRT-LLM or of the checkpoint, which is why the error names a header inside FlashInfer and why nothing in the invocation hints at it.

Recommended fix

Read the compute capability of the target device and compare it against the architectures the bundled kernel was built for, rather than assuming a current release covers current hardware.

How Denpex helps

Denpex matches TensorRT-LLM fails at model init because the FlashInfer FMHA kernel has no build for the GPU architecture across every rank in a distributed run and reports which rank failed first, so you act on the initiating node instead of the loudest one.

A runtime rejection in which the FlashInfer fused multi-head attention runner bundled with TensorRT-LLM finds no compiled kernel for the target GPU architecture and raises rather than falling back, after model initialisation has already succeeded.

Attention kernels are compiled ahead of time for an enumerated list of architectures and dispatched at runtime. Dispatch either finds an entry for the device or it does not, and there is no generic path to degrade to, so the runner raises from inside its own header. Because the enumeration belongs to the bundled kernel library rather than to the framework, nothing in the TensorRT-LLM invocation reflects it.

• The container release is current and the model is supported, yet initialisation fails on this card
• The error is raised from a FlashInfer header path, which does not appear in the TensorRT-LLM command that was run
• Both a small inference card and a new flagship can hit it, which makes the pattern look inconsistent
• Model loading reports success first, so the failure looks like a serving problem rather than a support one

• GPUs whose compute capability is outside the set the bundled FlashInfer FMHA was compiled for, in both directions
• Newer silicon released after the container image, such as SM120 Blackwell workstation parts
• Smaller inference cards such as the L4, which are sometimes omitted from a kernel's build list
• Release-candidate TensorRT-LLM containers, whose bundled kernel coverage moves between builds

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Query the device compute capability and check it against the architecture list the installed FlashInfer build advertises. A device absent from that list is the whole explanation.
• ✓Run the identical image and command on a different GPU generation. Success there separates a kernel coverage gap from a problem with the model or the configuration.
• ✓Confirm the failure occurs after the model init timing line. A rejection at that point is kernel dispatch rather than weight loading.

• Fused attention is not written per model; it is a prebuilt kernel selected at runtime. The build carries object code for an enumerated list of architectures, and a device outside that list has nothing to dispatch to, so the runner refuses rather than falling back.
• The list is a property of the bundled FlashInfer build rather than of TensorRT-LLM or of the checkpoint, which is why the error names a header inside FlashInfer and why nothing in the invocation hints at it.
• Coverage is not monotonic with newness. Hardware released after the image predates nothing it can use, and modest inference parts are sometimes left out of a build aimed at data-centre silicon, so both the newest and the smallest devices fail while the middle of the range works.

RuntimeError: Error in function 'TllmGenFmhaRunner' at /usr/local/lib/python3.12/dist-packages/flashinfer/data/include/flashinfer/trtllm/fmha/fmhaRunner.cuh:37: Unsupported architecture
Model init total -- 21.36s
[TRT-LLM] [W] [quantize] Inline 'config.json.quantization_config' diverges from 'hf_quant_config.json'

Use this text as a lookup key in logs and upstream issue trackers. It is not presented as a captured customer log. Confirm the cause from your own preceding events, versions, configuration and the cited references.

Changing to a build that enumerates the architecture supplies the object code dispatch was looking for, which is the only thing missing. Selecting a different attention provider works for the same reason: it moves the request to a kernel set that does cover the device. Neither is a tuning change, because nothing about the model or its configuration was wrong.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.