TensorRT-LLM serving fails with an AttributeError from the FlashInfer attention backend

The FlashInfer attention backend inside TensorRT-LLM reads fields from an attention metadata object that the release bundled beside it does not define. Serving fails with a plain Python AttributeError naming the missing field, so a version-skew problem between two components arrives looking like ordinary application code being wrong.

A serving failure in which TensorRT-LLM's FlashInfer attention backend accesses a metadata field the bundled build does not define, raising an AttributeError at the first fused-attention call rather than at load time.

Python resolves attributes when they are reached, not when the module is imported. A backend compiled against one metadata contract can therefore load cleanly beside a build that implements another, and the mismatch waits until fused attention runs. Because model defaults choose the backend automatically, the component that breaks is one the operator never selected and never sees until the traceback.

• The failure names an attribute, which reads as a coding error rather than a packaging one
• The model, the flags and the hardware are all supported, and the command is the documented one
• It appears in a release-candidate container that is newer than the last one that worked
• Changing model or batch settings has no effect, because none of them are involved

• trtllm-serve running any model whose defaults select the FlashInfer attention backend
• Release-candidate TensorRT-LLM containers, where the two components move independently between builds
• Environments where FlashInfer, TensorRT-LLM or torch has been upgraded in place inside the image
• Model families such as Gemma whose applied defaults choose FlashInfer without the operator asking for it

Apply the following checklist to a small reproduction: each box below is a positive signal that you are looking at this exact failure rather than a sibling in the same taxonomy.

• ✓Find the applied-model-defaults line in the startup log and confirm it selected FlashInfer, which identifies the component the traceback belongs to.
• ✓Compare the installed TensorRT-LLM and FlashInfer versions against the pairing the published image shipped with; a difference in either shows the image was modified.
• ✓Re-run with a different attention backend. Serving successfully confirms interface skew rather than a fault in the model or the request.

AttributeError: 'TrtllmAttentionMetadata' object has no attribute 'kv_layout'
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/attention_backend/flashinfer.py", line 1518, in forward_impl
    self.layer_idx, kv_layout=metadata.kv_layout)
[TRT-LLM] [I] [_torch] Applied model defaults for Gemma4ForConditionalGeneration: {'attn_backend': 'FLASHINFER'}

Timestamps and exact values vary across runs, but the pattern. An info-level start, an early WARN, an ERROR carrying the symptom. Is the actual fingerprint you should alert on. The Denpex platform flags this combination automatically.

Using a container whose components were released together restores the contract the backend was written against, so the field it reads exists. Overriding the backend achieves the same outcome from the other direction, by routing attention through an implementation this build does satisfy. Neither is a tuning change, because no setting of the model or the request was ever involved.

// Typical pattern:
import torch.optim as optim
optimizer = optim.AdamW(model.parameters(), lr=3e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_training_steps)
for step in range(num_training_steps):
    optimizer.zero_grad()
    loss = model(batch)
    loss.backward()
    optimizer.step()
    scheduler.step()

Adapt the snippet to your framework. The same pattern holds for PyTorch Lightning, Hugging Face Trainer, DeepSpeed, Megatron-LM, and vLLM training wrappers. Where the wrapper exposes a config flag (for examplelr_scheduler_type in Trainer), prefer the flag over the imperative API to keep the schedule declarative and reproducible.