TensorRT-LLM rejects FP8 quantization on a GPU below compute capability 8.9

FP8 in TensorRT-LLM requires hardware FP8 tensor cores, which exist from compute capability 8.9 onward. A build targeting an Ampere device such as the A100, which reports compute capability 8.0, is rejected because the silicon has no FP8 path at all.

A build-time rejection in which TensorRT-LLM declines to produce an FP8 engine because the target GPU reports a compute capability below 8.9 and therefore has no FP8 tensor cores for the quantized kernels to run on.

NVIDIA documents FP8 support as requiring compute capability above 8.9, covering Ada, Hopper and Blackwell. Ampere reports 8.0 and has no FP8 tensor cores at all, so there is no kernel for TensorRT-LLM to select. The check runs when the engine is built rather than when it is loaded, which is why building on an Ampere host for a Hopper target fails as well unless a supported cross-compilation flow is used.

• The engine build fails immediately on an A100 while the same command succeeds on an H100
• A blog post or vendor page claims the GPU is supported, contradicting the build result
• Cross-compiling on an Ampere host for a Hopper target fails even though the target would support FP8
• It is unclear which quantization to use instead without losing the memory saving that motivated FP8

• NVIDIA Ampere data-centre GPUs, including A100 SXM and PCIe and A30, all reporting compute capability 8.0
• Turing and Volta parts, which are further below the threshold
• Any host performing an engine build for FP8, because the check runs at build time rather than at load time

Apply the following checklist to a small reproduction: each box below is a positive signal that you are looking at this exact failure rather than a sibling in the same taxonomy.

• ✓Query the device compute capability directly, for example with nvidia-smi --query-gpu=compute_cap --format=csv, and compare it against the 8.9 threshold.
• ✓Re-run the identical build command on an Ada or Hopper device. Success there confirms the constraint is the architecture rather than the build configuration.
• ✓Check whether the build host and the deployment target are the same device, since the check is applied where the engine is built.

[TRT-LLM] [E] FP8 quantization is not supported on this GPU architecture (compute capability 8.0). Rebuild the engine with a supported quantization mode.
[TRT-LLM] [I] Detected device 0: NVIDIA A100-SXM4-80GB, compute capability 8.0

Timestamps and exact values vary across runs, but the pattern. An info-level start, an early WARN, an ERROR carrying the symptom. Is the actual fingerprint you should alert on. The Denpex platform flags this combination automatically.

INT8 SmoothQuant and INT4 AWQ target integer tensor cores that Ampere does have, so they recover most of the throughput and memory benefit that motivated FP8 while remaining buildable. The change is to the quantization format rather than to any setting, because the constraint is silicon and not configuration.

// Typical pattern:
import torch.optim as optim
optimizer = optim.AdamW(model.parameters(), lr=3e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_training_steps)
for step in range(num_training_steps):
    optimizer.zero_grad()
    loss = model(batch)
    loss.backward()
    optimizer.step()
    scheduler.step()

Adapt the snippet to your framework. The same pattern holds for PyTorch Lightning, Hugging Face Trainer, DeepSpeed, Megatron-LM, and vLLM training wrappers. Where the wrapper exposes a config flag (for examplelr_scheduler_type in Trainer), prefer the flag over the imperative API to keep the schedule declarative and reproducible.