TensorRT-LLM FP8 support depends on GPU architecture, backend and model

An FP8 architecture rejection requires checking the selected TensorRT-LLM release, backend, quantization recipe and GPU. NVIDIA's reviewed support matrix includes Ada SM89 and Hopper SM90 FP8, but does not list native FP8 for Ampere SM80 or SM86.

Symptom

ERROR: FP8 quantization is not supported on this GPU architecture

Root cause

The requested precision path is not supported by the detected architecture and software combination. Model or backend support can be narrower than hardware capability. A mismatched or reused engine artifact can require a separate compatibility investigation.

Recommended fix

Record the GPU model, compute capability, TensorRT-LLM version, backend, model and complete quantization configuration.

How Denpex helps

Denpex matches TensorRT-LLM FP8 support depends on GPU architecture, backend and model across every rank in a distributed run and reports which rank failed first, so you act on the initiating node instead of the loudest one.

An FP8 architecture rejection means the selected build or runtime path cannot use the requested FP8 configuration on the detected target. It is not evidence that every FP8-related representation or every backend has the same support boundary.

TensorRT-LLM support is a combination of GPU architecture, software release, backend, model and precision path. NVIDIA's reviewed support matrix lists FP8 for Ada SM89 and Hopper SM90, and lists other precisions for Ampere SM80 and SM86. The same documentation warns that quantized formats are not implemented for every model. A performance-guide sentence uses a strict greater-than-8.9 threshold while naming Ada; use the explicit architecture table rather than repeating that inconsistent inequality.

• A requested FP8 path is rejected on the detected GPU
• A general hardware capability claim does not match the selected model or backend
• The build environment and deployment environment use different GPU architectures or software versions

• TensorRT-LLM builds or inference paths requesting FP8
• Ampere deployments requesting a native FP8 path not listed for SM80 or SM86
• Ada, Hopper and newer deployments whose selected model, backend or recipe has additional restrictions

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Read the visible GPU compute capability and installed package versions
• ✓Match the selected model and precision recipe to the installed release's support table
• ✓Capture the complete error and identify whether rejection occurs in conversion, engine build or inference
• ✓Compare build and deployment environments without assuming a cross-build workflow is supported

• The requested precision path is not supported by the detected architecture and software combination
• Model or backend support can be narrower than hardware capability
• A mismatched or reused engine artifact can require a separate compatibility investigation

ERROR: FP8 quantization is not supported on this GPU architecture

Use this text as a lookup key in logs and upstream issue trackers. It is not presented as a captured customer log. Confirm the cause from your own preceding events, versions, configuration and the cited references.

Selecting a supported architecture, model and precision path removes a compatibility mismatch. Measuring the resulting model separately avoids replacing an unsupported format with an unverified accuracy or performance claim.

# Read-only environment inventory. Run in the environment that reports the error.
python - <<'PY'
import importlib.metadata as metadata
import torch
print('PyTorch:', torch.__version__, 'CUDA build:', torch.version.cuda)
try:
    print('TensorRT-LLM:', metadata.version('tensorrt-llm'))
except metadata.PackageNotFoundError:
    print('TensorRT-LLM package metadata not found in this environment')
for index in range(torch.cuda.device_count()):
    print(index, torch.cuda.get_device_name(index), torch.cuda.get_device_capability(index))
PY

Adapt the snippet to your framework. The same pattern holds for PyTorch Lightning, Hugging Face Trainer, DeepSpeed, Megatron-LM, and vLLM training wrappers. Where the wrapper exposes a config flag (for examplelr_scheduler_type in Trainer), prefer the flag over the imperative API to keep the schedule declarative and reproducible.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.