Sizing multimodal pipelines using text-only parameter math
High-resolution vision encoders and stacked image tokens expand activation peaks even when quantized LLM weights are small.
Operator task guide
Multimodal serving in vLLM-Omni adds image, audio, and video encoder buffers plus multimodal token embeddings on top of static LLM weights and paged KV-cache blocks.
NVFP4 on Blackwell (sm_100) and aggressive KV-cache compression methods like TurboQuant or FP8 reduce weight and KV footprints sharply, so remaining OOMs usually stem from encoder profiling spikes, un-tuned gpu_memory_utilization, or missing kernel support.
Reviewed . Reference guidance is not a diagnosis of your workload.
Capture startup free VRAM alongside the multimodal profiler run so you can see whether OOM occurs during vision/audio encoder warmup or long-context KV allocation.
nvidia-smi --query-gpu=index,memory.used,memory.free,memory.total --format=csv
python -c "import torch; print(torch.cuda.get_device_capability(0))"Confirm whether your GPU architecture natively accelerates FP4 block scaling (Blackwell B200 sm_100) versus falling back to software dequantization into BF16 buffers.
rg -n -i 'nvfp4|fp8|kv_cache|Free memory on device|EngineCore' vllm-server.logRight-size max-num-batched-tokens and limit multimodal items per prompt before lowering gpu_memory_utilization blindly.
| Signal | What it means | Next action |
|---|---|---|
| Startup OOM during encoder warmup in vLLM-Omni | Vision/audio encoder temporary activations exceed unreserved VRAM headroom | Lower --limit-mm-per-prompt or leave headroom in --gpu-memory-utilization (for example 0.85 to 0.88). |
| NVFP4 weight load uses double expected VRAM | Non-Blackwell GPU fell back to BF16/FP8 weight dequantization or un-fused buffers | Verify sm_100 compute capability and check quantization config kernels before benchmarking. |
| TurboQuant or FP8 KV-cache still hits cache block exhaustion | max_model_len exceeds remaining HBM at configured batch concurrency | Compute exact capacity in /tools/vram-calculator and lower --max-num-seqs or --max-model-len. |
High-resolution vision encoders and stacked image tokens expand activation peaks even when quantized LLM weights are small.
Using NVFP4 checkpoints on Hopper or Ampere hardware without native FP4 instructions can silently inflate latency or memory.
Multimodal models allocate temporary activation buffers for vision and audio towers alongside long visual token sequences during vLLM warmup profiling. Tightening media limits per prompt and reserving ~12% headroom avoids encoder OOMs.
Compressed KV caches shrink context memory dramatically per concurrent sequence, allowing higher batch sizes on a single GPU, but total model weight size and HBM bandwidth still determine when Tensor Parallelism is required.
Use the three free diagnoses to review your error and relevant evidence. Keep reference guidance separate from the cause and recovery status of your own workload.
Diagnose your incident