Skip to content
Free GPU tools

LLM inference VRAM and KV cache calculator

Estimate how much GPU memory your model weights and KV cache need. Compare context length, precision and tensor parallelism, then use the results to plan a vLLM or SGLang benchmark.

Cluster & Model Setup

70B dense model with GQA.

1 GPU4 GPUs8 GPUs
2K128K
Serving Feasibility Status
Cluster Configuration Viable
Max Concurrent Requests
28
Weights VRAM per GPU
32.88 GB
Across 2 GPUs (TP)
KV Cache per Request
1.25 GB
At 16,384 tokens
Aggregate Throughput
837 tok/s
29.9 tok/s per user stream

Per GPU Memory Allocation Breakdown

Weights
32.88 GB
KV Cache Pool
36.12 GB
CUDA Context
1.00 GB
Headroom Buffer
2.00 GB

Serving command templates

Set MODEL_PATH to your checkpoint. For FP8 or INT4 weight estimates, use a checkpoint in the selected format that your serving engine supports. Verify cache dtype support on your engine and GPU before running the command.

vLLM Serving Command
vllm serve "$MODEL_PATH" --tensor-parallel-size 2 --gpu-memory-utilization 0.9 --max-model-len 16384 --kv-cache-dtype fp8 --dtype auto
SGLang Serving Command
python3 -m sglang.launch_server --model-path "$MODEL_PATH" --tp 2 --mem-fraction-static 0.9 --context-length 16384 --kv-cache-dtype fp8_e4m3 --dtype auto

How context length affects concurrency

Longer sequences consume more KV cache per request. Compare your typical and longest sequences, then benchmark with representative prompt lengths and batch sizes to measure actual throughput and latency.

For serving configuration, read the vLLM context length and KV cache guide. For training memory and runtime estimates, use the LLM training calculator.