vLLM rejects a request because the prompt plus completion exceeds the served context length

vLLM returns an error for a request whose prompt tokens plus requested completion tokens exceed the context window the server was started with. The served window is not always the one advertised on the model card: it is taken from the model config and can be capped further when KV cache memory is short.

A request-time rejection in which vLLM refuses a completion because the prompt tokens plus the requested completion tokens exceed the context window the server is actually serving, which is often shorter than the window the model card advertises.

Two independent things shorten the served window. The model config's max_position_embeddings, not the model card, is what vLLM reads, and repositories exist where the card advertises a long context while the config declares a short one. Separately, the engine caps the window at startup when the KV cache cannot hold the full length within the memory budget. Either produces a served window smaller than expected, and every request is then judged against that smaller number, with the completion counted alongside the prompt.

• A request is rejected although the model card advertises a much longer context
• The same prompt succeeds against a different deployment of the same model
• Shortening the prompt clears the error even though the completion length was never changed
• Raising the served context length triggers a different startup error about KV cache capacity

• vLLM OpenAI-compatible server endpoints, which reproduce the OpenAI error wording
• Models whose config.json carries a max_position_embeddings smaller than the advertised context
• Long-context deployments where KV cache memory, not the model, is the binding limit
• Clients that hardcode max_tokens instead of deriving it from the remaining budget

Apply the following checklist to a small reproduction: each box below is a positive signal that you are looking at this exact failure rather than a sibling in the same taxonomy.

• ✓Add the prompt token count to the requested max_tokens and compare the sum against the number in the error; the error reports both halves separately for this purpose.
• ✓Read max_position_embeddings from the model's config.json and compare it against the length the engine logged at startup.
• ✓Check the startup log for a warning that the sequence length was capped by KV cache capacity, which explains a served window shorter than the one requested.

ValueError: This model's maximum context length is 4096 tokens, however you requested 5120 tokens (4608 in your prompt; 512 for the completion). Reduce max_model_len or the prompt length.
WARNING: The model's max seq len (32768) is larger than the maximum number of tokens that can be stored in KV cache (15424).

Timestamps and exact values vary across runs, but the pattern. An info-level start, an early WARN, an ERROR carrying the symptom. Is the actual fingerprint you should alert on. The Denpex platform flags this combination automatically.

Setting --max-model-len explicitly removes the inference step entirely, so the served window becomes a value you chose rather than one derived from a config file you may not have read. Deriving max_tokens on the client from the remaining budget fixes the other half, because the limit applies to the sum and a hardcoded completion length can exceed it no matter how short the prompt is.

// Typical pattern:
import torch.optim as optim
optimizer = optim.AdamW(model.parameters(), lr=3e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_training_steps)
for step in range(num_training_steps):
    optimizer.zero_grad()
    loss = model(batch)
    loss.backward()
    optimizer.step()
    scheduler.step()

Adapt the snippet to your framework. The same pattern holds for PyTorch Lightning, Hugging Face Trainer, DeepSpeed, Megatron-LM, and vLLM training wrappers. Where the wrapper exposes a config flag (for examplelr_scheduler_type in Trainer), prefer the flag over the imperative API to keep the schedule declarative and reproducible.