An inference scheduler deadlocks under sustained load and stops issuing work

A continuous-batching scheduler admits requests against a token budget it must not exceed. Under sustained concurrency the accounting can reach a state where no admitted request can advance and no new one can be admitted, so the server stops issuing work while remaining connected and apparently healthy.

A scheduling deadlock in a continuous-batching inference server where admitted requests collectively claim more of the token budget than it holds, leaving none able to progress and none able to release, so the server stops issuing work without failing.

Continuous batching trades simplicity for utilisation: requests join and leave a running batch, and a shared budget decides who is admitted. Any admitted request whose demand can grow turns that budget from a bound into a claim, and claims that cannot be satisfied or withdrawn are the definition of deadlock. High concurrency is what makes the interleaving that reaches it likely.

• Throughput reaches zero while the process stays up and keeps accepting connections
• It only appears above a certain request rate, so it never reproduces in a quiet environment
• An assertion about token accounting fires, which reads as an internal bug rather than saturation
• Restarting clears it until the load returns

• Continuous and in-flight batching schedulers that admit against a global token budget
• Deployments running near their configured maximum batched tokens rather than comfortably below it
• Speculative decoding and other features that make a request's token demand vary during its lifetime
• Long-context traffic, where a single request can consume a large share of the budget

Apply the following checklist to a small reproduction: each box below is a positive signal that you are looking at this exact failure rather than a sibling in the same taxonomy.

• ✓Check whether completions have stopped while the queue is non-empty; a wedged scheduler shows both at once, whereas an overloaded one still completes work slowly.
• ✓Correlate onset with request rate rather than with any deployment change. A threshold that appears above a particular concurrency and not below it indicates a state reachable only under contention.
• ✓Reduce the batched token budget and replay the same load. Surviving with a smaller ceiling confirms budget contention rather than a fault in any single request.

AssertionError: assert total_num_tokens <= self.max_num_tokens
[TRT-LLM] [E] Encountered an error in forward function
[TRT-LLM] [E] Error in event loop

Timestamps and exact values vary across runs, but the pattern. An info-level start, an early WARN, an ERROR carrying the symptom. Is the actual fingerprint you should alert on. The Denpex platform flags this combination automatically.

Operating below the ceiling means an admitted request's growth can be absorbed rather than contended, so the state is never reached. Reducing concurrency does the same from the other side by limiting how many claims exist at once. Neither improves peak throughput on paper, and both are what keep the measured throughput above zero.

// Typical pattern:
import torch.optim as optim
optimizer = optim.AdamW(model.parameters(), lr=3e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_training_steps)
for step in range(num_training_steps):
    optimizer.zero_grad()
    loss = model(batch)
    loss.backward()
    optimizer.step()
    scheduler.step()

Adapt the snippet to your framework. The same pattern holds for PyTorch Lightning, Hugging Face Trainer, DeepSpeed, Megatron-LM, and vLLM training wrappers. Where the wrapper exposes a config flag (for examplelr_scheduler_type in Trainer), prefer the flag over the imperative API to keep the schedule declarative and reproducible.