An inference scheduler deadlocks under sustained load and stops issuing work

A continuous-batching scheduler admits requests against a token budget it must not exceed. Under sustained concurrency the accounting can reach a state where no admitted request can advance and no new one can be admitted, so the server stops issuing work while remaining connected and apparently healthy.

Symptom

assert total_num_tokens <= self.max_num_tokens, (

Root cause

Admission is a resource-allocation decision made repeatedly against a shared budget. If a request's demand can grow after it has been admitted, the scheduler can reach a state where the total exceeds what it promised itself, with every admitted request needing more budget to finish and none able to release any. That is a deadlock in the classical sense rather than an overload: no participant can proceed and none will yield.

Recommended fix

Reduce the maximum batched token budget so the scheduler operates with headroom rather than at its ceiling, which removes the tight contention the state requires.

How Denpex helps

Denpex matches An inference scheduler deadlocks under sustained load and stops issuing work across every rank in a distributed run and reports which rank failed first, so you act on the initiating node instead of the loudest one.

A scheduling deadlock in a continuous-batching inference server where admitted requests collectively claim more of the token budget than it holds, leaving none able to progress and none able to release, so the server stops issuing work without failing.

Continuous batching trades simplicity for utilisation: requests join and leave a running batch, and a shared budget decides who is admitted. Any admitted request whose demand can grow turns that budget from a bound into a claim, and claims that cannot be satisfied or withdrawn are the definition of deadlock. High concurrency is what makes the interleaving that reaches it likely.

• Throughput reaches zero while the process stays up and keeps accepting connections
• It only appears above a certain request rate, so it never reproduces in a quiet environment
• An assertion about token accounting fires, which reads as an internal bug rather than saturation
• Restarting clears it until the load returns

• Continuous and in-flight batching schedulers that admit against a global token budget
• Deployments running near their configured maximum batched tokens rather than comfortably below it
• Speculative decoding and other features that make a request's token demand vary during its lifetime
• Long-context traffic, where a single request can consume a large share of the budget

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Compare the reported total against the configured maximum. An overshoot of a few tokens points at the admission arithmetic rather than at a request that is genuinely too large.
• ✓Raise the configured maximum and re-run the same traffic. Failing again at the new ceiling confirms the accounting rather than the value.
• ✓Turn partial reuse off. Reaching the assertion sooner rather than later shows it is not the cause and rules out one variable.
• ✓Check whether completions have stopped while the queue is non-empty; a wedged scheduler shows both at once, whereas an overloaded one still completes work slowly.
• ✓Correlate onset with request rate rather than with any deployment change. A threshold that appears above a particular concurrency and not below it indicates a state reachable only under contention.
• ✓Reduce the batched token budget and replay the same load. Surviving with a smaller ceiling confirms budget contention rather than a fault in any single request.

• Admission is a resource-allocation decision made repeatedly against a shared budget. If a request's demand can grow after it has been admitted, the scheduler can reach a state where the total exceeds what it promised itself, with every admitted request needing more budget to finish and none able to release any.
• That is a deadlock in the classical sense rather than an overload: no participant can proceed and none will yield. It is why the server neither completes work nor sheds it, and why nothing times out on its own.
• It is load-dependent because the state requires a particular interleaving of admissions and completions. At low concurrency the budget is never contended tightly enough to reach it, which is why it survives testing and appears in production.

    assert total_num_tokens <= self.max_num_tokens, (
[TRT-LLM] [E] Encountered an error in forward function:
[TRT-LLM] [E] Error in event loop: Sampling failed
total_num_tokens=4118 > 4096
  File ".../model_engine.py", line 2678
AssertionError: assert total_num_tokens <= self.max_num_tokens
[TRT-LLM] [E] Encountered an error in forward function
[TRT-LLM] [E] Error in event loop

Use this text as a lookup key in logs and upstream issue trackers. It is not presented as a captured customer log. Confirm the cause from your own preceding events, versions, configuration and the cited references.

Operating below the ceiling means an admitted request's growth can be absorbed rather than contended, so the state is never reached. Reducing concurrency does the same from the other side by limiting how many claims exist at once. Neither improves peak throughput on paper, and both are what keep the measured throughput above zero.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.