vLLM returns 500 on every request after the EngineCore process dies

The vLLM V1 API server survives while the separate EngineCore subprocess that owns the model does not. Once that child is gone the server keeps accepting connections and answers every one of them with an internal error until it is restarted, so an outage presents as a healthy-looking process serving nothing.

Symptom

ERROR: Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.

Root cause

EngineDeadError is a bereavement notice, not a diagnosis. The API server process discovers the child is gone when it next awaits output, and reports that discovery. The exception that actually killed the engine was raised inside the EngineCore subprocess and printed to its own stream earlier, which is what the message means by see stack trace above.

Recommended fix

VLLM_LOGGING_LEVEL=DEBUG

How Denpex helps

Denpex matches vLLM returns 500 on every request after the EngineCore process dies across every rank in a distributed run and reports which rank failed first, so you act on the initiating node instead of the loudest one.

An outage mode specific to vLLM's split-process design, in which the model-owning EngineCore subprocess terminates while the HTTP server that fronts it continues running and answering requests with internal errors.

vLLM V1 puts the model in its own process so the HTTP layer stays responsive. The consequence is that the two can fail independently. When the child dies, the parent finds out only when it next awaits output, and all it can honestly say is that the engine is gone. Nothing in that message distinguishes a missing CUDA kernel from a stalled worker from a subprocess reaped by the kernel, because the parent never saw any of them happen.

• Every completion request returns HTTP 500 while the container stays up and the port stays open
• A liveness probe that only checks the port keeps the dead instance in the load balancer
• The error the server reports names no cause, so there is nothing obvious to act on
• Restarting clears it for a while and it returns under the same conditions

• vLLM V1, where the API server and EngineCore are separate processes
• Containers whose health check tests only TCP reachability rather than a real completion
• Deployments using VLLM_WORKER_MULTIPROC_METHOD=spawn
• Any autoscaled fleet, where one dead replica silently absorbs its share of traffic

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Search the log backwards from the EngineDeadError for the first traceback attributed to a worker or engine-core process; that is the cause and it will name a device, a memory limit or a timeout.
• ✓Run dmesg -T | tail and look for an out-of-memory kill naming a Python process at the time of the failure. A kill leaves no application traceback.
• ✓Issue a one-token completion against the instance. If it returns 500 while the port accepts connections, the engine is dead and the process needs restarting rather than investigating live.

• EngineDeadError is a bereavement notice, not a diagnosis. The API server process discovers the child is gone when it next awaits output, and reports that discovery. The exception that actually killed the engine was raised inside the EngineCore subprocess and printed to its own stream earlier, which is what the message means by see stack trace above.
• The child dies for reasons that share nothing but their effect. Reported causes include kernels that do not exist for the device, where a quantized build raises CUDA error no kernel image is available for execution on the device on the first request; an execute_model remote call timing out against a stalled worker; the kernel out-of-memory killer reaping the subprocess under load; and shared-memory limits breaking worker startup.
• The failure is durable because nothing restarts the engine. The parent has no model of its own to fall back on, so it stays alive serving errors, and a port-based liveness probe cannot tell that apart from a working server.

ERROR: Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
INFO:     10.0.4.19:52344 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error

Use this text as a lookup key in logs and upstream issue trackers. It is not presented as a captured customer log. Confirm the cause from your own preceding events, versions, configuration and the cited references.

Reading upward finds the exception raised where the failure actually occurred, which names a device, a limit or a timeout and therefore points at a specific change. Replacing the liveness probe fixes the second half of the problem: it converts a permanent silent outage into a restart, so a cause you have not diagnosed yet still stops costing traffic while you diagnose it.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

vLLM failures cross request, model, KV cache, worker, kernel and host boundaries. The hub compares common literal signatures and the first control that separates their causal owners.