Treating all memory numbers as interchangeable
Host memory, device memory and allocator-reserved memory are different measurements. Record which domain each observation describes.
Operator task guide
Distinguish video-model CUDA OOM from host memory failure
Increasing frame count, resolution or batch size can change resource requirements, but the configuration change alone does not identify the failed memory domain. Preserve the actual allocation exception or host process-termination evidence.
PyTorch's allocated and reserved memory measurements describe different parts of its CUDA allocator state. A device-memory snapshot does not prove host memory exhaustion, and a generic killed-process message does not prove CUDA OOM.
Step-by-step method
- 1
Preserve the failed operation and input metadata
Record batch size, frame count, spatial dimensions, precision and the first exception. Keep the model revision and stage of execution, such as loading, forward pass or checkpoint save. Use metadata rather than uploading video or image content.
- 2
Identify the memory domain with evidence
A CUDA allocation exception supports a device-memory investigation. A kernel OOM-killer record or container OOMKilled status supports a host or container-memory investigation. Collect the process identity and relevant host event before treating a later peer timeout as the cause.
- 3
Interpret allocator observations in context
When available, retain PyTorch allocated/reserved memory and a memory summary around the failure. Check other device processes and the runtime version. Do not conclude fragmentation from reserved memory alone, or treat empty_cache as a way to free live tensors.
- 4
Verify one approved change at a time
Compare the explanation with a resolved incident. A workload owner can test a smaller input or batch within the model's documented constraints, while a host-memory investigation may instead require data-worker or container-limit checks. Verify completion and application quality separately.
Match the remedy to the failed memory domain
| Signal | What it means | Next action |
|---|---|---|
| Explicit CUDA allocation exception | A device allocation failed. | Collect allocator context, model/input settings and device process state. |
| Host OOM-killer or OOMKilled event | Host or container memory limits are implicated. | Identify the killed process and its memory domain. |
| Peer timeout after a worker disappears | The initiating failure is not established by the timeout. | Retrieve the missing worker and host evidence. |
Evidence checklist
- First exception and safe input dimensions
- Model revision, precision and execution stage
- Allocator summary or host kill evidence with process mapping
- Confirmed resolution and approved rerun outcome
Common mistakes
Changing context-cache settings on an unrelated engine
A text-model KV-cache example does not establish the memory controls for a visual or video runtime.
Frequently asked questions
Does empty_cache free the model's live tensors?
No. PyTorch documents empty_cache for unused cached allocations. It does not free memory occupied by live tensors.
Does a higher frame count prove the cause?
No. It is relevant configuration evidence, but the actual allocation or process-termination event must identify the failure.
Apply the method to your incident
Use the three free diagnoses to review your error and relevant evidence. Keep reference guidance separate from the cause and recovery status of your own workload.
Diagnose your incident