Uploading research data to explain an infrastructure event
Start with approved operational logs and safe metadata. Review paths and identifiers as well as the log body before cloud submission.
Operator task guide
Investigate the data-worker exit before the rank timeout
A scientific ML run can stop progressing when an input worker exits before the training ranks report a collective timeout. First collect that worker's exception or host process-termination event. The waiting peer's timeout does not identify why the input worker failed.
Start with operational metadata: runtime version, safe dimensions, worker configuration and process identities. Biological sequences, molecular structures, patient records and experimental measurements usually need not be part of the initial infrastructure replay.
Step-by-step method
- 1
Identify the input worker and first error
Preserve the first traceback, process identity and parent training rank. Record whether the process exited, was killed, or remained alive without producing input. Keep original per-process ordering and document clock uncertainty.
- 2
Record the data-loading configuration
Retain num_workers, prefetch settings, batch metadata and the relevant dataset/loader implementation version. PyTorch documents that multi-process loading can increase host memory use. A worker-count change is context, not proof of the incident cause.
- 3
Collect the host or container discriminator
Match a killed-process message to host OOM records, container termination status or other relevant operating-system evidence. If only a generic worker exit is retained, request the missing discriminator before calling it a host-memory or CUDA failure.
- 4
Review a small approved reproduction
Have the workload owner test the suspected input path with approved synthetic or non-sensitive data. A single-process loading run can provide clearer traces during debugging, but may change timing and must not be treated as a production performance result. Verify the resolved cause before expanding the replay.
Keep input, host and accelerator failures distinct
| Signal | What it means | Next action |
|---|---|---|
| Worker traceback before peer timeout | An earlier input-path failure is retained. | Investigate that exception and its process context first. |
| Matched host OOM-killer record | Host memory killed an identified process. | Review loader memory and container limits with the owner. |
| Worker vanished, no matching host event | The termination cause remains unresolved. | Collect process, container and host evidence before choosing a remedy. |
Evidence checklist
- First worker exception, process and parent-rank mapping
- Safe batch/shape metadata and data-loading settings
- Host or container termination evidence
- Known resolution and a bounded reproduction outcome
Common mistakes
Calling a successful rerun a scientific validation
Runtime completion does not establish model quality, experimental reproducibility or biological validity.
Frequently asked questions
Does a DataLoader exit prove host OOM?
No. Match the process to host or container termination evidence. A traceback may instead identify a different input-path error.
Can I begin without proprietary training data?
Yes. Start with approved runtime logs, process identities and safe configuration metadata. Scientific data may require a separate controlled investigation.
Apply the method to your incident
Use the three free diagnoses to review your error and relevant evidence. Keep reference guidance separate from the cause and recovery status of your own workload.
Diagnose your incident