For asu research computing infrastructure and model engineering teams
ASU Slurm job: host memory, GPU allocation or architecture mismatch?
Compare the job request and actual allocation with its first exception. Distinguish host-memory enforcement from CUDA allocation before changing GPU resources.
Free 30-day Team evaluation. No card or automatic subscription. Work email verification required.
Replay a resolved ASU Research Computing workflow incident
Upload your log file. Review the selected evidence and masking, then run diagnosis. Keep the original log for any follow-up questions.
Before uploading, review cloud data handling and local options.
Upload your log file. Large files are processed in your browser. Only selected evidence is submitted when you run diagnosis.
Try Denpex for 30 days. FREE.
No credit card. No automatic subscription. See whether Denpex helps with an incident you already understand.
Start your free 30-day Team evaluation
Up to 50 diagnoses a day for 30 days. Verify your work email to activate an eligible company evaluation, then replay a resolved incident and compare the evidence and next steps with your team's confirmed cause.
What a useful evaluation should establish
Compare one incident your team has already resolved with the first failure, quoted evidence and remaining questions. Begin with this question: Did host memory or device memory run out? Only assign a process, device or recovery state when the submitted evidence establishes it.
Did host memory or device memory run out?
Evidence to compare: Submitted memory request, Slurm accounting and the actual allocation exception.
CPU memory and GPU memory are different resources. A scheduler or kernel host-memory kill needs different evidence from a device allocator failure.
Was the requested accelerator available to this process?
Evidence to compare: GPU type request, allocated node and the process-visible device mapping.
A pending job or a process outside the allocation is not a failed GPU. Establish where the process ran before interpreting device visibility.
Does the environment match the allocated architecture?
Evidence to compare: Allocated CPU architecture, accelerator vendor and container or module build.
A heterogeneous cluster can expose incompatible binaries or extensions. Record the actual architecture rather than applying one CUDA diagnosis to every node family.
Use Denpex alongside your existing telemetry tools to investigate submitted incident evidence. With monitoring configured on a supported plan, detected failures can trigger alerts and an automatic diagnosis from the captured evidence.
Inside Denpex
See the evidence, not just the claim
Follow the causal chain, inspect the quoted evidence and prepare the next handoff. These actual product captures use founder-account examples and the VS Code extension, with the source identified under each image. Open a capture to inspect it at full size.
Follow the failure through the cascade
Prepare the hardware escalation
Ask for the evidence that is missing
Keep the investigation in the editor
