For nous research infrastructure and model engineering teams
Atropos rollout stalled: environment, inference or training failure?
Align the environment, inference-server and trainer timelines before treating missing rollouts as a GPU crash. Keep external API failures distinct from local runtime errors.
Free 30-day Team evaluation. No card or automatic subscription. Work email verification required.
Replay a resolved Nous Research workflow incident
Upload your log file. Review the selected evidence and masking, then run diagnosis. Keep the original log for any follow-up questions.
Before uploading, review cloud data handling and local options.
Upload your log file. Large files are processed in your browser. Only selected evidence is submitted when you run diagnosis.
Try Denpex for 30 days. FREE.
No credit card. No automatic subscription. See whether Denpex helps with an incident you already understand.
Start your free 30-day Team evaluation
Up to 50 diagnoses a day for 30 days. Verify your work email to activate an eligible company evaluation, then replay a resolved incident and compare the evidence and next steps with your team's confirmed cause.
What a useful evaluation should establish
Compare the first failure, quoted evidence and unresolved questions with an incident your team has already resolved. Begin with this question: Did rollout generation reach the inference backend? Only attribute a process, rank, device or recovery state when the submitted evidence establishes it.
Did rollout generation reach the inference backend?
Evidence to compare: Environment logs, request timing and selected backend type.
A missing response may come from credentials, transport or the environment itself. First establish the phase that was reached; do not assume a local GPU exists.
What stopped the trainer from making progress?
Evidence to compare: Rollout arrival history, trainer output and the earliest backend exception.
A waiting trainer can be downstream of a stopped environment. The example trainer and the inference server need separate process identities.
Was a local process actually out of memory?
Evidence to compare: Host termination events or the actual CUDA allocation exception.
Local host memory and accelerator memory are separate resources. Describe the failing process and supported evidence before changing batch or server settings.
Use Denpex alongside your existing telemetry tools to investigate submitted incident evidence. With monitoring configured on a supported plan, detected failures can trigger alerts and an automatic diagnosis from the captured evidence.
Inside Denpex
See the evidence, not just the claim
Follow the causal chain, inspect the quoted evidence and prepare the next handoff. These actual product captures use founder-account examples and the VS Code extension, with the source identified under each image. Open a capture to inspect it at full size.
Follow the failure through the cascade
Prepare the hardware escalation
Ask for the evidence that is missing
Keep the investigation in the editor
