For windborne systems infrastructure and model engineering teams

WeatherMesh training memory failure: host offload or GPU allocation?

Compare host-memory and device-memory evidence around the failed backward pass. Distinguish activation recomputation from a saved recovery checkpoint.

Replay one resolved incident Try 30 days free

Free 30-day Team evaluation. No card or automatic subscription. Work email verification required.

Why this workflow may fit WindBorne Systems

WindBorne describes WeatherMesh-2 training with activation checkpointing and CPU memory offload. Current documentation covers later WeatherMesh releases, so this historical example is not a description of the full current stack.

Read the official WindBorne Systems workflow reference Distinguish current WeatherMesh products from the historical example

Evaluation scope: The scope is the published WeatherMesh-2 example, not a claim about current WeatherMesh-6 hardware. Denpex cannot validate forecast accuracy or certify a saved checkpoint from activation-checkpointing settings.

Prepared from public materials. This is a proposed evaluation, not a report of an incident at WindBorne Systems or an endorsement.

Selectable demo logs are shared examples, not logs from WindBorne Systems. Upload a shareable incident from your own workflow to evaluate the fit.

Live diagnosis, no signup

Replay a resolved WindBorne Systems workflow incident

Upload your log file. Review the selected evidence and masking, then run diagnosis. Keep the original log for any follow-up questions.

Before uploading, review cloud data handling and local options.

training_logs.txt

No log to hand? Try one:

Upload or drop a plain-text log

Upload your log file. Large files are processed in your browser. Only selected evidence is submitted when you run diagnosis.

3 free diagnoses/day

Try Denpex for 30 days. FREE.

No credit card. No automatic subscription. See whether Denpex helps with an incident you already understand.

Start your free 30-day Team evaluation

Up to 50 diagnoses a day for 30 days. Verify your work email to activate an eligible company evaluation, then replay a resolved incident and compare the evidence and next steps with your team's confirmed cause.

What a useful evaluation should establish

Compare one incident your team has already resolved with the first failure, quoted evidence and remaining questions. Begin with this question: Was offloaded activation memory exhausted on the host? Only assign a process, device or recovery state when the submitted evidence establishes it.

Was offloaded activation memory exhausted on the host?

Evidence to compare: Kernel kill evidence, affected process and host-memory usage around the failed step.

CPU offload changes where tensors consume memory. A host kill can stop training while device memory remains available, requiring a different next check from CUDA allocation failure.

Did a transfer or backward operation fail first?

Evidence to compare: The first retained exception and the actual device-transfer timeline.

A later lost-worker symptom does not identify the failed transfer. Preserve the earlier operation rather than blaming every offload incident on bandwidth.

Is recovery based on a saved model artifact?

Evidence to compare: Training-checkpoint writer completion and the selected resume artifact.

Activation checkpointing recomputes tensors; it is not a recovery-file guarantee. Inspect the saved model state independently before suggesting a restart.

Use Denpex alongside your existing telemetry tools to investigate submitted incident evidence. With monitoring configured on a supported plan, detected failures can trigger alerts and an automatic diagnosis from the captured evidence.

Inside Denpex

See the evidence, not just the claim

Follow the causal chain, inspect the quoted evidence and prepare the next handoff. These actual product captures use founder-account examples and the VS Code extension, with the source identified under each image. Open a capture to inspect it at full size.