Platform and research-cluster workflows

Compare a GPU incident with its platform context.

A queued job, an interrupted worker and a failed checkpoint need different evidence. These researched examples connect public platform documentation with the logs that would make a useful incident handoff.

These are proposed evaluations, not customer endorsements or reports of private incidents. Every page links its primary sources and states the scope. Shared demos are illustrative; use approved logs from one incident you already resolved.

Each evaluation page includes the eligible free 30-day Team offer, up to 50 diagnoses a day, with no card or automatic subscription. Mailbox and organization checks still apply.

Researched incident comparisons

dstack
dstack run failed: spot interruption, provisioning or worker error?
Align the termination reason with executor and application logs. Review the phase that failed before changing the model or retrying the same resource request.
Verda
Verda Slinky GPU visibility: allocation or device failure?
Bring the Slurm job allocation and container context beside the first device-visibility error. A missing allocation and a failing GPU require different next checks.
Clarifai
Clarifai model deployment failed: build, events or runtime?
Compare the deployment events with the first model-worker exception. A failed build, unavailable deployment and failed prediction should not receive the same GPU remedy.
Runhouse Kubetorch
Kubetorch worker failure: Python exception or Kubernetes event?
Keep the remote traceback, pod lifecycle and host timeline together. Evaluate what a diagnosis adds to Kubetorch existing error propagation and recovery.
Aleph Alpha
Aleph Alpha Scaling: find the error before a training collective stalls.
Review the parallelism configuration and earliest rank exception together. Distinguish setup and memory evidence from a later collective timeout.
Black Forest Labs
FLUX loading failed: model access, initialization or CUDA memory?
Separate an unavailable weight download from a worker that loads the model and then fails. Keep model revision, dtype and the first runtime exception in one comparison.
Nous Research
Atropos rollout stalled: environment, inference or training failure?
Align the environment, inference-server and trainer timelines before treating missing rollouts as a GPU crash. Keep external API failures distinct from local runtime errors.
Wayve
Wayve model-training interruption: preserve the first worker failure.
Evaluate the infrastructure evidence from one resolved training run without vehicle recordings. Keep worker failures separate from model behavior and driving outcomes.
Princeton Research Computing
Princeton Slurm OOM: host memory or CUDA allocation?
Compare the requested RAM, measured process usage and first memory error. Avoid changing GPU batch size when the initiating evidence belongs to a Slurm host-memory limit.
Harvard FASRC
Harvard gpu_requeue: interrupted job or failed checkpoint recovery?
Keep scheduler restart history beside application output and checkpoint completion. A restarted batch script is not proof that training resumed safely.
Purdue RCAC Gautschi
Purdue Gautschi GPU job: idle, stalled or missing monitoring data?
Compare job phase and worker progress with the telemetry interval. Distinguish an application stall from delayed monitoring records before requesting hardware intervention.
Ohio Supercomputer Center
OSC out-of-memory jobs: allocated RAM or device-memory failure?
Preserve the job request, process termination and first runtime exception. Compare the actual memory boundary before a GPU workload is retried with unrelated settings.
PSC Bridges-2
Bridges-2 GPU job failed: allocation, launcher or worker?
Compare partition and effective GPU allocation with the worker launch. Review the first runtime error before treating every distributed exit as a network fault.
Mila
Mila checkpoint recovery: completed write or incomplete training state?
Keep save completion, interruption and the next load result in order. Review a known resolved run before deciding which artifact can support a safe continuation.
MIT Engaging
MIT Engaging GPU not visible: check the Slurm allocation first.
Compare the scheduler request, assigned compute node and first worker exception. Distinguish a missing GPU request from a device or runtime fault.
Berkeley Savio
Berkeley Savio QOSGrpGRES: queued allocation or failed GPU job?
Keep the pending reason, group entitlement and request separate from runtime failures. Only inspect worker errors after confirming the application started.
Juelich JUWELS
JUWELS GPU training stall: launch geometry or an earlier worker error?
Compare rank placement, assigned devices and the first exception from the selected JUWELS module. Keep device sharing and application parallelism distinct.
Yale YCRC
Yale training job failed: memory limit, disk quota or environment?
Bring the first error, allocation and affected path into a single review. Distinguish a host-memory kill from an unwritable checkpoint or environment setup failure.
University of Utah CHPC
Utah CHPC GPU job: allocation, task affinity or worker failure?
Align the requested GPU resources with launch placement and the first runtime exception. Keep host-memory requirements distinct from accelerator memory.
Columbia Research Computing
Columbia burst job requeued: policy interruption or runtime failure?
Compare the queue decision with the worker and checkpoint timeline. A requeued batch script is not evidence that the application recovered its previous state.
Vast.ai
Vast.ai instance stopped: scheduling, account limit or worker failure?
Compare the instance lifecycle and first workload exception. Avoid interpreting a capacity or account condition as evidence of a failed GPU.
Lambda
Lambda cloud GPU unavailable after upgrade: module build or workload error?
Keep the base image and kernel history beside the retained driver-build output. Distinguish a broken startup dependency from a model that exhausts memory after launch.
Beam
Beam task in RETRY: recovered attempt or an underlying worker failure?
Follow task identity and container attempts before diagnosing a stalled request. Compare the first application exception with the actual task lifecycle.
Thunder Compute
Thunder Compute function not implemented: compatibility or GPU failure?
Preserve the failing operation and runtime context before changing memory settings. Compare the provider compatibility boundary with the actual exception.
Hyperstack
Hyperstack Docker cannot select a GPU driver: container runtime or CUDA?
Compare the chosen image, Docker runtime and first container error. A container that cannot request a GPU is different from a model that starts and then fails.
CUDO Compute
CUDO workload failed: boot disk, attached volume or model process?
Keep the failing path and storage attachment alongside the workload exception. A machine restart and a complete recovery of training state are different outcomes.
San Francisco Compute
SF Compute cancelled order or failed training worker?
Keep the pool allocation window and instance lifecycle beside application logs. A capacity order that did not fill is not evidence of a GPU runtime failure.
GMI Cloud
GMI Cloud worker failed: managed cluster, container or bare metal?
Identify the compute product and first runtime exception before escalating a serving or training incident. Keep API symptoms separate from evidence available at the actual worker.
Genesis Cloud
Genesis Cloud instance active but GPU setup still running?
Compare instance readiness, startup-script completion and the first CUDA initialization error. Avoid diagnosing hardware before the documented configuration phase has finished.
Exoscale
Exoscale GPU incident: virtual-machine worker or managed inference?
Establish the product boundary before collecting the first model or host exception. An API failure alone cannot establish what happened on a GPU.

More company evaluation pages

Earlier outreach evaluations remain available. Their proposed workflow fit is separate from the reviewed search comparisons above.