Skip to content
API

Rate limits

Limits exist to keep the diagnosis engine fast for everyone. And they are designed so that data-center-scale telemetry never hits them. Exceeding a limit returns 429 with code: "RATE_LIMITED"; retry with backoff. Daily diagnosis quotas (a plan feature, not a rate limit) return code: "UPGRADE_REQUIRED".

Limits at a glance

General API (authenticated)20 requests / min / user

Dashboard reads, key management, alert config, and /api/diagnose share this budget.

General API (unauthenticated)10 requests / min / IP

Login, signup, and public endpoints.

Heartbeats, /api/ingest/heartbeat, /api/monitor/heartbeat(+/batch)Not rate-limited

A fleet legitimately beats often. 2,000 nodes at 30s is ~67 req/s for one tenant. Heartbeats shard across per-tenant Durable Objects and never touch the limiter. Use /api/monitor/heartbeat/batch to report a whole rack in one request.

Crash ingest, /api/ingest/crash30 distinct root causes / min / user · coalesced reports unlimited

Crash reports sharing a cluster and failure signature within a 5-minute window are coalesced into one diagnosis (see below), so the budget counts unique failures. Never how many nodes reported them.

Mass-crash coalescing: why a 2,000-node crash never drops a diagnosis

When a single fabric event. A bad InfiniBand switch, a PDU trip. Takes down a 16,384-GPU job, every node's agent reports the crash within seconds. Denpex groups reports that share a cluster and a failure signature (e.g. xid-79, nccl-timeout) inside a 5-minute window. Exactly one report runs the diagnosis; every other report returns the same diagnosisId with coalesced: true, links its job to it on the dashboard, and consumes no quota. You get one alert with the fix. Not thousands of pages.

# Follower response during a mass-crash event
{ "success": true, "diagnosisId": "a1b2…", "coalesced": true, "crashCount": 1998, "fix": { … } }

Grouping quality improves if you name your cluster: set DENPEX_CLUSTER=<name> in the agent's environment. Without it, Denpex falls back to the SLURM job id / MASTER_ADDR / Ray job id, which groups per-job rather than per-cluster.

Handling 429s

The shipped agent already retries with exponential backoff. You don't need to do anything. For custom integrations: back off for 30 to 60 seconds and retry. If you hit the crash-ingest budget, your fleet genuinely produced more than 30 distinct failure classes in one minute, contact support, because something remarkable is happening.