Rate limits
Limits exist to keep the diagnosis engine fast for everyone — and they are designed so that data-center-scale telemetry never hits them. Exceeding a limit returns 429 with code: "RATE_LIMITED"; retry with backoff. Daily diagnosis quotas (a plan feature, not a rate limit) return code: "UPGRADE_REQUIRED".
Limits at a glance
General API (authenticated)20 requests / min / userDashboard reads, key management, alert config, and /api/diagnose share this budget.
General API (unauthenticated)10 requests / min / IPLogin, signup, and public endpoints.
Heartbeats — /api/ingest/heartbeat, /api/monitor/heartbeat(+/batch)Not rate-limitedA fleet legitimately beats often — 2,000 nodes at 30s is ~67 req/s for one tenant. Heartbeats shard across per-tenant Durable Objects and never touch the limiter. Use /api/monitor/heartbeat/batch to report a whole rack in one request.
Crash ingest — /api/ingest/crash30 distinct root causes / min / user · coalesced reports unlimitedCrash reports sharing a cluster and failure signature within a 5-minute window are coalesced into one diagnosis (see below), so the budget counts unique failures — never how many nodes reported them.
Mass-crash coalescing: why a 2,000-node crash never drops a diagnosis
When a single fabric event — a bad InfiniBand switch, a PDU trip — takes down a 16,384-GPU job, every node's agent reports the crash within seconds. Denpex groups reports that share a cluster and a failure signature (e.g. xid-79, nccl-timeout) inside a 5-minute window. Exactly one report runs the diagnosis; every other report returns the same diagnosisId with coalesced: true, links its job to it on the dashboard, and consumes no quota. You get one alert with the fix — not thousands of pages.
Grouping quality improves if you name your cluster: set DENPEX_CLUSTER=<name> in the agent's environment. Without it, Denpex falls back to the SLURM job id / MASTER_ADDR / Ray job id, which groups per-job rather than per-cluster.
Handling 429s
The shipped agent already retries with exponential backoff — you don't need to do anything. For custom integrations: back off for 30–60 seconds and retry. If you hit the crash-ingest budget, your fleet genuinely produced more than 30 distinct failure classes in one minute — contact support, because something remarkable is happening.