MongoDB Info Cache Collection Queries Overloading Database During HuggingFace Hub Outage

The Hugging Face Hub suffered an extended 36-hour outage when repeated requests to space_info_cache and datasets_info_cache MongoDB collections overloaded the database cluster. A script repeatedly querying a specific Space listing URL triggered a cascade: increased MongoDB network I/O, replica lag, and ultimately complete database cluster failure. The outage required multiple waves of mitigation over two days.

MongoDB Info Cache Collection Queries Overloading Database During HuggingFace Hub Outage is a Infrastructure failure seen during ML training runs. The Hugging Face Hub suffered an extended 36-hour outage when repeated requests to space_info_cache and datasets_info_cache MongoDB collections overloaded the database cluster. A script repeatedly querying a specific Space listing URL triggered a cascade: increased MongoDB network I/O, replica lag, and ultimately complete database cluster failure. The outage required multiple waves of mitigation over two days. Common tags: Huggingface, Mongodb, Cache, Database Outage.

Repeated traffic to a specific listing endpoint caused continuous database queries on info_caches collection. Each request missed cache and triggered a full MongoDB query, creating a snowball effect as more requests queued. Collections lacked query-level timeouts or circuit breakers; slow queries accumulated and exhausted connection pool. Taken together, these mechanisms explain why the failure is reproducible, why it tends to surface on specific workloads or scales, and why generic mitigation attempts often fall short without addressing the underlying cause.

• MongoDB cluster becomes unresponsive under read load from info_cache collections
• Database network-out bandwidth saturates, causing replica lag across cluster
• Hub website slows to crawl or returns errors for all users

• MongoDB-backed web application with highly concurrent cache-miss traffic
• Kubernetes services where API server rate limiting can cascade to downstream databases
• Infrastructure with shared caches across multiple product surfaces (Spaces, Datasets, Models)

Use this checklist to test the hypothesis against a small reproduction. No single line proves the root cause, so preserve the preceding events and compare one variable at a time.

• ✓Reproduce the failure from a clean checkpoint/seed: the symptom must appear without warm-up state from a previous run.
• ✓Verified signal present: MongoDB metrics show _info_cache collections with 10x normal query volume and >5s p99 latency
• ✓Verified signal present: Database network out spikes to 5x baseline, correlating with traffic to specific Space URLs
• ✓Verified signal present: MongoDB primary node CPU at 100% with connection queue filling
• ✓A targeted fix from the "How to fix it" section eliminates or substantially reduces the symptom within one validation pass.

Select the traceback or open the failed terminal, then run Denpex locally to see the initiating rank, collateral failures, exact fix, and verification command without uploading the log.

• Repeated traffic to a specific listing endpoint caused continuous database queries on info_caches collection
• Each request missed cache and triggered a full MongoDB query, creating a snowball effect as more requests queued
• Collections lacked query-level timeouts or circuit breakers; slow queries accumulated and exhausted connection pool