Render uses Cloudflare to offer DDoS protection to all web services and as a CDN provider for static sites. A configuration change made by Cloudflare triggered a bug in Cloudflare systems, causing all Render web services and static sites to respond with a
500 HTTP error between 2023-08-11 4:00 UTC and 2023-08-11 5:26 UTC. Service was restored after Cloudflare reverted the change.
Between 4:00 UTC and 5:26 UTC, all HTTP traffic to web services and static sites hosted on Render received a Cloudflare Internal Server Error (HTTP error code
500). Traffic to render.com and dashboard.render.com was also impacted.
Render uses Cloudflare for DDoS protection for all HTTP services, and as a CDN for static sites. All requests to Render are therefore proxied through the closest Cloudflare data center. At 2023-08-11 4:00 UTC, Cloudflare added a rule to place an interstitial page in front of a specific Render-hosted website in response to a verified report of phishing. The application of this rule triggered a bug in the Cloudflare request processing engine, which resulted in all Render services, our dashboard, and render.com returning HTTP
Render's automated alerting brought the issue to our attention at 2023-08-11 4:01 UTC, and we responded by escalating the issue to Cloudflare and opening a public incident at 4:05. Given the severe and extensive impact of the outage, we coordinated with several members of the Cloudflare team while also investigating several options to restore services as quickly as possible. One of these investigations explored ways to temporarily circumvent Cloudflare's network to serve requests directly from Render's origin servers. We were able to deploy a working solution for dashboard.render.com at 5:03 UTC; this solution was manual and specific to the dashboard.render.com subdomain. We also began investigating ways to roll this solution out more broadly with customer assistance that would have involved updating their DNS records. Cloudflare identified the issue and rolled back the triggering rule at 5:26 UTC.