RCA

Summary

On Thursday November 9th, multiple databases were restarted across all regions due to unexpected restarts triggered by a code change. This code change caused databases to be unavailable for up to 8 minutes. A concurrent but unrelated bug caused some affected databases in the Ohio region to be down for up to 37 minutes.

Incident Timeline

Impact

This incident affected a subset of Point-In-Time Recovery-enabled Postgres instances which experienced an unexpected restart during this time. The event resolved itself between 3 and 8 minutes in each cluster, except for the Ohio region.

In the Ohio region, we were unable to scale up to handle the movement of databases. This extended downtime. We rolled back a separate change to resolve the issues with scale up. All affected databases became available again within 37 minutes. 2 Postgres instances in the Ohio region remained unavailable and had to be manually returned to service by an engineer.

Root Cause

This downtime was triggered by a code deploy of our database management system. The code change created an errant configuration change to specific database configurations which caused them to unexpectedly restart to pick up the new configuration.

Mitigations

• We will improve monitoring and alerting to decrease our time to detection.
• We will add additional guard rails to our database management system to prevent unintentional restarts, including those due to configuration changes. We have already developed the feature and are planning its roll out.
• We will improve our database management system to prevent the configuration change of this incident from triggering a restart.
• We will add additional testing to detect restart-triggering changes before we deploy them.