The fix was fully deployed as of 4:50pm Pacific. Engineering is re-deploying all affected services in order to ensure that they are running the latest successful deployment as expected. Affected services will see a "Triggered by Render" note for a deployment invoked between 5:00pm and 5:30pm Pacific.
Posted Oct 27, 2023 - 00:19 UTC
Identified
Services that use a Pre-Deploy Command may experience an issue where a previous deploy is made live approximately 2 hours later. This is problematic when multiple deploys take place over a 2 hour period, and suddenly code from multiple deployments back is seen executing again.
For example; if a service was deployed at 11:00am, 11:30am, and 12:15am, then around 1:00pm the 11:00am deployment may be seen again unexpectedly.
This issue would only occur on services that use a Pre-Deploy command after one is configured, and only when multiple deploys within 2 hours take place.
A fix for the past deployment being made live is already complete, and is being deployed.
Posted Oct 26, 2023 - 23:25 UTC
This incident affected: Oregon (Web Services, Background Workers, Builds and Deploys), Frankfurt (Web Services, Background Workers, Builds and Deploys), Ohio (Web Services, Background Workers, Builds and Deploys), and Singapore (Web Services, Background Workers, Builds and Deploys).