Logging in all regions is performing well. If you are experiencing any remaining issues with log reliability, please reach out to support@render.com
Posted Jul 15, 2022 - 21:14 UTC
Update
The deploy has completed and we are seeing logs flowing to both the dashboard UI as well as to log streams. We will continue to monitor logs this afternoon to ensure there were no adverse effects, and to validate that these changes improve the log loss and delayed logs some users were seeing.
Posted Jun 16, 2022 - 17:13 UTC
Update
We believe we have identified all the causes behind log loss and log delays that some users are experiencing, and we pushed a fix this morning. Unfortunately, the fix inadvertently dropped logs from log streams from 13:46-14:17 UTC. We reverted to investigate.
We have fixed the issue and are redeploying. We will be monitoring to ensure there is no disruption to log forwarding. We will update the status when the deploy completes.
Posted Jun 16, 2022 - 16:44 UTC
Monitoring
We have reverted the changes that caused the disruption. Services should be receiving logs via the dashboard and log streams. We are working to make sure the next rollout goes more smoothly and will update this status when the changes are deployed.
Posted Jun 13, 2022 - 22:29 UTC
Investigating
During a rollout of a change to improve logging reliability some of the necessary components failed to update properly. This caused logs and log streams for some services to be interrupted. We are working on resolving this as quickly as possible.
Posted Jun 13, 2022 - 19:46 UTC
Monitoring
We've deployed a change to one of our clusters that our initial testing has shown improves log reliability. We will be monitoring our logging system to ensure there's no negative impact on any user logs. If the change performs well in production we will then roll it out to all of our clusters.
Posted Jun 13, 2022 - 17:15 UTC
Identified
Since 05/18 a small number of users have been experiencing sporadic logging issues that we determined are more systemic than initially thought. During our investigation we made several changes to the existing infrastructure that did not alleviate the problem and ultimately made the decision to implement a more robust tool to better handle our log forwarding needs. The changes necessary to implement this tool are in place and will be deployed shortly. Once deployed, we will monitor the results and ensure that the new system performs as we are expecting.