Today, starting at approximately 10:03 UTC, a misconfiguration at a major external network provider caused connectivity disruptions across the Internet. These disruptions impacted the operations of many service providers, including Cloudflare. Cloudflare’s systems routed around the problem, however, some Internet properties that use Cloudflare saw increased error rates during the impact period despite our mitigations.During this period we saw traffic drop by about 3.5% across our network as well as an increase in HTTP errors which may have presented as 522 (Connection Timeout) status codes.

Cloudflare have posted an initial detailed analysis of this incident written by Cloudflare CEO on the Cloudflare blog. Presently, Cloudflare is no longer seeing additional impact from these transit provider issues.

These events can be disruptive, Cloudflare will continue to provide the latest updates for issues impacting Cloudflare's network on cloudflarestatus.com.

Below is the sequence of the investigation carried out and the resolution of the problem.

Third party transit provider issue causing increased HTTP 5xx errors
Resolved - This incident has been resolved.
Aug 30, 16:12 UTC
Update - We are continuing to monitor for any further issues.
Aug 30, 15:31 UTC
Monitoring - At this moment we are seeing network stability and reachability return back to normal levels, and we are monitoring the situation. We are updating the status of this incident to Monitoring, and if the situation changes we will update accordingly.
Aug 30, 15:16 UTC
Update - As the major provider that was experiencing issues has implemented a fix in their network, other global network providers have started to re-enable their peering sessions with it. We are continuing to see improved reachability and monitoring the situation.
Aug 30, 15:05 UTC
Update - We are observing significantly improved stability and reachability in the network as the third party provider is implementing a fix for their issue and we have implemented measures to work around the problem
Aug 30, 14:46 UTC
Update - Major transit providers are taking action to work around the network that is experiencing issues and affecting global traffic.

We are applying corrective action in our data centers as the situation changes in order to improve reachability
Aug 30, 14:26 UTC
Update - We are continuing to engage with our partners, fixes are being implemented with other transit providers.
Aug 30, 13:58 UTC
Update - We are continuing to work on a fix for this issue.
Aug 30, 12:59 UTC
Update - We have identified an issue with a transit provider which is causing 5xx class HTTP errors, such as HTTP 522, 502, 503.

This is affecting all data centers that make use of this transit provider and we are working on implementing mitigations to alleviate this issue.
Aug 30, 11:57 UTC
Identified - The issue has been identified and a fix is being implemented.
Aug 30, 11:39 UTC
Update - We are continuing to investigate this issue.
Aug 30, 11:08 UTC
Update - Cloudflare is investigating an increased level of HTTP 5xx class errors.

We are working to understand the full impact and mitigate this problem. More updates to follow shortly.
Aug 30, 10:45 UTC
Investigating - Cloudflare is aware of, and investigating an issue which potentially impacts multiple customers. Further detail will be provided as more information becomes available.
Aug 30, 10:21 UTC


Monday, August 31, 2020





« Back