ZZ [Archive] Sauce Labs European Data Center Status

2020-August-30 Service Incident

Incident Report for ZZ [Archive] Sauce Labs European Data Center

Postmortem

Dates:

Sunday August 30 14:00 CEST to Sunday August 30 20:20 CEST

What happened?

Some Sauce Labs customers experienced intermittent connectivity issues.

Why it happened?

CenturyLink, a global Internet backbone network, experienced a network outage that affected the Internet globally, particularly in Europe.

How we fixed it?

Internet Service Providers began disconnecting from CenturyLink while they worked on resolving their outage. As CenturyLink’s outage began to resolve, we started recovering Real Devices that did not recover automatically, and restarted some services.

What we are doing to prevent it from happening again?

We’re adding additional monitoring of external endpoints to provide better insight into issues experienced on the Internet and enable more rapid response.

Posted Sep 16, 2020 - 21:26 CEST

Resolved

We have taken remedial action and the Real Device Cloud has recovered.

Posted Aug 30, 2020 - 20:23 CEST

Monitoring

We are taking remedial action and the Real Device Cloud is recovering. We are monitoring the recovery.

Posted Aug 30, 2020 - 19:48 CEST

Update

EU Real Device availability continues to recover with 70% of devices now available for testing. The legacy RDC UI, the Sauce Labs wiki and the Sauce Labs dashboard are operational.

Posted Aug 30, 2020 - 17:25 CEST

Update

We have made some changes to resolve the degraded performance of the Sauce Labs dashboard, but issues affecting the availability of real devices and access to https://app.testobject.com are ongoing.

Posted Aug 30, 2020 - 16:03 CEST

Identified

We are seeing some improvements, but the resolution of this issue is very much dependent on Level3/CenturyLink. https://app.testobject.com is available from some locations, but not others, and our Real Device availability is gradually recovering.

Posted Aug 30, 2020 - 14:56 CEST

Investigating

A major global internet provider (Level3/CenturyLink) is experiencing issues which are causing parts of our service to be unavailable or degraded. Currently Real Device testing, the legacy RDC UI at app.testobject.com and the Sauce Labs wiki are unavailable, and the Sauce Labs dashboard is experiencing intermittent issues.

Posted Aug 30, 2020 - 14:07 CEST