A SSL certificate in the chain of the certificate used for TLS 1.2 on https://app.testobject.com has expired.
Any client (web browser, REST API client) using TLS 1.2 and making requests to https://app.testobject.com may see “certificate has expired” errors. Any client using TLS 1.3 is unaffected. We are working to rectify this issue.
Any client (web browser, REST API client) using TLS 1.2 and making requests to https://app.testobject.com may see “certificate has expired” errors. Any client using TLS 1.3 is unaffected. We are working to rectify this issue.
2020-August-30 Service Incident
Incident
Report for Sauce Labs European Data Center
Postmortem
Dates:
Sunday August 30 14:00 CEST to Sunday August 30 20:20 CEST
What happened?
Some Sauce Labs customers experienced intermittent connectivity issues.
Why it happened?
CenturyLink, a global Internet backbone network, experienced a network outage that affected the Internet globally, particularly in Europe.
How we fixed it?
Internet Service Providers began disconnecting from CenturyLink while they worked on resolving their outage. As CenturyLink’s outage began to resolve, we started recovering Real Devices that did not recover automatically, and restarted some services.
What we are doing to prevent it from happening again?
We’re adding additional monitoring of external endpoints to provide better insight into issues experienced on the Internet and enable more rapid response.
Resolved
We have taken remedial action and the Real Device Cloud has recovered.
Monitoring
We are taking remedial action and the Real Device Cloud is recovering. We are monitoring the recovery.
Update
EU Real Device availability continues to recover with 70% of devices now available for testing. The legacy RDC UI, the Sauce Labs wiki and the Sauce Labs dashboard are operational.
Update
We have made some changes to resolve the degraded performance of the Sauce Labs dashboard, but issues affecting the availability of real devices and access to https://app.testobject.com are ongoing.
Identified
We are seeing some improvements, but the resolution of this issue is very much dependent on Level3/CenturyLink. https://app.testobject.com is available from some locations, but not others, and our Real Device availability is gradually recovering.
Investigating
A major global internet provider (Level3/CenturyLink) is experiencing issues which are causing parts of our service to be unavailable or degraded. Currently Real Device testing, the legacy RDC UI at app.testobject.com and the Sauce Labs wiki are unavailable, and the Sauce Labs dashboard is experiencing intermittent issues.
This incident affected: Web Interface (Sauce UI, Legacy Real Device Cloud UI, Analytics), Automated RDC Testing (Automated RDC Testing), and Manual Testing (Manual RDC Testing).