A SSL certificate in the chain of the certificate used for TLS 1.2 on https://app.testobject.com has expired.
Any client (web browser, REST API client) using TLS 1.2 and making requests to https://app.testobject.com may see “certificate has expired” errors. Any client using TLS 1.3 is unaffected. We are working to rectify this issue.
2020-November-19 Service Incident
Incident Report for Sauce Labs European Data Center
Postmortem

Dates:

Tuesday, November 19 17:40 - 19:30 CET

What happened:

We experienced an increased error rate for jobs in the us-west-1, eu-central-1 and us-east-1 datacenters.

Why it happened:

Back pressure from a third party provider caused a delay and build up of requests within the internal service responsible for starting new jobs. This delay caused a crash loop within our service.

How we fixed it:

Calls to the third party provider experience back pressure were disabled and rerouted to different end points that weren’t experiencing delays.

What we are doing to prevent it from happening again:

We’ve modified our services to route traffic via a more resilient path to our third party provider.  We’ve also modified our behaviour to drop messages on a queue without waiting for a response to eliminate the chance that back pressure causes our internal services to become overwhelmed.  In addition, we are going to add additional monitoring and alerting for these subsystems

Posted Nov 24, 2020 - 18:43 CET

Resolved
Error rates have subsided. All services are fully operational
Posted Nov 19, 2020 - 20:39 CET
Monitoring
We have taken remedial action and are seeing improvements. We are monitoring.
Posted Nov 19, 2020 - 19:45 CET
Update
We are seeing a high error rate on automated browser and emusim tests, as well as non-legacy RDC tests. We are continuing to investigate.
Posted Nov 19, 2020 - 18:46 CET
Investigating
We are experiencing a service incident. We are currently determining the scope of the issue and will update shortly once that has been established.
Posted Nov 19, 2020 - 18:34 CET
This incident affected: Automated RDC Testing (Automated RDC Testing) and Automated VM Testing (Automated PC Testing, Automated Mac Testing, Automated iOS Simulator Testing, Automated Android Emulator Testing).