2020-December-08 Resolved Service Incident
Postmortem

Dates:

Tuesday December 8, 14:15 - 14:26 CET

What happened:

The Sauce Labs Dashboard (EU), Legacy RDC UI and Automated and Live RDC testing were unavailable for ~10 minutes, displaying 503 error messages.

Why it happened:

A new index was created unintentionally on the primary node of our database, causing our database to become unresponsive and thereby making these services unavailable.

How we fixed it:

The database recovered after the operations were completed and our services became responsive again.

What we are doing to prevent it from happening again:

We are introducing limits to the operations run against our database and optimizing our processes to introduce restrictions that will prevent such high utilization operations in the future

Posted Dec 14, 2020 - 20:56 CET

Resolved
The Sauce Labs Dashboard (EU), Legacy RDC UI and Automated and Live RDC testing were unavailable between 14:15 and 14:26 CET today. The issue was resolved and all services are operational.
Posted Dec 08, 2020 - 14:30 CET