2020-December-09 Service Incident
Postmortem

Dates:

Wednesday December 9, 13:40 - 13:53 CET

What happened:

The Sauce Labs Dashboard (EU), Legacy RDC UI and Automated and Live RDC testing were unavailable for ~13 minutes, displaying 503 error messages.

Why it happened:

A new index was created unintentionally on the primary node of our database, causing our database to become unresponsive and thereby making these services unavailable.

How we fixed it:

The database recovered after the operations were completed and our services became responsive again.

What we are doing to prevent it from happening again:

We are introducing limits to the operations run against our database and optimizing our processes to introduce restrictions that will prevent such heavy operations in the future.

Posted Dec 14, 2020 - 20:58 CET

Resolved
This incident has been resolved.
Posted Dec 09, 2020 - 14:32 CET
Monitoring
Service has recovered and we are monitoring to make sure we are back at full capacity.
Posted Dec 09, 2020 - 14:24 CET
Investigating
The Sauce Labs Dashboard (EU), Legacy RDC UI and Automated and Live RDC testing are unavailable. We are investigating.
Posted Dec 09, 2020 - 13:51 CET