Dates:

Tuesday December 8, 14:15 - 14:26 CET

What happened:

The Sauce Labs Dashboard (EU), Legacy RDC UI and Automated and Live RDC testing were unavailable for ~10 minutes, displaying 503 error messages.

Why it happened:

A new index was created unintentionally on the primary node of our database, causing our database to become unresponsive and thereby making these services unavailable.

How we fixed it:

The database recovered after the operations were completed and our services became responsive again.

What we are doing to prevent it from happening again:

We are introducing limits to the operations run against our database and optimizing our processes to introduce restrictions that will prevent such high utilization operations in the future