2020-September-4 Service Incident
Incident Report for ZZ [Archive] Sauce Labs European Data Center
Postmortem

Dates:

Friday, September 4 21:30 - 22:00 CEST 

What Happened?

A bug in our MySQL production database caused a service crash. 

Why it happened?

The production database bug caused a rare race condition.

How Was It Resolved?

We restarted the MySQL primary writer and service was fully restored.

What Are We Doing To Ensure It Doesn’t Happen Again?

We have made changes in the MySQL configuration and routing to prevent a similar issue from occurring in the future.

Posted Sep 18, 2020 - 17:25 CEST
Resolved
We have fully recovered. All services are fully operational.
Posted Sep 04, 2020 - 21:59 CEST
Monitoring
After remedial action all services are recovering. We are monitoring.
Posted Sep 04, 2020 - 21:47 CEST
Investigating
app.saucelabs.com is not available and all automated tests, API calls and tunnels will fail to start. We are investigating and taking remedial action.
Posted Sep 04, 2020 - 21:40 CEST
← Current Status Powered by Atlassian Statuspage