2020-September-4 Service Incident

Incident Report for ZZ [Archive] Sauce Labs European Data Center

Postmortem

Dates:

Friday, September 4 21:30 - 22:00 CEST

What Happened?

A bug in our MySQL production database caused a service crash.

Why it happened?

The production database bug caused a rare race condition.

How Was It Resolved?

We restarted the MySQL primary writer and service was fully restored.

What Are We Doing To Ensure It Doesn’t Happen Again?

We have made changes in the MySQL configuration and routing to prevent a similar issue from occurring in the future.

Posted Sep 18, 2020 - 17:25 CEST

Resolved

We have fully recovered. All services are fully operational.

Posted Sep 04, 2020 - 21:59 CEST

Monitoring

After remedial action all services are recovering. We are monitoring.

Posted Sep 04, 2020 - 21:47 CEST

Investigating

app.saucelabs.com is not available and all automated tests, API calls and tunnels will fail to start. We are investigating and taking remedial action.

Posted Sep 04, 2020 - 21:40 CEST