2020-November-09 Resolved Service Incident
Postmortem

Dates:

Saturday, November 7 21:00 CET - Monday, November 9 20:10 CET 

What happened:

Sauce Labs Real Device customers running tests with Sauce Connect on the Unified Platform in the EU-Central data center experienced intermittent test failures. 

Why it happened:

A planned controller failover during a maintenance window caused a traffic pattern to change, resulting in some traffic hitting an Access Control List that did not allow communication back to Sauce Connect tunnels. Although tests were in place to monitor the affected services they did not trigger any production alerts. 

How we fixed it:

The Access Control List was updated to allow this communication flow.

What we are doing to prevent it from happening again:

We are adding additional alerts to our systems, and ensure that they will trigger an alert in case of an outage. We are reviewing our standards on the implementation of monitoring for new, and existing systems, and adapt our incident training.

Posted Nov 24, 2020 - 18:36 CET

Resolved
We detected some RDC tests using Sauce Connect on our unified platform were not able to start correctly for a period of time. We have taken remedial action and fixed the issue that caused this. All services are fully operational.
Posted Nov 09, 2020 - 20:18 CET