Saturday, November 7 12:00 PST - Monday, November 9 11:10 AM PST
Sauce Labs Real Device customers running tests on the Unified Platform in the EU-Central data center experienced intermittent test failures with Sauce Connect.
A planned controller failover during a maintenance window caused a traffic pattern to change, resulting in some traffic hitting an Access Control List that did not allow communication back to Sauce Connect tunnels. Although tests were in place to monitor the affected services they did not trigger any production alerts.
The Access Control List was updated to allow this communication flow.
We are adding additional alerts to our systems, and ensure that they will trigger an alert in case of an outage.
We are reviewing our standards on the implementation of monitoring for new, and existing systems, and adapt our incident training.