2019-April-08 Service Incident
Incident Report for Sauce Labs European Data Center
Postmortem

What Happened?

New Sauce Connect tunnels would not start in our European datacenter.

Why Did It Happen?

We inadvertently made a network level change to our Sauce Connect Tunnel Cloud that made it unable to receive requests to start new tunnels Apr 9 1:58 CEST.

How Was It Resolved?

The change was reverted at 3:10 CEST and network connectivity was restored. New tunnels could be started again, but there was a large backlog of tunnels requests to process at the same time. This artificial "spike" in demand for new tunnels overwhelmed the Tunnel Cloud and, in effect, new tunnels still could not be started. At 3:58 CEST additional changes were made to throttle the incoming requests and allow our Tunnel Cloud to stabilize. By 4:30 CEST we had fully recovered and established all requested Sauce Connect Tunnels.

Why Are We Doing To Ensure It Never Happens Again?

*Resolving the bug found in our configuration that caused a loss of network connectivity.

*Improving our response process to allow our Sauce Connect Tunnel cloud to recover more rapidly.

*Adding additional capacity to our Sauce Connect Tunnel Cloud in Europe to improve resiliency and better meet spikes in demand.

*Making additional software and code updates to improve the responsiveness of our Sauce Connect Cloud under load.

Posted 4 months ago. Apr 18, 2019 - 17:16 CEST

Resolved
The problem has been resolved. All services are fully operational. Please contact support@saucelabs.com if you continue to experience issues.
Posted 5 months ago. Apr 09, 2019 - 04:47 CEST
Identified
Sauce Connect tunnels can not connect reliably to our EU Virtual Device Cloud. Our Engineers are aware of the cause and are taking mitigating action.
Posted 5 months ago. Apr 09, 2019 - 04:24 CEST
This incident affected: Sauce Connect (Sauce Connect VM).