2019-04-04 Service Incident
Incident
Report for Sauce Labs European Data Center
Postmortem
Dates:
April 3rd, 2019 11:46 PM - April 4th, 2019 02:39 AM PDT
What happened:
Customers of the Real Device Cloud in the EU were unable to execute any type of automated tests or perform Live Testing.
Why it happened:
Firewall rules pushed to our DNS servers in the EU for security reasons that worked in our other regions caused issues in the EU due to environmental differences.
How we fixed it:
The broken firewall rules were hot patched and the Real Device Cloud in the EU was redeployed.
What we are doing to prevent it from happening again:
The DNS servers in the EU will be rebuilt to match the configuration of DNS servers in other regions our service supports. This will cause no downtime for customers.
Resolved
The problem has been resolved. All services are fully operational. Please contact support@saucelabs.com if you continue to experience issues.
Identified
Android Real Devices continue to operate normally. Our engineers have identified the problematic components causing iOS Real Device issues, and are working on remediation.
All US Devices are functioning normally.
All US Devices are functioning normally.
Monitoring
The problem has been corrected and service for Android devices is restored. We continue to investigate a problem with iOS devices.
Investigating
All EU Real Devices are Offline. Customers may experience problems with the REST API. Our developers are investigating.
This incident affected: Automated RDC Testing (Automated RDC Testing) and REST API (REST API RDC).