Dates:

Wednesday, December 16, 8:50AM - 12:00PM CET

What happened:

Video recordings were missing from automated tests on our Real Device Cloud. This affected all Appium, XCUITest and Espresso tests for public and private devices in all regions.

Why it happened:

The roles for a process were changed as part of a release. This change caused the process responsible for recording the video to be unable to start.

How we fixed it:

The necessary roles were adapted, and a patched version was deployed.

What we are doing to prevent it from happening again:

The problem was caught on production, not during release validation. A test will be executed as part of the release validation process going forward. We have also adjusted the time it would take for a single failing test to trigger our alerting in this situation.