2020-December-16 Resolved Service Incident
Postmortem

Dates:

Wednesday, December 16, 8:50AM - 12:00PM CET

What happened:

Video recordings were missing from automated tests on our Real Device Cloud. This affected all Appium, XCUITest and Espresso tests for public and private devices in all regions.

Why it happened:

The roles for a process were changed as part of a release. This change caused the process responsible for recording the video to be unable to start.

How we fixed it:

The necessary roles were adapted, and a patched version was deployed.

What we are doing to prevent it from happening again:

The problem was caught on production, not during release validation. A test will be executed as part of the release validation process going forward. We have also adjusted the time it would take for a single failing test to trigger our alerting in this situation.

Posted Dec 22, 2020 - 12:56 CET

Resolved
Videos on the Real Device Cloud were not recorded today between 12am (PDT) and 3am (PDT). Our engineering team took remedial action and recordings are now working as expected.
Posted Dec 16, 2020 - 12:41 CET