2021-01-15 Resolved Service Incident
Postmortem

Dates:

Friday, January 15th, 08:30 - 15:00 CET

What happened:

During the time period above, some files uploaded to Sauce Storage became corrupted, making them unusable for testing.

Why it happened:

A version update to a third party library used by our app storage service introduced a race condition where under some circumstances files were not uploaded properly even after returning a success message.

How we fixed it:

The App Storage Service deployment was reverted to an earlier version containing the older snapshot of the third party library component. Any affected files uploaded during the time period above need to be re-uploaded.

What we are doing to prevent it from happening again:

We improved the alerting notifications to expose more warnings to be able to detect potential problems faster in our staging environment before hitting production.

Posted Jan 21, 2021 - 09:42 CET

Resolved
Between 0800 and 1500 CET, some apps uploaded to the new Sauce Application Storage service were corrupted. This caused app installation errors on Automated and Live Real Device tests and Automated Emulator/Simulator tests using the new Sauce Application Storage service (using the prefix "storage:" in the "app" capability). The problem has been fixed, but any app uploaded between these times that is exhibiting app installation errors should be uploaded again.
Posted Jan 15, 2021 - 16:00 CET