Dates:

Tuesday, November 17, 10:00 - 20:00 CET

What happened:

The number of available devices decreased over a period of 10 hours to a point where 20% were unavailable. This impacted mostly customers with private devices or customers querying for very specific public device models.

Why it happened:

We deployed an update to our real device cloud which caused a small portion of live tests to not free up the device under use after the session was closed. 

How we fixed it:

We rolled back the changes which were causing the problem and released all affected devices so that they became available for use again.

What we are doing to prevent it from happening again:

We enhanced our monitoring to catch situations where we see a reduction of available devices, no matter the reason availability is reduced. In addition, we are going to add alerting for long-running sessions.