2020-Nov-17 RDC Service Incident


Tuesday, November 17, 10:00 - 20:00 CET

What happened:

The number of available devices decreased over a period of 10 hours to a point where 20% were unavailable. This impacted mostly customers with private devices or customers querying for very specific public device models.

Why it happened:

We deployed an update to our real device cloud which caused a small portion of live tests to not free up the device under use after the session was closed. 

How we fixed it:

We rolled back the changes which were causing the problem and released all affected devices so that they became available for use again.

What we are doing to prevent it from happening again:

We enhanced our monitoring to catch situations where we see a reduction of available devices, no matter the reason availability is reduced. In addition, we are going to add alerting for long-running sessions.

Posted Nov 24, 2020 - 18:39 CET

Device availability is back to normal. All services are operational.
Posted Nov 17, 2020 - 21:40 CET
We have identified the issue and took remedial actions. Device availability is back to normal. We are monitoring the situation.
Posted Nov 17, 2020 - 20:23 CET
We have identified the problem and are currently working on a fix.
Posted Nov 17, 2020 - 19:28 CET
Real devices are intermittently not being released for use after test sessions, resulting in reduced numbers of available devices and user concurrency being consumed. We are investigating
Posted Nov 17, 2020 - 18:29 CET