2020-September-9 Service Incident


Wednesday, September 9 17:30 - 21:40 CEST

What happened?

Some customers received erroneous messages stating they were running at their maximum concurrency.

Why did it happen?

During a routine deploy, a serialization library was updated. This update contained a hidden dependency with a non-compatible, breaking change which made the read and write values of this service to not work.

How was it resolved?

The deployment was rolled back.

What are we doing to prevent it from happening again?

We are improving our endpoint monitoring to detect stale values and other anomalies and improving dependency monitoring and alerts when a cross-dependency gets updated. We are also enhancing the deploy process of this service to ensure changes are properly validated before getting deployed.

Posted Sep 25, 2020 - 11:45 CEST

We have taken remedial action and the rest/v1.2/concurrency endpoint should return accurate data. Your concurrency meter in the user interface should also be working again.
Posted Sep 09, 2020 - 21:49 CEST
The /rest/v1.2/users/USERNAME/concurrency endpoint is not showing an accurate concurrency value. We are investigating and taking remedial action. Your concurrency meter in the user interface may not show the correct value.
Posted Sep 09, 2020 - 21:39 CEST