Dates:

Wednesday, September 9 17:30 - 21:40 CEST

What happened?

Some customers received erroneous messages stating they were running at their maximum concurrency.

Why did it happen?

During a routine deploy, a serialization library was updated. This update contained a hidden dependency with a non-compatible, breaking change which made the read and write values of this service to not work.

How was it resolved?

The deployment was rolled back.

What are we doing to prevent it from happening again?

We are improving our endpoint monitoring to detect stale values and other anomalies and improving dependency monitoring and alerts when a cross-dependency gets updated. We are also enhancing the deploy process of this service to ensure changes are properly validated before getting deployed.