Connection problems resolved, 04/29/2020 12:11PM UTC Hub Pages

Notification history

04/29/2020 12:11PM UTC
Problem resolved. RFO is below.

What happened

Our monitoring systems detected connectivity issues on 2020-04-29 09:39 UTC

After investigation the issue was related to unusually high traffic coming to hub pages. At the peak there was 50x more requests than 30 minutes prior to the issue.
The issue resulted in a portion of end-users not being able to access hub pages fully. The total amount of requests that resulted in errors globally was 16%.

Because StatusHub utilise caching heavily such a spike in traffic shouldn’t cause the experienced problems so we have started to search for the root cause which was the cache invalidation mechanism being too sensitive connected with the auto-reload feature of hub pages. 
This problem was due to certain actions performed by subscribers were erroneously invalidating the whole cache for hub page and requesting an auto-reload. This error connected with high traffic caused the connectivity issues.

The issue was fully resolved on 2020-04-29 11:11 UTC after a hot-fix release addressing the cache invalidation issue.

Prevention and follow-up

We have changed the granularity of cache key which will reduce the number of unwanted cache invalidations going forward.

04/29/2020 10:39AM UTC
Connection issues detected.