Slow Performance of Matrix Booking system

Incident Report for Matrix Booking

Postmortem

On the morning of the 9th April 2024, the Matrix Booking platform suffered from degraded performance during peak times which resulted in users not being able to access the system.

At around 9:27am the system alerted us to issues with the infrastructure and an increase in the response times, investigation uncovered that that a number of long processes were causing an contention on the database. Attempts were made to resolve the issue without forcing a fail over but were unsuccessful.

At 10:04 it was decided to force a fail over of the platform which took approximately 30 minutes and resulted in the resolution of the issue at 10:32.

We sincerely apologise for any inconvenience this caused.

As a result further investigation will be done into the underlying issues that resulted in the degradation of the system performance and also the speed in which we were able to return the system to normal performance levels.

Posted Apr 16, 2024 - 09:45 UTC

Resolved

This incident has been resolved.
Posted Apr 09, 2024 - 10:09 UTC

Update

We are continuing to monitor for any further issues.
Posted Apr 09, 2024 - 10:08 UTC

Monitoring

we are continuing to monitor the production performance.
Posted Apr 09, 2024 - 09:45 UTC

Identified

The immediate issue has been resolved and performance has returned to normal.
Posted Apr 09, 2024 - 09:44 UTC

Investigating

We currently investigating an issue with slow performance of the Matrix Booking platform.
Posted Apr 09, 2024 - 09:08 UTC
This incident affected: Web applications, Mobile applications, Email delivery service, API service, Matrix Welcome apps (inc Digital Signage), Office 365 Calendar Integration, and Outlook .net AddIn.