Unexpected downtime, Sunday the 14th of May

Posted

On Saturday night at approximatley 00:30 GMT the iHasco LMS and Training Suite became very slow to respond for all customers for approximately 8 hours. When we became aware of this issue at approx 8:30 GMT we decided to take the LMS and Training Suite offline while we worked to resolve the performance issues. The application was subsequently offline for approximatley 5 hours. Everything is up and running smoothly again now, but we wanted to apologise becuase we know how many people rely on us to deliver online learning to their staff.

Here’s what happened

First of all, any downtime is inexcusable. We’re incredibly sorry for preventing your staff from being able to complete their training or denying you access to your management reports.

We pushed live an update to our application which made a number of changes to our database structure. Whilst the migration ran without issues in our local and remote test environments it stalled heavily on the live server.

Here’s what we did

The moment we became aware of the performance issues we disabled access to the LMS and Training Suite so that we could run the migration locally and then deploy that to the live server. This was a time consuming process and once complete we had to run a number of tests to ensure the integrity of the migrated data before switching the LMS and Training Suite access back on.

Here’s what we’re going to do

In the future we will be taking the application offline for more significant migrations which carry a risk of degrading performance and will provide sufficient warning of any such maintenance.

Finally, I just wanted to say a massive thanks for your understanding during the 12 or 13 hours during which the application was unavailable.

Comments