At approximately 7:45AM GMT+8 yesterday morning (14/02) the data center which houses one of our email and web hosting servers experienced a major unscheduled outage. The outage caused a disconnection of most of the internet services to that data center, which resulted in this hosting server effectively going “offline”. At 12:00AM GMT+8 connectivity was mainly restored, with the connection being completely restored a couple of hours after that. Below is the connectivity graph from our monitoring system. Rises above zero indicate a connection loss / period of connection loss.

As this occurred during business hours and we had limited information for an ETA, two attempts to shift the services to another data center were made with insufficient success to make them viable solutions. One reason for this is that the backups that were attempting to be restored from were designed for data recovery, not service recovery. While we cannot control these outages, we have updated our backup / recovery procedures to be able to shift services in future should we need to.