Last night (6th of January) we finished testing the migration of all web server services to a new, upgraded server. Most hosted websites have been operating on the new server all week,  but we wanted to ensure that all upgrade issues for older sites were resolved before switching the old servers off. Users should notice a decrease in latency and page load times for websites (some websites will have this more pronounced than others depending on what server they were on previously).
At approximately 7:45AM GMT+8 yesterday morning (14/02) the data center which houses one of our email and web hosting servers experienced a major unscheduled outage. The outage caused a disconnection of most of the internet services to that data center, which resulted in this hosting server effectively going “offline”. At 12:00AM GMT+8 connectivity was mainly restored, with the connection being completely restored a couple of hours after that. Below is the connectivity graph from our monitoring system. Rises above zero indicate a connection loss / period of connection loss.



As this occurred during business hours and we had limited information for an ETA, two attempts to shift the services to another data center were made with insufficient success to make them viable solutions. One reason for this is that the backups that were attempting to be restored from were designed for data recovery, not service recovery. While we cannot control these outages, we have updated our backup / recovery procedures to be able to shift services in future should we need to.
To correspond with the new year we have released a newly designed website!