All,
An outage of services hosted on the serv.eidolonhost.com machine begun at 05:30 EST, this outage was due to a backup mechanism that created a backup of 486 GBytes. This backup fully filled up the server's hard disk and impaired the server's operating ability, including database transactions and web services. Below is a sequence of events from my perspective:
05:30 EST - Downtime in play
06:00 - I was called to manage and rectify the situation
06:25 - I traced the issue to being disk space usage by way of the MySQL (database service) logs, I then begun work to discover where the disk usage was expended to.
06:30 - The expendature was traced to a backup hosted on the same machine, the backup was in the amount of 486 GBytes, I begun purging parts of this backup up to 147 GBytes to provide more operational disk space until a longterm solution can be created.
06:35 - MySQL was restored, as was web services, I begun checking around the system for any further impairments.
06:50 - Various forums and similar software had blank pages and errors displayed.
07:00 - The event at 06:50 was due to MySQL tables marked as "crashed" and needed checking, I executed a check and repair operation on these.
07:15 - Repair operations completed.
07:25 - Systems operational as normal
~ Kradorex Xeron
Executive Director
Digibase Operations, Research and Development
