General Category > Bug Reports

A Statement on recent events between 29 June and 4 July

(1/2) > >>

Kradorex Xeron:
All,

In the interests of transparency these are the course of events that have occurred that have caused various operational interruptions and their remediations:

As of 29 June, there was an interruption of power at the datacenter facility that houses some of EidolonHost's infrastructure and resulted in the server housing these forums becoming non-bootable. I received a formal invite to consult on the matter and weighed options and performed diagnostics alongside Keiro with prompt action. It was determined that reinstalling the server's operating system on a new HDD set while keeping user data on the old HDD set was the best course of action as to prevent data loss or risk installing an operating system alongside user data on the same disk.

On 30 June The previous HDD was then connected to the new environment and data restoration procedures put into play. The procedures were largely overseen and orchestrated by myself. I proceeded to divise an efficient cycle of recover, check, restore on two data sets (The user site data and MySQL raw database data) — none of the data needed to be restored from backup. This shortened the timeframe needed for the outage to continue. I worked largely through to late morning, 1 July.

On 1 July, restoration of the final group of accounts had been completed. restoration of most, if not everything had been completed and feedback was seemingly positive on the timeframe of action. It was determined that a "wait and see" approach would be for the best to determine if any additional restoration work would be needed.

On 4 July - early morning, Keiro had been re-attempting to restore an unknown component from the MySQL database system from the old filesystem despite all user data having been restored, in doing so, data had been overwritten and reset back to the 29 June binaries (updates since 1 July had been over-written) and caused various corruptions on at least one database unrelated to FSTDT that the procedures that I had developed were designed to prevent. I happened to have a copy of a backup on hand for FSTDT's forum database as of 09:10 EDT 3 July and proceeded to implement it.

As of 05:30 EDT 4 July: The FSTDT forums are in a stable operational state.

ThunderWulf:
So is that why people's Karma reverted a little bit?  I noticed that this morning.

Oh, and I just now noticed the missing posts.


I really hope that's the end of this.  I hope we don't have to go server hunting yet again.

nickiknack:

--- Quote from: Kradorex Xeron on July 04, 2012, 05:33:29 am ---As of 05:30 EDT 4 July: The FSTDT forums are in a stable operational state.

--- End quote ---

I really hope so, because really early this morning when I was having trouble with the forums, I was like "Oh Geez, Not Again"

Sleepy:
The server issues have gotten pretty fucking old.

ironbite:
Fucking Keiro

Navigation

[0] Message Index

[#] Next page