Author Topic: A Statement on recent events between 29 June and 4 July  (Read 3475 times)

0 Members and 1 Guest are viewing this topic.

Offline Kradorex Xeron

  • Technical Consultant
  • Administrator
  • Bishop
  • *****
  • Posts: 115
  • Gender: Male
  • Digitally Original
    • Personal Site
A Statement on recent events between 29 June and 4 July
« on: July 04, 2012, 05:33:29 am »
All,

In the interests of transparency these are the course of events that have occurred that have caused various operational interruptions and their remediations:

As of 29 June, there was an interruption of power at the datacenter facility that houses some of EidolonHost's infrastructure and resulted in the server housing these forums becoming non-bootable. I received a formal invite to consult on the matter and weighed options and performed diagnostics alongside Keiro with prompt action. It was determined that reinstalling the server's operating system on a new HDD set while keeping user data on the old HDD set was the best course of action as to prevent data loss or risk installing an operating system alongside user data on the same disk.

On 30 June The previous HDD was then connected to the new environment and data restoration procedures put into play. The procedures were largely overseen and orchestrated by myself. I proceeded to divise an efficient cycle of recover, check, restore on two data sets (The user site data and MySQL raw database data) — none of the data needed to be restored from backup. This shortened the timeframe needed for the outage to continue. I worked largely through to late morning, 1 July.

On 1 July, restoration of the final group of accounts had been completed. restoration of most, if not everything had been completed and feedback was seemingly positive on the timeframe of action. It was determined that a "wait and see" approach would be for the best to determine if any additional restoration work would be needed.

On 4 July - early morning, Keiro had been re-attempting to restore an unknown component from the MySQL database system from the old filesystem despite all user data having been restored, in doing so, data had been overwritten and reset back to the 29 June binaries (updates since 1 July had been over-written) and caused various corruptions on at least one database unrelated to FSTDT that the procedures that I had developed were designed to prevent. I happened to have a copy of a backup on hand for FSTDT's forum database as of 09:10 EDT 3 July and proceeded to implement it.

As of 05:30 EDT 4 July: The FSTDT forums are in a stable operational state.

~Krad Xeron

Offline ThunderWulf

  • Strange, even crazy, but never dull
  • The Beast
  • *****
  • Posts: 2920
  • Gender: Male
  • By Odin's beard!
Re: A Statement on recent events between 29 June and 4 July
« Reply #1 on: July 04, 2012, 11:31:03 am »
So is that why people's Karma reverted a little bit?  I noticed that this morning.

Oh, and I just now noticed the missing posts.


I really hope that's the end of this.  I hope we don't have to go server hunting yet again.
« Last Edit: July 04, 2012, 11:46:00 am by TGRwulf »
a.k.a. TGRwulf
"hehehehe. you said member." ~ Shepard/Booker
"it's kind of like my right left hand on a sunday every night. How so? It beats the fuck out of me!" ~ Saturn500
"Drinking, fighting, fucking...they basically outlawed 99% of the lifestyle of your typical Irishman.  Much less your typical Viking." ~ RavynousHunter

Offline nickiknack

  • I Find Your Lack of Ponies... Disturbing
  • The Beast
  • *****
  • Posts: 6037
  • Gender: Female
  • HAS A KINK FOR SPACE NAZIS
Re: A Statement on recent events between 29 June and 4 July
« Reply #2 on: July 04, 2012, 11:38:47 am »
As of 05:30 EDT 4 July: The FSTDT forums are in a stable operational state.

I really hope so, because really early this morning when I was having trouble with the forums, I was like "Oh Geez, Not Again"
« Last Edit: July 04, 2012, 11:48:51 am by nickiknack »

Offline Sleepy

  • Fuck Yes Sunshine In a Bag
  • The Beast
  • *****
  • Posts: 4598
  • Gender: Female
  • Danger zone
Re: A Statement on recent events between 29 June and 4 July
« Reply #3 on: July 04, 2012, 11:53:22 am »
The server issues have gotten pretty fucking old.
Guys, this is getting creepy. Can we talk about cannibalism instead?

If a clown eats salmon on Tuesday, how much does a triangle weigh on Jupiter? Ask Mr. Wiggins for 10% off of your next dry cleaning bill. -Hades

Offline ironbite

  • Overlord of all that is good in Iacon City
  • Kakarot
  • ******
  • Posts: 10686
  • Gender: Male
  • Stuck in the middle with you.
Re: A Statement on recent events between 29 June and 4 July
« Reply #4 on: July 04, 2012, 06:00:35 pm »
Fucking Keiro

Offline Bezron

  • a mysterious riddled spicy burrito wrapped enigma...stuck in some hot chick's colon dreaming of fanginas
  • Pope
  • ****
  • Posts: 400
  • Gender: Male
Re: A Statement on recent events between 29 June and 4 July
« Reply #5 on: July 09, 2012, 12:44:49 pm »
The server issues have gotten pretty fucking old.

This...

I understand we aren't paying customers or anything, but this stability issue over the last 6-8 months is kind of concerning.

Offline Keiro Dreamwalker

  • An Puisín Faolchú
  • Pope
  • ****
  • Posts: 371
  • Gender: Male
Re: A Statement on recent events between 29 June and 4 July
« Reply #6 on: July 11, 2012, 04:18:33 pm »
This server will be moved to R5-Dallas soon. :U

You'll be noticing a suspended screen for here once I announce the schedule for the outage, and ask Distind to change the IP this subdomain points to.

After that, it's a matter of DNS resolving to the new server, and since we don't directly handle FSTDT's DNS, this won't be an issue for us.
You have made a place in my heart where I thought there was no room for anything else. You have made flowers grow where I cultivated dust and stones. Remember this, on this journey you insist on making. If you die, I will not survive you long. Dovie'andi se tovya sagain!



People so seldom say "I love you". And then it's either too late or love goes. So when I tell you I love you, It doesn't mean I know you'll never go, only that I wish you didn't have to.