Last night's forum meltdown

[ubeaut]

Major melt down at the server last night has meant we were off the ail with a big problem and the loss of a days worth of threads and posts

Steven has been up all night working on the problem which was basically due to corrupt ram which was installed last week.

Problem was completely out of our hands and occurred at the server hosted by The Planet in Galveston, Texas. This is the 2nd time they've installed corrupt ram.

Here's the running emails from Steven to me, as the problem unfolded last night and this morning.

July 5th 2010
8:14pm - rebooting again

9:57pm - another 9 minutes of downtime

10:42pm - They (the Planet) have suggested taking the server down and check the new hardware
that was added last week. I'm waiting on confirmation regarding how long
it will take before I OK it.

10:58pm - I've booked the window in to start from 1am.

July 6th 2010
2:47am - The DC is going to run a file system check, I'm not sure how long it'll take.

3:05am - If we need to recover from backups, I have assistance organized to move to new hardware with a different provider. The file check is running now, I don't have an ETA.

5:51am - We're just working through some table corruption issues, its been a long night.

We're just restoring a backup and will recover the data for the corrupt
tables, its nothing major (post table is ok) just some tables and thread table. So we may loose some threads that were created yesterday, posts sent to older threads will be ok. I will update you again soon.
________________________________________

Then the problems started as the Thread Table was corrupt and all the posts associated with the threads then became new threads themselves, throwing the whole forums into a state of utter chaos, looking like a bit of a jumble sale of words.

The only way to repair it was to reinstall from yesterdays back up which is done in the wee small hours.

All is fine again now except for the loss of most new Threads and posts from yesterday as the backup was done at 2am Monday morning meaning the loss of 24 hrs of threads and posts.

We apologize for any inconvenience and hope it doesn't happen again for the least another 10 years. Hopefully never.

Cheers - Neil

[chrisb691]

Many thanks to all involved, especially Steven, for getting the forum back on the air.

I personally, do not believe that any apology is warranted, or needed, from Ubeaut. When something like this forum is handed out freely, then the recipients should never quibble when things go wrong, but be greatful for the efforts put into rectification.

Thanks Neil.

[Woodwould]

Well done chaps and many thanks for your continued efforts.

[snowyskiesau]

All your hard work is appreciated.

[Horaldic]

Thanks for all your efforts. This type of thing is very frustrating to work on and I appreciate the work that has gone in.

[bobsreturn2003]

GREAT TO HAVE IT BACK . WELL DONE

[rsser]

I can relate to corrupt RAM ;-}

[NeilS]

Hardware and backup system failure happens, with even the best run systems..... I know from experience having been there on one of those all night recovery sessions. Thanks to all concerned for their efforts.

.....

[munruben]

A big thank you to all the guys and gals behind the scenes who keep this forum running. and a special thanks to Steven in this instance. A great effort all round.

[Charleville]

A big thank you to all the guys and gals behind the scenes who keep this forum running. and a special thanks to Steven in this instance. A great effort all round.

Ditto! I get a lot of joy and education from this forum.

Many thanks.


.

[fletty]

A big thank you to all the guys and gals behind the scenes who keep this forum running. and a special thanks to Steven in this instance. A great effort all round.

very, very ditto!
fletty

[HavinaGo]

Thanks for a great place to share info and learn and for keeping it running.

[gawdelpus]

Thank goodness for backups A lot of people have no Idea just what is required to keep something like this ticking over smoothly ,so much behind the scenes just has to be done ,a bit disappointing for the efforts of upgrading the equipment ony for it to fail ,great recovery though all round cheers ~ John

[Spanner69]

Well done for a number of things.

1/ getting things back so quickly
2/ keeping all us in the loop with information.
3/being such bril' people to have such a site as this.

[rodney]

G'Day

Just a suggestion next time you are looking into an external hosted solution ask the company if they can provide a virtual environment. This will shield you from physical hardware issues, when hardware fails or a maintenance event is planned the host can be moved dynamically onto another server. The product we are about to install can do the fail over automatically when an issue happens with the underlying physical server.

This doesnt protect you from corruption in the file system and/or database, however these are much rarer events. The only way to protect from these is regular maintenance.

Just curious, is this using Open Source software under the hood?

Cheers
Rodney