Last night's forum meltdown

[ubeaut]

Major melt down at the server last night has meant we were off the ail with a big problem and the loss of a days worth of threads and posts

Steven has been up all night working on the problem which was basically due to corrupt ram which was installed last week.

Problem was completely out of our hands and occurred at the server hosted by The Planet in Galveston, Texas. This is the 2nd time they've installed corrupt ram.

Here's the running emails from Steven to me, as the problem unfolded last night and this morning.

July 5th 2010
8:14pm - rebooting again

9:57pm - another 9 minutes of downtime

10:42pm - They (the Planet) have suggested taking the server down and check the new hardware
that was added last week. I'm waiting on confirmation regarding how long
it will take before I OK it.

10:58pm - I've booked the window in to start from 1am.

July 6th 2010
2:47am - The DC is going to run a file system check, I'm not sure how long it'll take.

3:05am - If we need to recover from backups, I have assistance organized to move to new hardware with a different provider. The file check is running now, I don't have an ETA.

5:51am - We're just working through some table corruption issues, its been a long night.

We're just restoring a backup and will recover the data for the corrupt
tables, its nothing major (post table is ok) just some tables and thread table. So we may loose some threads that were created yesterday, posts sent to older threads will be ok. I will update you again soon.
________________________________________

Then the problems started as the Thread Table was corrupt and all the posts associated with the threads then became new threads themselves, throwing the whole forums into a state of utter chaos, looking like a bit of a jumble sale of words.

The only way to repair it was to reinstall from yesterdays back up which is done in the wee small hours.

All is fine again now except for the loss of most new Threads and posts from yesterday as the backup was done at 2am Monday morning meaning the loss of 24 hrs of threads and posts.

We apologize for any inconvenience and hope it doesn't happen again for the least another 10 years. Hopefully never.

Cheers - Neil :U

[chrisb691]

Many thanks to all involved, especially Steven, for getting the forum back on the air.

I personally, do not believe that any apology is warranted, or needed, from Ubeaut. When something like this forum is handed out freely, then the recipients should never quibble when things go wrong, but be greatful for the efforts put into rectification.

Thanks Neil.

[Woodwould]

Well done chaps and many thanks for your continued efforts. :thyel:

[snowyskiesau]

All your hard work is appreciated.

[Horaldic]

Thanks for all your efforts. This type of thing is very frustrating to work on and I appreciate the work that has gone in.

[bobsreturn2003]

GREAT TO HAVE IT BACK . WELL DONE :2tsup:

[rsser]

I can relate to corrupt RAM ;-}

[NeilS]

Hardware and backup system failure happens, with even the best run systems..... I know from experience having been there on one of those all night recovery sessions. Thanks to all concerned for their efforts.

.....

[munruben]

A big thank you to all the guys and gals behind the scenes who keep this forum running. and a special thanks to Steven in this instance. A great effort all round.:2tsup::2tsup::2tsup:

[Charleville]

A big thank you to all the guys and gals behind the scenes who keep this forum running. and a special thanks to Steven in this instance. A great effort all round.:2tsup::2tsup::2tsup:


Ditto! I get a lot of joy and education from this forum.

Many thanks.


.

[fletty]

A big thank you to all the guys and gals behind the scenes who keep this forum running. and a special thanks to Steven in this instance. A great effort all round.:2tsup::2tsup::2tsup:

very, very ditto!
fletty

[HavinaGo]

:2tsup: Thanks for a great place to share info and learn and for keeping it running.

[gawdelpus]

Thank goodness for backups :) A lot of people have no Idea just what is required to keep something like this ticking over smoothly ,so much behind the scenes just has to be done ,a bit disappointing for the efforts of upgrading the equipment ony for it to fail :( ,great recovery though all round :2tsup: cheers ~ John

[Spanner69]

Well done for a number of things.

1/ getting things back so quickly
2/ keeping all us in the loop with information.
3/being such bril' people to have such a site as this.

[rodney]

G'Day

Just a suggestion next time you are looking into an external hosted solution ask the company if they can provide a virtual environment. This will shield you from physical hardware issues, when hardware fails or a maintenance event is planned the host can be moved dynamically onto another server. The product we are about to install can do the fail over automatically when an issue happens with the underlying physical server.

This doesnt protect you from corruption in the file system and/or database, however these are much rarer events. The only way to protect from these is regular maintenance.

Just curious, is this using Open Source software under the hood?

Cheers
Rodney

[Old farmer]

My thanks, too, for all your work and a great site.

[mkypenturner]

thank for fixing it go get some well earned rest now :2tsup:

[wheelinround]

I just found that I had ! saved a page of thread from the Ornamental Turning section posts and all. Shame there is no way it could be worked back in...........or is there:rolleyes::U

[Mulgabill]

As they say "S...t happens" Thank you to all involved in the late night recovery process. Been there, done that, but not any more!
Again well done!

[Fencepost2]

Thank you for heroic efforts and concern. Really appreciate this Forum

[Brigalow]

Good to have you back.
Keep up the good work on what has to be about the most informative and friendly web site that I have come across.

[dai sensei]

Great to be back up, and hats off to Steve for the all nighter effort :2tsup:

ps: I have no idea what I posted in the last 24 hrs, so if I can't remember and I have no evidence, it didn't happen :U

[graemet]

I am astounded at the lengths Neil and the team go to in order to maintain our playtime! I'll add my congratulations and thanks to such a dedicated cohort. I don't know what it would be called, but you lot deserve a medal for your efforts.
Thanks heaps!
Cheers
Graeme

[Gunado]

Neil you & steven did a great job in getting it back up in such a short time.

Cheers
Phill

[Christos]

Cool running on the return to normal. :2tsup: Sometimes these things happen. :C

[Calm]

Great work Steven & Neil. As usual our fees to use this forum have not risen - thanks a lot.:2tsup::2tsup:

Only problem is i forgot what was missed - i know Jefferson had some garbage that is better omitted :p:p:oo: but i think i had some of my best work in that time frame. :D:D

Noel did i have any jokes that i need to put back up. The emails would still be here if you need them again. :D:D

cheers

david

[Ironwood]

Many thanks Guys, your efforts are much appreciated :2tsup:

[philf]

Great effort guys, and a big thanks to Steve:2tsup:

[bab600]

Great work guys, on a tremendous job well done

Brian

PS. SWMO says to thanks for the breakdown as she was able to

A. Use the computer a lot earlier

B. See and talk to me for a change:-

THANKS AGAIN

[Christopha]

What forum?

[wheelinround]

What forum?


Hey shows over go back to your fishing till next year :p

[Woodlee]

Thanks to all who got us back up and running .
Now I have a request ,in the tablesaw and combination forum I have two post with the same title ,(New Tablesaw) one posted on 4th July and the other is dated 7th July , could a moderator please delete the 4th July one as it has no replies ,I think it just self produced when the melt down occurred.
Its annoying me because I keep opening it to see if there are any follow up posts and am finding nowt

Much appreciated
Kev.

[RETIRED]

Thanks to all who got us back up and running .
Now I have a request ,in the tablesaw and combination forum I have two post with the same title ,(New Tablesaw) one posted on 4th July and the other is dated 7th July , could a moderator please delete the 4th July one as it has no replies ,I think it just self produced when the melt down occurred.
Its annoying me because I keep opening it to see if there are any follow up posts and am finding nowt

Much appreciated
Kev.Fixed.

[Woodlee]

Thank you .
I salute you for your talent and expedience .

New table saw great forum ,great hard working moderators ,what else could a forumite ask for ?

Kev

[ToothFairy]

I've been sick and out of circulation for a bit - was really looking forward to having a forum catch-up session. I certainly missed a lot of fun! Just want to take the opportunity now to add my belated thanks to Neil and the whole team - for fighting off the lawyers and for keeping this incredible community up and running. Eleven years between problems? Even my shed padlock can't claim that sort of record!

So thanks, guys, for everything you do.

- Michael