PEQ Grand Creation Down?

[spoon]

When the machine came back up today, we noticed our /home directory had reverted to a state it was when this machine was first built.

Are you meaning that you had mounted a partition to /home and now that partition no longer mounts so you have an empty /home directory? That would make sense if that partition was on part of the array that failed.

At first glance, it appeared a drive in the array failed, and the mirror failed to rebuild the data properly.

It should be fairly easy to see if a drive failed right? Just check /proc/mdstat and /var/log/messages. But what do you mean it failed to rebuild the data? It sounds like you didn't replace the broken drive yet, so how would you "rebuild the data?" Did removing the broken drive from the array not work?

# mdadm --manage /dev/md0 --fail /dev/sdb0

Is that not working? Maybe thats what you mean by

Further investigation shows we have a controller that isn't responding to the kernel at all.

What we might lose, is the website code. The forum code, all of our tools, editors, scripts, etc. It can be reconstructed, but it will be painful ... Misc internal scripts, PEQBot, and other such things that made my life easier may also be on the chopping block.

If it does come to that, why don't you post a list of what you need and I'd be willing to try and get a couple of those items taken care of.

Thanks for all your work!

[Fingel]

What about projecteq.net/quests ? that was probably my favorite feature of PEQ. Is that one of the website parts that are going to take a lot of work to rebuild?

[renoofturks1]

No worries. If it's fixable, then great. We will rebuild.

If not, yes, the offer still stands cave, if that is what you are referring to.

[cavedude]

Spoon: My terminology was a bit confusing, I'm sorry. "rebuilding" is generally reserved for RAIDs with parity. I think what it came down to is without pointing any fingers, the mirror at some point failed us. When the machine was turned on, the one controller wasn't working, so Linux used the mirror drive for /home which isn't current. I was only given partial information, we were more concerned with damage control than why did this happen last night. Either way, once it comes time to decide permanent solutions, I'm going to push for a hardware RAID5 (I would love RAID6, but that's a dream!)

Fingel: The quest status data (which quest are completed, the authors, etc) is safe. The PHP page, however may be lost and would need to be rewritten.

BoginDA: We're not going to worry about donations or hardware until we have more information and when FNW comes back from his trip. Thank you for the nice offer!

renoofturks1: Yeah, we may need to take you up on your offer as a failsafe plan. Thank you very much! I'll contact you and Cantus through PM.

I would like to make it clear that I am going to do everything I can to ensure the server is up by Monday, so sit tight guys, we're almost there!

[cavedude]

We're back up, and didn't lose a thing! Enjoy!

The website is currently down, but that looks to be a network error.

[elation]

Sweet! Just in time for the weekend. So much cleaning house, getting my car fixed, eating, etc.

[So_1337]

Oh man! Such great news!

What wound up being the problem?

[BoginDA]

Cavedude, keep us informed as to the servers status. If you do indeed need to purchase new hardware let us know. I don't have tons of money, which is why I play free EQ, but I would not hesitate to paypal you guys a donation to help cover the cost of getting the server back on its feet. If the website comes down again post a link to your paypal here.

BoginDA