APUG.ORG OUTAGE NOTICE

[Sean]

APUG.ORG OUTAGE NOTICE

We are recovering from catastrophic hardware failure which had also
unfortunately corrupted our primary data and backup data (worst case scenario pretty much). We have been working around the clock to get the site back up. Currently we have recovered a good filesystem from the 15th and good databases from just before the crash. The problems began with faulty hardware, surprisingly on a brand new high end server. Possibly a combination of faulty ram and a faulty primary disk. Over the past day or so the owner of the hosting company has personally worked with me and other
unix techs on the issue for over 30hrs straight. We are now on a new
server and partially up and running. The host is very distraught over this outage and is sending the server back to Dell demanding a full analysis and 'post mortem' as they call it in IT. He has also given us a month free of hosting. This is some horrible bad luck we've had and luckily our provider did not just give us a "sorry, your data is corrupted nothing we can do" and leave us hanging. They spent about 12hrs working on a disk to recover our databases not to mention setting up a new server as quickly as possible. Needless to say many new things will be put in place to ensure a better disaster recovery system for us. Still working on things so better run. Thank you for your patience, very sorry for the length of the outage.

Sean

 

[MurrayMinchin]

Quit being a Hero and go have a nap

Murray

 

[JLP]

Perfectly said Murray.
Keep up the good work Sean but don't burn the candle in both ends.

jan

 

[DBP]

Take a breather. At a certain point working becomes less productive than resting.

 

[ricksplace]

I was told once the best way to fix a faulty server after a catastrophic failure such as you describe.

12 guage with 00 buckshot.

Thanks for being there when Murphy's Law strikes.

 

[CRhymer]

Thank you for all your work. One really appreciates what this site has to offer when it is offline for even a short while. I was starting to get the DT's. Hope you get a rest soon.

Cheers,
Clarence

 

[copake_ham]

Bummer to be sure. Glad site is back up and thanks for working so hard to do so. I missed this place while it was down.

 

[Jim Chinn]

APUG.ORG OUTAGE NOTICE

We are recovering from catastrophic hardware failure which had also
unfortunately corrupted our primary data and backup data (worst case scenario pretty much)....

The problems began with faulty hardware, surprisingly on a brand new high end server. Possibly a combination of faulty ram and a faulty primary disk....

The host is very distraught over this outage and is sending the server back to Dell demanding a full analysis and 'post mortem' as they call it in IT. Sean

No wonder Dell's stock price took a major dive into the crapper on Friday!

Thanks for all your efforts Sean. When the gremlins hit, it always seems to be worst case scenario in my experience.

 

[Bill Hahn]

Ditto to what CRhymer said.

 

[Donald Miller]

Thanks Sean, Your efforts are appreciated. Have some well deserved rest.

 

[naturephoto1]

Sean,

Sorry for your headache, but as others have said, I missed this place. Hopefully you can get some rest soon and you do not experience anything like this again any time soon.

Rich

 

[Joe Lipka]

Well, as the IT folks will tell you, the failures either happen right away or much, much later. (But you knew that.) Thanks for your effort in getting life back to "normal" here.

 

[Photo Engineer]

I would like to applaud Sean and his networking organization for diligent and dedicated work reconnecting us all!

Thanks.

PE

 

[bnelson]

I for one missed this site when it was down. Keep up the good work and thanks for working on this hard problem

 

[papagene]

A great big THANK YOU to you Sean. Again you proved the BEST forum operator around.
Get some rest... spend some good time with your family... and then go out and expose some film!!

gene

 

[David A. Goldfarb]

Thanks again, Sean, for getting APUG back up and running!

 

[Sean]

Thanks for the comments everyone, hopefully we are in the clear now.

 

[papagene]

cool... even the chatroom is working!!

gene

 

[reellis67]

I'd like to add my thanks for your hard work, this site is an oasis in the desert. Thanks as well for all the work we don't hear about. (I will refrain from giving my opinions of Dell however...)

- Randy

 

[naturephoto1]

Randy,

Don't forget, Dell only assembles computers from parts from many different sources. Others make the parts that fail (ed) and they should the accountable party. We will not know what failed until the unit is analyzed.

Rich

 

[reellis67]

True, but I deal with their problems on a daily basis (117 servers in the data center to date, 23 more on order) so I've seen their idea of quality control. They sell a product with a warranty and a claim of a certain level of performance, and in my experience, they are overly optimistic about that level of performance. Just because they don't make to component doesn't mean they have no responsibility for failures.

- Randy

 

[wilsonneal]

Wanted to say that while I missed access to the site, the outage wasn't inconvenient. No worries. Thanks for working to get it back up so quickly.
N

 

[Q17]

Thanks for all of your hard work, Sean! Excellent job getting it going again! Although I missed having the site here, it prompted me to get out, shoot some photos and go to the Diane Arbus show.

Now, get some sleep!

=michelle=

 

[Krockmitaine]

Thanks a buddle for all the efforts to put everything back to normal. Must have been quite a stressful day that you had. You've earn the respect of everyone.

And get some sleep, we'll survive.

Marc

 

[John Bartley]

Absolutely amazing !! The dedication that Sean gives in support of a forum that (for most of us) is a recreation site is above and beyond praise.

Thank you !!