APUG.ORG OUTAGE NOTICE

[david b]

Sean,
thank you for all of you hard work.

I quickly found out how much I enjoy APUG and how much time I actually spend here.

Yikes.

 

[arigram]

Damn, APUG is back up!
...and I was starting to have a life again...

 

[Guillaume Zuili]

Thanks for the hard work. APUG is back !

 

[dlin]

Thanks Sean for your dedication. Truly heroic.

 

[roteague]

True, but I deal with their problems on a daily basis (117 servers in the data center to date, 23 more on order) so I've seen their idea of quality control. They sell a product with a warranty and a claim of a certain level of performance, and in my experience, they are overly optimistic about that level of performance.

Tell me about it. I bought a Dell laptop, at the time a top of the line model. It went through 3 motherboards in less than 2 years. The first two were replaced under warranty, but third motherboard they wanted $500 to repair. It was their poor design, but they still wouldn't stand behind their product. Needless to say, I will not buy another Dell. FWIW, last quarter Dell reported lower earnings than they expected - not surprising, given their poor customer service.

 

[roteague]

Thanks Sean for all your hard work and dedication.

 

[MikeM1977]

Although failures happen to everybody, this is why there is still a very large market for expensive machines from Sun, IBM, and other providers who do more than assemble x86 boxes.

 

[JeffD]

Good time to review the backup procedure. I can't figure out how a failure on the server would screw up your backup, but I guess it did. I started backing up my web server off site daily, after an event similar to yours convinced me it was essential. Now I sleep better at night.

 

[Sean]

Good time to review the backup procedure. I can't figure out how a failure on the server would screw up your backup, but I guess it did. I started backing up my web server off site daily, after an event similar to yours convinced me it was essential. Now I sleep better at night.

Well I had 4 backups in play luckily and between them we were able to recover everything.

1 backup on server's secondary drive (it was partially corrupted as it was a copy of the corrupt primary disk)
1 decommissioned server in california (it's data was good but from the 15th)
1 off site full rsync backup (data was good but only had apug.org and was from the 17th)
1 full site backup from the 19th on my local pc with good data but hard to transfer with my slow NZDSL connection

The problem is most of the data was corrupt in production, and that corrupt data then copied over to the nightly backups which made the backups corrupt. We had no indication there was an issue until the crashes. When you have about 6.5gigs of data in the backup you never truly know it's good until you try to use it. I never anticipated this because usually a primary disk dies and you fall back to the secondary with last nights data. In this case last nights data was also bad. Luckily we had a variety of backups + acquiring some still good db's off the old disks.

In the future I plan to not overwrite nightly backups anymore, for example maybe have 15days worth of backups on the backup disk. If data goes bad we can keep going back a day until it's clean (hopefully always having a recoverable set of databases less than a day old). I'll be examining a lot of options. At this very moment I have a newly generated zip of the entire site transferring to my offsite storage and my local pc. I'll then take nightly database backups until the new solution is in place..

 

[Sean]

-also working on a failover page so in the event of an outage we can have a notice with temporary chatroom.

 

[Markok765]

-also working on a failover page so in the event of an outage we can have a notice with temporary chatroom.

Would the chatroom be open only to subscribers? Thanks for the hard work!

 

[mikebarger]

Thanks Sean, I thought maybe I was on everyone's ignore list

Mike

 

[MikeM1977]

Well I had 4 backups in play luckily and between them we were able to recover everything.

What is your database? MySQL?

 

[Aggie]

Dell Sucks! (I like the word sucks it is so to the point) I too have had one of their laptops, which was delivered without all the parts ordered on it. Everytime I had to try and get the proper part they would tell me "You get what you pay for" to which I would reply, "No I didn't that is the problem." It took 8 months for them to send me the correct parts, and then it became a doorstop due to multiple failures withint another 8 months. My son had a desktop from them. He had more problems than I did. He finally gave up and built one of his won from scratch.

 

[Sean]

What is your database? MySQL?

yup

 

[Sean]

Dell Sucks! (I like the word sucks it is so to the point) I too have had one of their laptops, which was delivered without all the parts ordered on it. Everytime I had to try and get the proper part they would tell me "You get what you pay for" to which I would reply, "No I didn't that is the problem." It took 8 months for them to send me the correct parts, and then it became a doorstop due to multiple failures withint another 8 months. My son had a desktop from them. He had more problems than I did. He finally gave up and built one of his won from scratch.

I hear ya, unfortunately they've got the mid-range server market cornered. All you can do in my price range is roll the dice and hope your server wasn't assembled on monday! The last 2 dells we had were pretty reliable and all of our previous outages were network and 1 hard drive failure. Fingers crossed this one does ok..

 

[Sean]

Would the chatroom be open only to subscribers? Thanks for the hard work!

No, the emergency chatroom is always just an open java chat.

 

[MikeM1977]

I hear ya, unfortunately they've got the mid-range server market cornered. All you can do in my price range is roll the dice and hope your server wasn't assembled on monday! The last 2 dells we had were pretty reliable and all of our previous outages were network and 1 hard drive failure. Fingers crossed this one does ok..

They may command the market, but there are other options. Sun even offers some nice mid-range Opteron servers. You at least buy machines with ECC memory, right?

I've personally seen data corruption with MySQL and have heard it from others too. Doesn't seem terribly robust. I guess you get what you pay for (which in the case of MySQL, is quite a bit

 

[removed account4]

great work sean!

thanks a million!
-john

 

[sienarot]

yup

Do you have update logging enabled?

The webmail database for the university I work for failed on me one time and was corrupted beyond repair. Along with our nightly backups, we also store all update logging on a separate log server (a year's worth of logs). When our database failed, we recovered the backup from the night before and applied all updates right up until the time of the corruption.


By the way, great work!

 

[roy]

Well done Sean, I was wondering what had happened. Makes my mail lost in Outlook Express mentioned elsewhere look a puny problem !

 

[blokeman]

........ and thanks from me too Sean & thanks for coinciding perfectly the breakdown with my four day trip to the country!

 

[CarlRadford]

You have my...

Respect & Gratitude!

 

[pentaxuser]

Sterling work. Anyone wondering whether to continue to subscribe should have found this focussed their attention as they say, when they had to manage for about 36 hours without APUG. So if we all realise its value even more than before then some good may have come out of it.

By the way can I take it that it is all back to normal in terms of running speed etc. The reason I ask is that I have already lost one reply when attempting to send to Arigram and the response rate has gone back to the very slow rate it was at before the changeover.

It is certainly not back to what it was but this may be coincidentally my ISP provider.

Thanks

pentaxuser

 

[brimc76]

Thanks Sean.