APUG database crashed, had to restore from backup..

[Sean]

Hi All,

Just when things were starting to ease back into place the most obscure of things crashed the entire database. In my efforts to find and purge old code that was possibly giving us issues, I removed an old plugin. Nothing unusual and something I have probably done 300 times over the last decade. Often if removal of such a plugin code gives grief it is easy to recover. In this case it crashed a key area of the database used to iron out any glitches, something I had never seen before. I had a vbulletin tech on hand a couple hours later who had also never seen such a thing happen. Anyway, after failing to recover the database I fell back to my backups from around 3hrs before. So, you may have some loss of data. As far as I can tell the restore has been successful.

Times of data loss April 23rd:
Database Backup occurred at roughly 7pm UTC
Database crash occurred at roughly 10:25pm UTC
Data between 7pm-10:25pm will be missing
here is a time zone calculator if needed: http://www.timeanddate.com/worldclock/converter.html

The good news is I've made contact with a highly skilled vBulletin tech who is available for some work. After catching up on some sleep I will be turning over some of the big issues to him.

I am sorry for the recent issues with APUG stability. I will continue to do my best to ensure this is sorted out as soon as humanly possible. Thanks again for your patience.

Sean

 

[andrew.roos]

Congratulations on having an effective disaster recivery plan in place. It could have been so much worse.

 

[k_jupiter]

According to your time zone calculator link, your database mishap won't occur for three and a half hours. Plenty of time to get it fixed.

I feel like i am in an episode of Dr. Who.

tim in san jose

 

[Sean]

I don't want to get my hopes up yet, but it almost seems the database dump file and import of the backup purged some corruption. As of now the site is performing normally. Before the database restore it took me 20 seconds to get into my PM box (now it's less than a second), it also took gallery images 5-7 seconds or so to load (now within 2). Also our response times are averaging 102ms which has never been this low before (prior to the crash 500ms). If anything it will be interesting to see how long this performance lasts. Will keep an eye on it, but for now.. sleeeeeeep ZzzzzZzzzz.....

 

[MattKing]

I blame the deleted thread - or Aristophanes

 

[FiatluX]

I´ve seen several vbulletin forums crash or being hacked lately, seems that one of the ways is a java enabled iframe! Check any suspicious code as the other forums have had unwelcome exploiters that inserted malicious code which spread viruses to the visitor computers

 

[TheFlyingCamera]

Sean-

there are still some wonky bits - see the Joke thread for example -

(there was a url link here which no longer exists)

It says there is page 454, but when you click on the link to it, it returns you back to 453.

 

[hrst]

Sean-

there are still some wonky bits - see the Joke thread for example -

(there was a url link here which no longer exists)

It says there is page 454, but when you click on the link to it, it returns you back to 453.

Same with (there was a url link here which no longer exists) from
(there was a url link here which no longer exists) ; the top forum shows "tars" has made the newest post as of today, but Page 11 redirects to Page 10 and this new post cannot be seen.

And wow, thanks for the quick response to the problem, once again. It seems you really care about this site.

 

[MattKing]

Same with (there was a url link here which no longer exists) from
(there was a url link here which no longer exists) ; the top forum shows "tars" has made the newest post as of today, but Page 11 redirects to Page 10 and this new post cannot be seen.

And wow, thanks for the quick response to the problem, once again. It seems you really care about this site.

It must have something to do with how user preferences are set. I have increased the number of posts per page (to 40), and I don't have any problems with all three of the example threads.

I have in the past, however, had similar problems with other threads.

 

[Marco B]

I am sorry for the recent issues with APUG stability. I will continue to do my best to ensure this is sorted out as soon as humanly possible. Thanks again for your patience.

Sean

Hi Sean,

I know I don't just speak for myself here, and you've heard it before, but in keeping accordance with one of the above post's SF "Dr. Who" reference: I think your dedication to support the forum considering it's size and complexity, is more akin to "super-human" ...

Good to hear the backups did their job. I once worked for a major government organization running a huge spatial database with unfortunately outdated and even end-of-life soft- and hardware (one of the reasons I was working there: to provide insight in and plan for a major change and complete replacement of the systems with more modern equivalents). During some routine work, some person noticed the backup hadn't run properly for months on end... Fortunately, the main system had...

Marco

 

[Sean]

It must have something to do with how user preferences are set. I have increased the number of posts per page (to 40), and I don't have any problems with all three of the example threads.

I have in the past, however, had similar problems with other threads.

That's interesting, I might need to adjust the default for that. Good spotting

 

[Sean]

Ok am back on line now. Thanks for your support and reporting on a few glitches. The server has been running very good after the database re-import. Response times still under 200ms. I think the database backup and following restore must have purged some deep corruption that the standard database tools could not get rid of. Here's hoping we have some stability for a while and can simply improve upon that. Looks like the crash, although stressful may have helped in the end. We'll see how it goes. There are still several issues with bad database queries (this is from poorly designed code or add-ons). I have a guru coder available to make new, more efficient codes that put far less strain on the server. I will setup a clone of APUG on a hidden domain for any further testing.

Thanks

 

[polyglot]

One more bit of corruption: clicking the "My Posts" link is showing up a couple of threads ("Identify this old Gitzo" and your friend's masters research thread) that I've never posted in.

 

[Sean]

i probably need to rebuild the search since we restored will do that in a few.

 

[Sean]

For those having trouble getting to the last page of some threads, I found the fix for that. A vBulletin utility is running right now that updates the thread/page counts. This will probably need to run for another 10mins before it completes, then the last pages should work fine. If not let me know. Thanks

*Actually just noticed, still some weirdness there with not showing additional pages in search results, investigating

 

[Sean]

search indexes have been rebuilt, any weirdness with "my posts" "my threads" should be ok now. At the moment I cut results back to 100 max, but will increase this again soon.

 

[Sean]

Ok I think all searches are working properly now (my posts) etc

 

[eddie]

Sean- Thanks for all you do.

 

[ozphoto]

Hi Sean, not sure if you've discussed previously, but I can't customise the sidebar any further than having "Recent Classifieds" appear, even though I've selected "My Recent Posts" to appear.

 

[Sean]

Hi Sean, not sure if you've discussed previously, but I can't customise the sidebar any further than having "Recent Classifieds" appear, even though I've selected "My Recent Posts" to appear.

Yeah some of the sidebar has been temporarily disabled. Several of the blocks were pegging the database with ridiculous amounts of queries. I have a coder tasked to redo a few blocks so they use far less resource.

 

[Ross Chambers]

Dear Sean, please come over and work for my bank. When their online facility goes under they don't explain, don't apologise, don't seem to worry unduly at all; just ride it out.

Many thanks for your hard work.

 

[FiatluX]

Awesome work, thanks!