Bit of grief with site stability today..

Sean · Feb 21, 2005

Hi guys, for some reason the application blocking all of the search and spam bots quit working (leaving us vulnerable to bot attacks again). I have raised a call with that company for help and am just waiting to hear back. In the meantime it is a game of cat and mouse to manually ban the bots giving us trouble until I can get the app back up. I may have caught the few giving us issues today, if not we'll have to go offline again. Thanks, Sean

Kevin Caulfield · Feb 21, 2005

Thanks, Sean, for getting it up again, and also for providing the temporary chat room for we desperadoes.

Sean · Feb 21, 2005

*update*

I haven't gotten the bot blocking application back on yet, but I did do a few things that seem to have us stable (cleaned up some code, removed old unused tables from our database, and ran a full repair against all tables in the database). So far it's been over an hour without an issue. I have re-opened the gallery as well. Thanks

Michael Mutmansky · Feb 21, 2005

Sean,

Thanks for the effort...

I was wondering, would it be possible to eliminate the bots by making everyone go through a hompeage where the access is granted through an encoded password like those used when people are registering an ID on some places (you know, the code is a graphic with fuzzy lines and other junk so a computer can't automatically read the file)?

Then, you change the access on a semi-random pattern, with semi-random codes. Once you enter the code, it updates your cookie, giving you access to the entirety of the site. The next time you try to enter, the code won't be up to date, and you'll have to manually enter a new code. If the code changes while someone is browsing, it'll pop up the access page when they try to load a new page after the code change. Manually enter the new code, and you're on your way.

I'm not a programmer, so tell me how stupid of an idea that is.

---Michael

Sean · Feb 21, 2005

Well, that would probably work, but it would then block the site to all visitors and guests. It would be quite easy to totally lock the forum and gallery so that only registered users can access them, the big drawback though is denying visitor access. My long term plan is to get all bots under control, then open us to a few specific bots every now and then (yahoo, google) just to keep our indexes on the net. The net is turning into a warzone though, so I would not be surprised if many sites start opting for a registration only access model to keep all the bots and spammers off their sites..

Baxter Bradford · Feb 22, 2005

Are you sure that the threat was purely external?

It might be that the site itself was expressing its distress about all of the opinionary fisticuffs and fireworks that were happening over in "Ethics and Philosophy". I'm surprised Sean that you weren't contacted by Don King to try to get that thread (where academic brilliance threatened to overtake the thread subject matter) as pay per view..........

Digidurst · Feb 22, 2005

Hi Sean You probably already know this but just in case, I'm not getting anything when I click on the links for either 'Chemistry Recipes' or 'Articles'.

Thanks for working so hard to get the site back up! Take care and cheers!

Sean · Feb 22, 2005

Digidurst said:
Hi Sean You probably already know this but just in case, I'm not getting anything when I click on the links for either 'Chemistry Recipes' or 'Articles'.

Thanks for working so hard to get the site back up! Take care and cheers!

Thanks for the heads up, it's fixed now

Ed Sukach · Feb 22, 2005

Baxter Bradford said:
Are you sure that the threat was purely external?

It might be that the site itself was expressing its distress about all of the opinionary fisticuffs and fireworks that were happening over in "Ethics and Philosophy". I'm surprised Sean that you weren't contacted by Don King to try to get that thread (where academic brilliance threatened to overtake the thread subject matter) as pay per view..........

Ah, yes. There is a LOT of "Making waves" over there. Consider, though; the waves are what keeps the water clear. Without them --- well ... think of a swamp.

I had just completed a brilliant, and involved, post when .... "Pardon the Inconvenience While ...", came up on the screen. Not a "bad" thing though - gave me - FORCED me - to reflect on what I was writing, and being concise.

Will S · Feb 22, 2005

Sean said:
Hi guys, for some reason the application blocking all of the search and spam bots quit working (leaving us vulnerable to bot attacks again). I have raised a call with that company for help and am just waiting to hear back. In the meantime it is a game of cat and mouse to manually ban the bots giving us trouble until I can get the app back up. I may have caught the few giving us issues today, if not we'll have to go offline again. Thanks, Sean

I thought that robots.txt just worked all on its lonesome and that the bot read it and processed accordingly. These must be bots that aren't adhering to the standard? Or did someone remove your files?

Good luck with that. I once ran a site in the early 90s that kicked off large database queries in a lot of the links. We got brought down by Google for about a week. And this was before we had any means of telling the spider to stop! We had to call Google and ask them to quit it, and they wouldn't kill their jobs once they had started, etc., etc. Thankfully the robots.txt method showed up shortly thereafter (this was in 1994).

Best,

Will

Sean · Feb 22, 2005

Hi Will, yeah robots.txt seems quite useless these days..

Sean · Feb 22, 2005

*UPDATE*

Hello all, rather than leave the site down, I have made it so registered users have access. This means searchbots & spambots which are categorized as "guests" will not be able to attach to us... which means we should be ok. I'll keep an eye on things. Unfortunately the day job is calling so hopefully this will keep us going until I can resolve the other issues. Thanks

Sean · Feb 22, 2005

Might be ok now

The robot blocking application is back in action, and I'm shocked to see it has blocked about 75 Inktomi Slurp spiders from our forums! This is a Yahoo spider system. I like us being indexed on the internet but these search sites like Yahoo, google, msn, need to learn how to control their systems! Anyway, things look ok right now, time will tell. Thanks

titrisol · Feb 22, 2005

Can the RSS feed have something to do with it?
I added apug to My Yahoo! for one

Sean said:
The robot blocking application is back in action, and I'm shocked to see it has blocked about 75 Inktomi Slurp spiders from our forums! This is a Yahoo spider system. I like us being indexed on the internet but these search sites like Yahoo, google, msn, need to learn how to control their systems! Anyway, things look ok right now, time will tell. Thanks

Sean · Feb 22, 2005

Nope, it's definitely search spiders. There are still some issues to work through so no guarantees we are 100% up. The robot control application is running in a limitted way until I can get hold of their support again..

Sean · Feb 22, 2005

*update..

I hate to say all is well.. again but i did make progress in getting the spider blocking app fully functional. I noticed all the Inktomi Slurp spider listings showing in our guests list have dropped off. We'll see how it goes.

Flotsam · Feb 22, 2005

Did you really change servers again?

Bit of grief with site stability today..

Sean

Admin

Kevin Caulfield

Sean

Admin

Michael Mutmansky

Sean

Admin

Baxter Bradford

Digidurst

Sean

Admin

Ed Sukach

Will S

Sean

Admin

Sean

Admin

Sean

Admin

titrisol

Sean

Admin

Sean

Admin

Flotsam

Bit of grief with site stability today..

Admin

Admin

Admin

Admin

Admin

Admin

Admin

Admin

Admin

Privacy & Transparency

Privacy & Transparency