• Welcome to Photrio!
    Registration is fast and free. Join today to unlock search, see fewer ads, and access all forum features.
    Click here to sign up

Random site performance issue *update

Status
Not open for further replies.

Sean

Admin
Admin
Allowing Ads
Joined
Aug 29, 2002
Messages
13,500
Location
New Zealand
Format
Multi Format
Hi All,

We've had some random performance problems the last few days as discussed in this thread:

https://www.photrio.com/forum/threads/slow-response-time.165232/

We go from lightning quick to nearly unresponsive, anywhere from 30mins to several hours. I've been monitoring for the last 24hrs with the web host techs and we've identified the issue. It appears a rogue search index bot is hitting the site with millions of requests during this time.

Next steps:

I have installed a new tool that serves unknown search bots a blank white html page. I'm hopeful this is all we need and will boost our performance overall.

I have the ip address of the rogue bot and am investigating where it's from.

I'll keep you posted.

Thanks for your patience,
Sean
 
OP
OP

Sean

Admin
Admin
Allowing Ads
Joined
Aug 29, 2002
Messages
13,500
Location
New Zealand
Format
Multi Format
The offending bot:



I have just tested accessing the site as this bot and got the blank page, so the tool is confirmed to block this bot. I'm still investigating..
 
OP
OP

Sean

Admin
Admin
Allowing Ads
Joined
Aug 29, 2002
Messages
13,500
Location
New Zealand
Format
Multi Format
I've contacted dotbot and they claim their bot will obey robots.txt entries. I've added:

User-agent: dotbot
Disallow: /

This should prevent the bot from even coming here, but may take a couple of days until then. In the meantime, the bot will only get blank pages.

Hopefully, this is the end of the performance issues.
 
OP
OP

Sean

Admin
Admin
Allowing Ads
Joined
Aug 29, 2002
Messages
13,500
Location
New Zealand
Format
Multi Format
We are still having some intermittent issues with the server. I'll post updates as I get them. Thanks
 
OP
OP

Sean

Admin
Admin
Allowing Ads
Joined
Aug 29, 2002
Messages
13,500
Location
New Zealand
Format
Multi Format
We've identified some possible memory issues, more details soon...
 
OP
OP

Sean

Admin
Admin
Allowing Ads
Joined
Aug 29, 2002
Messages
13,500
Location
New Zealand
Format
Multi Format
We are having some difficulties with the server. We've identified some memory problems and have placed an urgent request for a memory upgrade. Performance may be erratic until the upgrade is complete, there may be some downtime when the module is physically added to the server. Thanks for your patience, Sean
 
OP
OP

Sean

Admin
Admin
Allowing Ads
Joined
Aug 29, 2002
Messages
13,500
Location
New Zealand
Format
Multi Format
Ok, we're up and running with the new memory:



The techs will now do some new allocation to ensure we don't hit any walls again. Fingers crossed it's smooth sailing now.
 
OP
OP

Sean

Admin
Admin
Allowing Ads
Joined
Aug 29, 2002
Messages
13,500
Location
New Zealand
Format
Multi Format
Looks like we're still having some issues...
 
OP
OP

Sean

Admin
Admin
Allowing Ads
Joined
Aug 29, 2002
Messages
13,500
Location
New Zealand
Format
Multi Format
Ok, some more tweaks so will continue to monitor.
 
OP
OP

Sean

Admin
Admin
Allowing Ads
Joined
Aug 29, 2002
Messages
13,500
Location
New Zealand
Format
Multi Format
We had a good run for a while but I noticed issues again when I woke up this morning, have put it back to the techs. These issues can be hard to troubleshoot when intermittent but hopefully more progress today.
 
OP
OP

Sean

Admin
Admin
Allowing Ads
Joined
Aug 29, 2002
Messages
13,500
Location
New Zealand
Format
Multi Format
Site seems ok now that we're out of peak time, will allocate more resource to the site for the next peak and see how we go. Also, I am bringing in a Linux/Xenforo expert to do a full server optimisation. This should take place in the next 3 days.
 
OP
OP

Sean

Admin
Admin
Allowing Ads
Joined
Aug 29, 2002
Messages
13,500
Location
New Zealand
Format
Multi Format
Still going good, next test will be peak usage tomorrow and how we handle it. We've had some new settings put in tonight to take full advantage of the extra memory upgrade. This should more than triple our capability during peak load times.
 
OP
OP

Sean

Admin
Admin
Allowing Ads
Joined
Aug 29, 2002
Messages
13,500
Location
New Zealand
Format
Multi Format
Ok, we blasted through peak times with plenty of resource and approached no limits, things should be ok now (knock on wood).
 
Status
Not open for further replies.
Menu
Log in
Register