Random site performance issue *update

[Sean]

Hi All,

We've had some random performance problems the last few days as discussed in this thread:

https://www.photrio.com/forum/threads/slow-response-time.165232/

We go from lightning quick to nearly unresponsive, anywhere from 30mins to several hours. I've been monitoring for the last 24hrs with the web host techs and we've identified the issue. It appears a rogue search index bot is hitting the site with millions of requests during this time.

Next steps:

I have installed a new tool that serves unknown search bots a blank white html page. I'm hopeful this is all we need and will boost our performance overall.

I have the ip address of the rogue bot and am investigating where it's from.

I'll keep you posted.

Thanks for your patience,
Sean

 

[Sean]

The offending bot:

I have just tested accessing the site as this bot and got the blank page, so the tool is confirmed to block this bot. I'm still investigating..

 

[Sean]

I've contacted dotbot and they claim their bot will obey robots.txt entries. I've added:

User-agent: dotbot
Disallow: /

This should prevent the bot from even coming here, but may take a couple of days until then. In the meantime, the bot will only get blank pages.

Hopefully, this is the end of the performance issues.

 

[Sean]

We are still having some intermittent issues with the server. I'll post updates as I get them. Thanks

 

[Sean]

We've identified some possible memory issues, more details soon...

 

[Sean]

We are having some difficulties with the server. We've identified some memory problems and have placed an urgent request for a memory upgrade. Performance may be erratic until the upgrade is complete, there may be some downtime when the module is physically added to the server. Thanks for your patience, Sean

 

[Sean]

Ok, we're up and running with the new memory:

The techs will now do some new allocation to ensure we don't hit any walls again. Fingers crossed it's smooth sailing now.

 

[Sean]

Looks like we're still having some issues...

 

[Sean]

Ok, some more tweaks so will continue to monitor.

 

[Sean]

We had a good run for a while but I noticed issues again when I woke up this morning, have put it back to the techs. These issues can be hard to troubleshoot when intermittent but hopefully more progress today.

 

[Sean]

Site seems ok now that we're out of peak time, will allocate more resource to the site for the next peak and see how we go. Also, I am bringing in a Linux/Xenforo expert to do a full server optimisation. This should take place in the next 3 days.

 

[Sean]

Still going good, next test will be peak usage tomorrow and how we handle it. We've had some new settings put in tonight to take full advantage of the extra memory upgrade. This should more than triple our capability during peak load times.

 

[Sean]

Ok, we blasted through peak times with plenty of resource and approached no limits, things should be ok now (knock on wood).