StopBadBot significantly slows down website

Hi Michael,

A couple of months ago I have a problem with one site and they use Cloudflare.
Their firewall blocks our firewall.
Check also if your hosting have a firewall enabled.

Here are our troubleshooting page:
https://siterightaway.net/troubleshooting/

Check if your MySql config file if fine.

We have around 30 WordPress site, all load very quickly with our plugin.

For example:

https://www.StopBadBots.com
https://www.AntiHackerPlugin.com
https://www.siterightaway.net

Cheers,
Bill

Thanks for the reply. No Cloudfare, no firewall in the way. MySQL is perfectly configured.

Can you please forward this to a developer? He needs to install the Query Monitor plugin and he will immediately see the problem.

Note that this is a problem for large sites. The sites you linked to are tiny.

• This reply was modified 3 years, 11 months ago by Michael.

I’m the developer.
I will install Monday the Query Monitor plugin, check and let you know later.
Talk with your hosting company if you need a faster server.
We have one server with more than 30 WordPress sites and around 69 non WordPress sites and run without problem. We have many users running many sites.
We have versions for non WordPress, Joomla and docker.

Trust me, this is not our server. Our systems run in AWS and we have plenty of CPU power. Without StopBadBots there is very little CPU load. When we activate StopBadBots MYSQL CPU load goes through the roof on a busy day. Query Monitor makes it obvious what is happening.

Just because a site appears to load fast when you open it in your browser, doesn’t mean that your plugin does not slow down the site. In a professional / commercial environment you need to measure things and there is no doubt that StopBadBots is not fit for large sites.

You need to add a feature that allows admins to disable all logging. Only write into the database if you absolutely must. Imagine what is happening with the database system on a site with hundreds or thousands of simultaneous sessions.

Measure the time for your UPDATE wp_sbb_visitorslog query and you will see.

Needless to say that we would really love to use StopBadBots. However, in its current state it is nothing for commercial sites.

Thanks for looking into this!

Hi Michael,

We ran the plugin Query Monitor and all looks fine in our server.
Anyway, we made some improvements to try to help you (and all users):

1) We included one control to let you keep the visitors log table with only one day. (or more). Look at plugin settings page. Remember: you need keep WordPress cron job system working.

2) We created 2 new indexes on that table to let the mysql work easier.

3) We improved that query.

After that, we feel the query run faster.

In this moment, we can’t remove the visitors log table, because we need that to control the number of visits and block bots.

After that, if your mysql server overload, you can talk with AWS to improve your instance. Or, you can look for one external service, where you change your DNS and send the user to one external firewall to do the job, then the firewall block or send back the user to you. That because we are unable to make the customization requested this year.

I hope this new version can help you.

Marry Christmas, happy holidays.

Cheers,

Bill

Bill, thanks a lot for looking into this so quickly. Unfortunately, we still get the same result even though we enabled the 1 day logging option.

There are two queries that can take up to 0.5s each. This means that StopBadBots alone sometimes needs 1s to deliver a page. This is very slow which is why Query Monitor lists those queries under Slow.

To give you an idea, our page load time with about 60 active plugins is about 0.3s. So StopBadBots slows down our page loads by more than 300%!

By the way, our Visits Log shows no entries and this message:

Showing 0 to 0 of 0 entries (filtered from 307,455 total entries)

Seems there is a bug here.

The two slow queries are in functions/functions.php in line 225 and 227. It seems what makes StopBadBots so slow is writing to the log.

If we disable those 2 queries everything appears to be ok. I wonder why you need to log Human access? Does it hurt if we run StopBadBots without those 2 queries?

Hi,

It is no bug. Visitors Log don’t find records because you disable the queries and information is not complete.

We need take one week to redesign the plugin and remove all that logs. We have no time available now, because we have around 20 other plugins and themes to take care.

After that done, the plugin will loose many security check functions.

If you really need this, please, visit our support site and we can create an estimate to you. We can’t use WordPress system for that.
https://billminozzi.com/support

I suggest also you look for one firewall as service and use their CPU and queries to protect your site. Then, that doesn’t overload your current server. They will request to you to change your DNS and send the users to them. Then, they run a filter and send back or block. Usually this solution can increase your page load in around 3 seconds or more.

Or talk with amazon and request to them increase the power of your database server, then you can get a better hacker/bot protection without pay for external services.

No pain, no gain!

Cheers,
Bill

Bill, thanks!

Of course, we already have a firewall. We only use your sbbcrawlerDetect function in our own scripts for a couple of things (for instance, to detect certain bots and block their IPs at the network level with iptables). We disabled all blocking done by StopBadBots.

We are working with a high compute EC2 instance and even during prime time we have 90% CPU capacity available. Switching to the next faster instance would cost several thousand dollar more per year. I think you understand that it doesn’t make sense for just one plugin. It would be a lot cheaper to just let the bots do what they want.

In the meantime, we disabled those 2 queries. It seems to solve the problem for now.

I hope you can fix this. I bet many other big sites would love to use and buy StopBadBots. More and more of this pest clogs the net.

Cheers,
Michael

Hi Michael,
Thanks for the suggestion, I will work on next year, after clear the current queue.
Cheers,
Bill

It seems version 6.22 solves the performance problem although I can only say for sure next week when we have more strain on the server.

However, Query Monitor now reports a database error:

Query:
INSERT INTO wp_sbb_visitorslog (reason,ip, response, human, bot, method, url, referer, access, ua)

Error Message:
Duplicate entry ‘8388607’ for key ‘id

It could also be that this database query bug made the performance problem disappear because the plugin failed to to write into the database to log the page access.

Did you consider allowing admins to completely disable any kind of logging?

Hi Michael,

That duplicity error happens just once?
If yes, just ignore it.
Please, let me know.

Cheers,
Bill

Bill, yes it only happens once. Will you fix that in the next release?

Hi Michael,

Looks like your SQL server fail and tried to use the same record id twice.
You told you have tons of visits. Then, is a very, very small fail mysql fail.
Please, let me know if that happens again and I can study how to avoid it.

Don’t worry about that. That mean one visit among millions was not recorded.

Cheers,
Bill

Okay, this was a misunderstanding. I meant there is only one error. Query Monitor shows this error on every page. So the error reoccurs all the time.

The error message is always the same: Duplicate entry ‘8388607’ for key ‘id’

The caller is do_action(‘template_redirect’)