Hi,
Reading this post, I see a problem with the wrong robots.
For that, I found a solution, initially given for Prestashop, by Webbax, but which I use in exactly the same way on WordPress, and it works of thunder.
For this to work, you must have access to the site’s log files, in order to find the user agents of the bots. In Notepad ++ (for example), I look for the words “bot” or else “crawl” to find them. Then, I just add them in the code that I put you
And it is acting directly at the level of the index.php file in order to act above all
You must replace the entire index.php file with the content of the code I give you after (in fact, the code I put you contains all the code to put: that of Webbax + riginal code of the file “index.php “WordPress), that’s why you have to replace all the content. If you are afraid of making a mistake, first save your original index.php file.
In production, at the root of the site, a “RobotsDebug.txt” file will be created which will list all the robot connections. Compared to the method of Webbax, I added a code allowing to recover in addition, the IP address of the robot, in order, possibly, to be able to make precise blockings of robots & hackers (even if they use proxy servers …)
<?php
/**
* Front to the WordPress application. This file doesn't do anything, but loads
* wp-blog-header.php which does and tells WordPress to load the theme.
*
* @package WordPress
*/
/**
* Tells WordPress to load the WordPress theme and output it.
*
* @var bool
*/
/* Webbax - TUTO 69 - Sécurité antibots */
$logs = true;
if($logs){
$filename = "robotsDEBUG.txt";
$log_file = fopen($filename, "a+") or die("Unable to open file!");
}
$bad_bots = array('AhrefsBot','SEOkicks','SemrushBot','MJ12bot','YandexBot','istellabot','Seekport Crawler','MegaIndex','ZoominfoBot','Sogou web spider','CCBot','Go-http-client','SearchAtlas','SeznamBot','Nimbostratus-Bot','SEOkicks','AspiegelBot','serpstatbot','CATExplorador','MojeekBot','crawler4j','https://sar-pravo.ru/','https://avtolombard-voronezh.ru/','https://go.mail.ru/help/robots','VelenPublicWebCrawler','Specificfeeds','SiteLockSpider','Datanyze','Dataprovider','Pandalytics','rc-crawler','AlphaBot','google-xrawler','zgrab','alibaba');
$user_agent = $_SERVER['HTTP_USER_AGENT']."\n";
$ip = $_SERVER["REMOTE_ADDR"];
if ($deep_detect) {
if (filter_var(@$_SERVER['HTTP_X_FORWARDED_FOR'], FILTER_VALIDATE_IP))
$ip = $_SERVER['HTTP_X_FORWARDED_FOR'];
if (filter_var(@$_SERVER['HTTP_CLIENT_IP'], FILTER_VALIDATE_IP))
$ip = $_SERVER['HTTP_CLIENT_IP'];
}
if($logs){
fwrite($log_file,date('Y-m-d H:i:s').' - '.$user_agent. $ip.' - ');
fclose($log_file);
}
foreach($bad_bots as $bad_bot){
if(strpos($user_agent,$bad_bot)!==false){
if($logs){
$log_file = fopen($filename,"a+") or die("Unable to open file!");
fwrite($log_file,date('Y-m-d H:i:s').' - '."BLOCKED : ".$bad_bot."\n");
fclose($log_file);
}
die('blocked bot');
}
}
/* --- */
require(dirname(__FILE__).'/config/config.inc.php');
Dispatcher::getInstance()->dispatch();
In the code, there is the line $bad_bots = array('AhrefsBot','SEOkicks','SemrushBot','MJ12bot','YandexBot','istellabot','Seekport Crawler','MegaIndex','ZoominfoBot','Sogou web spider','CCBot','Go-http-client','SearchAtlas','SeznamBot','Nimbostratus-Bot','SEOkicks','AspiegelBot','serpstatbot','CATExplorador','MojeekBot','crawler4j','https://sar-pravo.ru/','https://avtolombard-voronezh.ru/','https://go.mail.ru/help/robots','VelenPublicWebCrawler','Specificfeeds','SiteLockSpider','Datanyze','Dataprovider','Pandalytics','rc-crawler','AlphaBot','google-xrawler','zgrab','alibaba');
This is where the robots to be blocked are listed.
When you have a robot to block, at the end of the last one – alibaba in my list – you only have to add a comma, then, the quotes of key 4, the name of the bot to block then close the quotes, which gives for example:, ‘bot’
For example, if I wanted to add Google to the list of robots to block – but do not do that if you value your SEO – the code in this line would become:
$bad_bots = array('AhrefsBot','SEOkicks','SemrushBot','MJ12bot','YandexBot','istellabot','Seekport Crawler','MegaIndex','ZoominfoBot','Sogou web spider','CCBot','Go-http-client','SearchAtlas','SeznamBot','Nimbostratus-Bot','SEOkicks','AspiegelBot','serpstatbot','CATExplorador','MojeekBot','crawler4j','https://sar-pravo.ru/','https://avtolombard-voronezh.ru/','https://go.mail.ru/help/robots','VelenPublicWebCrawler','Specificfeeds','SiteLockSpider','Datanyze','Dataprovider','Pandalytics','rc-crawler','AlphaBot','google-xrawler','zgrab','alibaba','google');
Regularly monitor the RobotsDEBUG.txt file because it will quickly become huge (it all depends on the number of visitors to your site).
Leave the file for example, over a week to see which bots are passing on your site in order to block it.
In the robotsDEBUG.txt file, you will find lines like this:
2020-04-20 01:24:49 - Mozilla/5.0 (compatible; YandexBot/3.0; +https://yandex.com/bots)
2020-04-20 01:24:49 - BLOCKED : YandexBot
At the bottom, the name of the blocked robot and above, the list of blockings of this robot.
Add all the robots you want to block here.
When you think you have them all, in the code in the index.php file, replace $logs=true; with $logs=false;. This will stop event logging and the robotsDEBUG.txt file will no longer grow.
There are regular new bots, so re-enable the robotsDEBUG.txt file from time to time to find the new bots.
As you see in the example that I use on one of my client sites, we can also block url directly (usually from Russia or China)
Si vous voulez lire l’article et voir la vidéo (en Fran?ais – Suisse), je vous mets le lien.
Ces explications seront plus claires que les miennes surement.
Mais en tout ca, ca fonctionne parfaitement bien.
If you want to read the article and see the video (in French – Switzerland), I put the link.
These explanations will be clearer than mine surely (https://www.webbax.ch/2019/02/14/prestashop-1-7-boostez-hebergement-ep-69/)
But in all of that, it works perfectly well (and don’t foreget : he explains for Prestashop, but it also works with WordPress).
Yep, something else: once you have finished modifying your index.php file, save it because each time WordPress is updated, you will have to restore it since WordPress, when updating, remits its own index.php file.
Well, I hope it will help some.
