Block some spiders
-
Hello;
Since two days ago I’m receiving lots of Error logs notifications like these ones:>>>>>>>>>>> 403 GET or Other Request Error Logged – 20 enero, 2014 – <<<<<<<<<<<
REMOTE_ADDR: 81.52.143.16
Host Name: crawl-b03-s1.voilabot.orange.fr
SERVER_PROTOCOL: HTTP/1.1
HTTP_CLIENT_IP:
HTTP_FORWARDED:
HTTP_X_FORWARDED_FOR:
HTTP_X_CLUSTER_CLIENT_IP:
REQUEST_METHOD: GET
HTTP_REFERER:
REQUEST_URI: /
QUERY_STRING:
HTTP_USER_AGENT: Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ([email protected])>>>>>>>>>>> 403 GET or Other Request Error Logged – 20 enero, 2014 – <<<<<<<<<<<
REMOTE_ADDR: 199.30.20.24
Host Name: msnbot-199-30-20-24.search.msn.com
SERVER_PROTOCOL: HTTP/1.1
HTTP_CLIENT_IP:
HTTP_FORWARDED:
HTTP_X_FORWARDED_FOR:
HTTP_X_CLUSTER_CLIENT_IP:
REQUEST_METHOD: GET
HTTP_REFERER:
REQUEST_URI: /robots.txt
QUERY_STRING:
HTTP_USER_AGENT: msnbot-media/1.1 (+https://search.msn.com/msnbot.htm)>>>>>>>>>>> 403 GET or Other Request Error Logged – 20 enero, 2014 – <<<<<<<<<<<
REMOTE_ADDR: 199.30.20.24
Host Name: msnbot-199-30-20-24.search.msn.com
SERVER_PROTOCOL: HTTP/1.1
HTTP_CLIENT_IP:
HTTP_FORWARDED:
HTTP_X_FORWARDED_FOR:
HTTP_X_CLUSTER_CLIENT_IP:
REQUEST_METHOD: GET
HTTP_REFERER:
REQUEST_URI: /el-coloso-en-llamas/
QUERY_STRING:
HTTP_USER_AGENT: msnbot-media/1.1 (+https://search.msn.com/msnbot.htm)What can I do to block them.
- The topic ‘Block some spiders’ is closed to new replies.