Web Harvester detected

  • Resolved kv1953

    (@kv1953)


    2 years, 9 months ago

    WE are recieving warning emails from are reciving Alfirin who created the Pedigree Database https://irishsetterpedigrees.co.uk/search.html this is accessed through WordPress Website https://irishsetterpedigrees.co.uk/ . I have carried out both recomendations below but as an example received 18 email warnings delivered between 23.31 and 23.40 last night. Can i do anything in CleanTalk?

    -----Original Message-----
From: PEDIGREE WEBSITE SCRIPT <[email protected]>
To: [email protected]
Sent: Fri, 4 Feb 2022 19:53
Subject: Web Harvester Detected
Web harvester "a possible robot" detected.
 
HTTP_USER_AGENT: facebookexternalhit/1.1 (+https://www.facebook.com/externalhit_uatext.php)
REMOTE_ADDR: 31.13.115.111 [Reverse DNS: "fwdproxy-lla-111.fbsv.net"]
HTTP_HOST: irishsetterpedigrees.co.uk
REQUEST_URI: /cgi-bin/geneal.pl?op=tree&index=8991&gens=5&db=exported.dbw
SERVER_SOFTWARE: LiteSpeed

This message is being sent to you because the Pedigree Website Script detected that an automated program has attempted to harvest data from your database.
Such programs are known by various names, including: Web Harvesters, Crawlers, Creepers, Spiders, or Robots.

To suppress the generation of these emails (at least from the harvesters that follow the rules), you need to:
?	Create a file (e.g., with Windows Notepad) and call it "robots.txt".
?	Put the following text in the file:  
 
User-agent: *
Disallow: /cgi-bin/
 
?	Then upload the file to your web site's Document Root using FTP in ASCII mode.
 
For the harvesters that do not follow the rules, you need to add a new snippet of harvester blocker code in file "security.pl" (look near the bottom of the file) and use an identifiable part of the HTTP_USER_AGENT captured above as the matching pattern.

    The page I need help with: [log in to see the link]

Viewing 1 replies (of 1 total)
  • Plugin Support SergeM

    (@serge00)

    Hello, @kv1953.

    Please, do the following:
    1. Go to WordPress Admin Page —> Settings —> Anti-Spam by CleanTalk —> Advanced settings —>
    2. —> check that each form type should be set to “On” —>
    3. —> enable the options Anti-Flood and Anti-Crawler —> Save Changes.

    After doing that, synchronize the plugin with your Control Panel here:
    ? WordPress Admin Page —> Settings —> Anti-Spam by CleanTalk —> press the button “Synchronize with Cloud”

    Also, please create a private ticket and copy-paste your question there:
    https://cleantalk.org/my/support/open

    WordPess.org Support Forum is for plugins only, not the service.

Viewing 1 replies (of 1 total)
  • The topic ‘Web Harvester detected’ is closed to new replies.