Rule-Breaking Visit Not Flagged for AhrefsBot

  • Resolved Dennis Bareis

    (@dbareis)


    3 months, 1 week ago

    This is logged:

    51.222.253.4 - - [17/Aug/2024:14:16:12 +1000] "GET /search-plants?Soil[]=Fertile&ShowMatchingPlants HTTP/2" 200 316813 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +https://ahrefs.com/robot/)"

    While “robots.txt” contains:

    User-agent: AhrefsBot
    Disallow: /

    To me the log above shows that this is a “Rule-Breaking Visit” but is not shown as such.

Viewing 4 replies - 1 through 4 (of 4 total)
  • Plugin Author gavindarkvisitors

    (@gavindarkvisitors)

    Did you add AhrefsBot to your robots.txt within the past day?

    Dark Visitors takes some time to re-read your robots.txt, which it currently does every 24 hours. It does this in order to not spam your website by reading it constantly. Rule-breaking visits will start to show up as expected after it re-reads the updated version.

    Hope that helps!

    Gavin

    Thread Starter Dennis Bareis

    (@dbareis)

    OK, I’ll wait longer, the other issue is whether or not it is breaking the rules or continuing a scan, I’ll let it sit for a couple of days….

    I didn’t see a 24 hour delay after adding some of my other entries but who knows? If you don’t scan it everytime I refresh the “/robots-txt” page, then it should probably say when it was scanned and perhaps have a rescan button….

    Plugin Author gavindarkvisitors

    (@gavindarkvisitors)

    Hi @dbareis, checking back in. Let me know if things haven’t fixed themselves as expected. In the meantime, I’ll mark this question as resolved.

    Thread Starter Dennis Bareis

    (@dbareis)

    I can’t tell any more and without knowing the timestamp of the robots.txt file you use there is no resolving this one way or the other. At least “AhrefsBot” is not ignoring the “robots.txt” file so it must have been continuing an existing scan at the time.

Viewing 4 replies - 1 through 4 (of 4 total)
  • You must be logged in to reply to this topic.