user agent blocking is not working

  • I have user agent custom rule like this:
    *semrush.com/bot.html)

    But I am still getting their requests with code 200

    IP: 85.208.96.193 Hostname: 193.bl.bot.semrush.com
Mozilla/5.0 (compatible; SemrushBot/7~bl; +https://www.semrush.com/bot.html)

    404 rate limiting also is not working. Setting is to limit bots 4 request over 1 minute

Viewing 1 replies (of 1 total)
  • Plugin Support wfpeter

    (@wfpeter)

    Hi @forusak, thanks for getting in touch.

    I do feel that your custom rule should in theory work from what I can see, but it may be appropriate to instead block this bot using robots.txt:

    User-agent: SemrushBot
Disallow: /

    The legitimate SEMrush bot appears to observe this, so it might be better for your site than serving a block every time it visits. Naturally this doesn’t stop spoofed User-Agents, in which case I would try reducing your custom block to something like *SemrushBot/* to see if it helps.

    If it doesn’t, or the amount allowed through still seems excessive, it would help to see some filtered Live Traffic results along with your saved Rate Limiting settings so we can see precise timetamps to determine what might be happening.

    Thanks,
    Peter.

Viewing 1 replies (of 1 total)
  • The topic ‘user agent blocking is not working’ is closed to new replies.