Rate limiting with caching

  • Hi there, thanks for a great plugin! I understand that your rate limiting functions aren’t compatible with certain page caching plugins. Do you have any ideas for workarounds to this? Some of my client sites are getting bogged down with too much crawler traffic. The requests themselves are harmless, but the volume is causing performance issues. I’d love it if I could simply slow these requests down.

Viewing 2 replies - 1 through 2 (of 2 total)
  • Plugin Support wfpeter

    (@wfpeter)

    Hi @solventweb,

    If a hit is to a cached page, the hit wouldn’t register in Live Traffic or be counted by rate limiting. That said, the 404 rate limiting should still be effective since those pages shouldn’t be cached.

    You could try adding a crawl delay to your?robots.txt?file for bots that you often see hitting the site, like this example for Facebook that tells it to crawl 1 page every 5 seconds:

    User-Agent: FacebookBot                              
    Crawl-delay: 5

    Real crawlers?should?respect the crawl-delay to automatically throttle themselves when visiting these sites, which may reduce the amount of data/connections they’re generally using up.

    Thanks,
    Peter.

    Thread Starter solventweb

    (@solventweb)

    Thanks @wfpeter. That’s good advice. FYI the pages in question aren’t 404s.

    My main hesitation with the robots.txt change is that not all crawlers will respect it (especially those operating in grey areas). I’d prefer to enforce a rate limit on our end. Do you have any advice on how this might be accomplished with your plugin? For example, could we customize your WAF’s auto_prepend_file to include your rate limiting functions?

Viewing 2 replies - 1 through 2 (of 2 total)
  • You must be logged in to reply to this topic.