Automatically Whitelist Own Server / LiteSpeed Crawler’s / Certain Servers

  • Resolved Ryder

    (@hyflex)


    7 years ago

    Hi,

    We ran into a problem yesterday on our website where anyone who wasn’t logged in to the website would see a Wordfence blocked page on one specific page, I worked out eventually what had gone wrong and fixed it my side but this fix can be automated…

    The problem is that LiteSpeed Cralwer had been blocked for crawling too many times and then LiteSpeed had cached the page as a static page.

    This can be resolved very easily by making Wordfence automatically whitelist the servers own IP (IPV4 & IPv6, if available) AND/OR whitelisting the LiteSpeed Crawler’s user agent if LiteSpeed Caches WordPress Plugin is installed.

    I tried to clear the cache over at Cloudflare as I am making use of Cloudflare services too however that didn’t do anything but at the same time I went and found a list of all of Cloudflares IP’s so I could whitelist those IP’s too. I’d suggest having an option to automatically whitelist Cloudflare IP’s and make it so Wordfence automatically updates the IP list from https://www.cloudflare.com/ips/ every time they change.

    Thanks.

Viewing 3 replies - 1 through 3 (of 3 total)
  • Thread Starter Ryder

    (@hyflex)

    Bumping this…

    wfyann

    (@wfyann)

    Hi @hyflex,

    Sorry about the delayed response.

    I discussed this issue with my colleagues and we believe that rather than white-listing IPs, it would be interesting to understand why the blocked page is being cached and also to look at the Rate Limiting Rules.

    Could you please confirm exactly which blocked page is being cached?

    Also, are you hosting the site yourself or are you on a shared host?

    Thread Starter Ryder

    (@hyflex)

    Hi,

    The issue can be resolved by both LiteSpeed & Wordfence.

    The page is being cached because LiteSpeed have a crawler that routinely loads all of the pages and caches a static copy; if that crawler exceeds the rate limits and then loads another page it will cache the blocked page.

    For reference my rate limits were set to:

    • If anyone’s requests exceed 120 per minute then throttle it.
    • If a crawler’s page views exceed 60 per minute then throttle it.
    • If a crawler’s pages not found (404s) exceed 10 per minute then block it.
    • If a human’s page views exceed 60 per minute then throttle it.
    • If a human’s pages not found (404s) exceed 30 per minute then block it.
    • If 404s for known vulnerable URLs exceed 5 per minute then block it.

    I shouldn’t need to change my rate limits to prevent this from happening, it can be resolved fairly easily by Wordfence automatically whitelisting it’s own IP/hostname (IPv4 and IPv6) as the crawler will more than likely run on it’s the same server.

    Of course LiteSpeed can fix this too by detecting if the page is a wordfence blocked page and not caching that page, they could then either email the admin’s email or slow the crawler down automatically.

    As for whitelisting other servers, Cloudflare themselves recommend users whitelist their servers for WordPress installations that use Wordfence. It would make more sense if Wordfence either included a “Whitelist Cloudflare Servers” checkbox or at least made it easy to whitelist the servers… Why on earth is Wordfence using some funky method of whitelisting IP ranges, use the normal CIDR style as then users can just copy and paste the ranges crom Cloudflare across to Wordfence rather than having to use this odd style:

    • 103.21.[244-247].[0-255]
    • 2400:cb00:[0-ffff]:[0-ffff]:[0-ffff]:[0-ffff]:[0-ffff]:[0-ffff]

    As for the cached page it was on our /pricing/ page and it was showing: https://i.imgur.com/draGPEN.png for anyone who wasn’t logged in to an account.

    We are on a shared server for this specific site in question, I can answer most things about the server as I can speak to the host very easily. (Fantastic hosts!). He did offer to change LiteSpeeds crawler to slow how fast it crawls the pages but I opted not to change this instead I whitelisted the servers IP’s and contacted both LiteSpeed & yourselves to try and get a fix for other users.

    Thanks

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘Automatically Whitelist Own Server / LiteSpeed Crawler’s / Certain Servers’ is closed to new replies.