Do Allowlisted 404 URLs apply to humans?

  • Resolved p15h

    (@prestonwordsworth)


    1 year, 6 months ago

    Dear WF team

    Your annotation next to the Allowlisted 404 URLs box and your knowledge-base article on rate limiting both explain that ‘crawlers’ visiting these URLs won’t get blocked. I wondered if this wording is meant to imply that humans generating 404 at these allowlisted URLs will still get blocked according to the rate-limiting rules for humans?

Viewing 2 replies - 1 through 2 (of 2 total)
  • Plugin Support wfpeter

    (@wfpeter)

    Hi @prestonwordsworth, thanks for your question.

    The “Allowlisted 404 URLs” are designed to exempt pages or files that might be caught by the “If a crawler’s pages not found (404s) exceed” setting only and prevent them counting towards blocking a crawler unintentionally. The exemptions don’t affect blocking settings for humans, especially as these are usually set to more forgiving thresholds.

    Thanks,
    Peter.

    Thread Starter p15h

    (@prestonwordsworth)

    Thanks for confirming this, Peter. I suggest adjusting the wording next to the box to make it absolutely unambiguous!

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Do Allowlisted 404 URLs apply to humans?’ is closed to new replies.