meta-externalagent/1.1 thousands of requests

  • Resolved wizard71

    (@wizard71)


    3 weeks ago

    Hi, I have an issue wherer meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) is crawling website way to aggressively, and sending hundreds of requests per minute.

    While crawling product pages would be okay, they are crawling product filters pages, thus overloading the database with these requests. Sample of request page is: .com/?filtering=1&filter_product_cat=740,3615,718,2493 etc

    How to limit the request volume without directly blocking the FB bots?

    The page I need help with: [log in to see the link]

Viewing 3 replies - 1 through 3 (of 3 total)
Viewing 3 replies - 1 through 3 (of 3 total)
  • You must be logged in to reply to this topic.