Crawl Limit & Sitemap Size Relationship

  • Resolved Nic Windley

    (@nisiwi)


    1 year, 6 months ago

    Hello support / https://mastodon.social/@ramonfincken

    Can someone help me understand the relationship between crawl Limit and Sitemap size.

    Is the crawl Limit just the max number of pages per request irrespective of sitemap size until it eventually crawls all pages or is the limit based on the size of the crawled sitemap – i.e. if I have 20000 pages should I set the limit to 20000 or set it to 1000 and make 20 requests over X period ?

    Many thanks

Viewing 1 replies (of 1 total)
  • Plugin Author ramon fincken

    (@ramon-fincken)

    setting the limit very high is not something you want as timeouts will occur and the script will stop crawling.

    so it is better to crawl often with less then crawl (for instance) once a day with thousand(s) of items

Viewing 1 replies (of 1 total)
  • The topic ‘Crawl Limit & Sitemap Size Relationship’ is closed to new replies.