Crawler Doesn’t Seem to Crawl Product Pages

  • Resolved arjunmurali1993

    (@arjunmurali1993)


    4 years, 5 months ago

    Hi,

    I have a pretty large site – a woocommerce store with over 6000 products. I recently moved my site to an openlitespeed server on Google Cloud with Litespeed Cache. I feel it would be useful to have the crawler cache the site so that users don’t hit uncached pages, which is very often since we have a large site.

    I’ve tried setting up the crawler by following the instructions here:
    https://docs.litespeedtech.com/lscache/lscwp/crawler/

    When I refresh the Map, it only detects about 285 pages and completely skips all custom taxonomies despite the Exclude Custom Taxonomies being empty. The crawler also seems to classify almost all the pages that it does crawl as Cache Miss or Blacklisted despite all those pages working properly when I visit them manually. I’m attaching a screenshot of the same.

    To fix the sitemap, I added our sitemap URL /sitemap_index.xml in the Sitemap Settings. When I tried to refresh crawl map, it just went to 0 pages. I removed that sitemap url and it went back to 285 odd pages still ignoring the product pages and custom taxonomies.

    The watch crawler status is almost always completely empty and only shows dots progressing.

    Could you please help me out?

    Report Details:
    Report number: MMKBVNMM

    Report date: 06/11/2020 08:21:29

    Screenshot: https://imgur.com/pJIjqK2.jpg

    Regards,
    Arjun

    • This topic was modified 4 years, 5 months ago by arjunmurali1993. Reason: Missed Screenshot

    The page I need help with: [log in to see the link]

Viewing 7 replies - 1 through 7 (of 7 total)
  • Plugin Support qtwrk

    (@qtwrk)

    Hi,

    Could you please go to admin area , left-side bar —> LiteSpeed Cache —> Toolbox —> [6] Debug Settings

    in this page:

    set Debug Log to Admin IP Only

    add your IP into Admin IPs (you will see your IP below.)

    set Debug Level to Advanced

    save changes.

    Now ,open a new window, go to sitemap page , don’t refresh it yet.

    then click button [D] Clear Log in previous window, keep this window open, don’t close it nor navigate away from it.

    now refresh sitemap, after it completed, go back to Log View window and refresh this page, you should see the debug log is showing up.

    Then please copy all the text , paste it into https://paste.ubuntu.com/

    then post the link it generated for you.

    Best regards,

    Thread Starter arjunmurali1993

    (@arjunmurali1993)

    Hi,

    Thanks for the quick response. Here’s the link to the generated log:
    https://paste.ubuntu.com/p/hQmZP3K98b/

    Regards,
    Arjun

    Plugin Support qtwrk

    (@qtwrk)

    Hi,

    Is this with custom sitemap URL ?

    Best regards,

    Thread Starter arjunmurali1993

    (@arjunmurali1993)

    Hi,

    My site has sitemap generated by Yoast plugin.

    However, I haven’t entered anything into the sitemap section on Litespeed settings. I’ve left the sitemap field blank.

    Regards,
    Arjun

    Plugin Support qtwrk

    (@qtwrk)

    Hi,

    Please set your custom sitemap , then refresh it with debug log on , then post the log

    Best regards,

    Thread Starter arjunmurali1993

    (@arjunmurali1993)

    Hi,

    Here’s the log with the custom sitemap:
    https://paste.ubuntu.com/p/gGxdGPYF6D/

    Regards,
    Arjun

    Plugin Support qtwrk

    (@qtwrk)

    Hi,

    Please create a ticket here

    Best regards,

Viewing 7 replies - 1 through 7 (of 7 total)
  • The topic ‘Crawler Doesn’t Seem to Crawl Product Pages’ is closed to new replies.