How the crawler works ?

  • Resolved joy0114

    (@joy0114)


    3 years, 7 months ago

    Hi,

    My report is: OWUUMYXV
    08/04/2021 11:09:39

    I definitely didn’t understand how the crawler have to work…

    I’ve got 4 crawlers:
    0 – H?te – WebP
    1 – H?te – WebP (guest mode)
    2 – H?te
    3 – H?te (guest mode)

    For “Waiting for exploration – Already Cached – Successfully Crawled – Blocked”

    Crawler 0 shows me :
    96 1 181 –

    Crawler 1 shows me :
    – 168 2 108

    Crawler 2 shows me :
    – 276 2 –

    Crawler 3 shows me :
    – 276 2 –

    Sitemap contains 278 elements.

    Does the goal of crawler must be all sitemap elements in “Already Cached”, or “Successfully Crawled” ?

    I tried one thread, and after this two threads, but crawlers stopped after crawling several pages.

    Does the crawler act for creating a preload cache, in visiting pages for generating html cache, and ccss cache (when cron execute the requests) ?

    Thanks a lot if you can enlight me with some explanations about crawling. I carefully read doc but I did not find answer to these basic questions.

    Best regards

    • This topic was modified 3 years, 7 months ago by joy0114. Reason: syntax error
Viewing 4 replies - 1 through 4 (of 4 total)

  • Hi,

    1. Yes, the goal of crawler is to precache your site, “Already Cached” means that there is already a cache copy exist for that site before the crawler visit it. “Successfully Crawled” means that the site is being precached by the crawler, so when you visit the site for the first time, the site will be cached so it will load faster even on your first visit. The generated page should have ccss in it if it’s available.

    2. Regarding the crawler stopped when crawling several pages, can you provide a screenshot on your Crawler status?

    Regards,
    Lehan

    Thread Starter joy0114

    (@joy0114)

    Hi Lehan,

    Thanks a lot for your explanations. It’s much more clear for me !

    Here is a screenshot of crawler status. I noticed that situation is different compared to my first post.
    But now, it seems stopped, without finishing for the 1st crawler.

    screenshot crawler status

    Thanks again for your attention.
    Best regards

    Hi,

    Can you press manually run twice or three times to see what will happen?

    Regards,
    Lehan

    Thread Starter joy0114

    (@joy0114)

    Hi Lehan,

    Ok, I followed your advice and now, it’s ok: crawler has reached the end for all kind of crawl:
    277 Successfully Crawled and 1 Already Cached

    It was necessary to launch it several times. Perhaps this behavior is related to the load of server…

    Thank again.
    Best regards

    PS: it’s strange: your answers on this forum do not appears before several hours as the board shows you have posted an answer…

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘How the crawler works ?’ is closed to new replies.