What is the Crawler?

Hi, @theselecter

The purpose of the crawler is to traverse your site and refresh pages that have expired in the cache. It’s a way of pre-caching pages so that you don’t have to wait for visitors to hit the backend on those pages.

You can get some details on what all of the settings mean in our wiki.

I hope this helps!
Lisa @ LiteSpeed

Added:
Oops! Sorry, I missed the other few questions you had.

Really, the only downside would be if you have limited system resources, and can’t handle a process running in the background, caching the pages of your site. However, there are several settings that let you control just how many resources the crawler can use, so you should be able to keep it at a minimum, if that’s what you need.

• This reply was modified 7 years, 9 months ago by LiteSpeed Lisa.
• This reply was modified 7 years, 9 months ago by LiteSpeed Lisa.

Thanks for the quick response Lisa!

On the Crawler File page I need to enabled the Crawler Cron and generate the Crawler File for this to work properly. Is that correct?

Thanks,
Dave

No problem. Happy to help ??

Yes, you generate a crawler file, activate the crawler, and then if you want to start it immediately, you can press the Manually run button.

You can also watch it do its thing through the Watch Crawler Status window, which can be kind of fun.

Let us know if you have any other questions!
Lisa

Excellent! Thank you! ??

My pleasure!