Guest Mode Crawls too long

it is showing 00:42:15 up 423 days, 8:13, 1 user, load average: 2.56, 2.94, 2.49

I am currently running on a 4 core cpu with 8 Gb memory on a vps server. I read from somewhere that server load limit should be 70% of the number of cpus. So I set server load limit to 3. If the load average does not go above 3, it should be fine?

I also set number of threads to 16. Under server load limit settings, it says the following: “The number of crawler threads in use will be actively reduced until average server load falls under this limit. If this cannot be achieved with a single thread, the current crawler run will be terminated.” So its’ safe to say even I set it to 16, the number of threads will drop when the system detects high average load?

A percentage value does not provide any security for this. With the current values, however, your server is running at the limit and you should definitely reduce the value for load limit.

But there is another problem. The built-in crawler is not really known for being economical with server resources. In plain language, it is a resource hog. But that depends on the optimization settings. The more you use it, the higher the load.

Reason I am asking is because the crawler is only crawling 15 URLs per 13 seconds according to the “watch crawler status” report.

I have guest mode turned on but guest optimization is turned off. Replace with webp images is also turned off. But for some reason there are four cron names under the summary table (2 Guests, 2 Webp). This adds up to a total of 18000 URLs to crawl. It will take more than 10 hours to complete.

I have another question regarding on cache hit. Under Chrome inspect>network, it will show a cache hit on the first visit of a cached page but the load time is around 2 – 3 seconds. Subsequent clicks from that page will lead to around 600 ms load time which is much faster. If I close the browser and enter the same page, load time will be back to 2 – 3 seconds but subsequents clicks to other pages will be as low as 600 ms. Why does the first load take so long even if a page is cached?

Reason I am asking is because the crawler is only crawling 15 URLs per 13 seconds according to the “watch crawler status” report.

This is not bad if the URLs are not cached.

This adds up to a total of 18000 URLs to crawl. It will take more than 10 hours to complete.

10 hours is too much.

I have another question regarding on cache hit. Under Chrome inspect>network, it will show a cache hit on the first visit of a cached page but the load time is around 2 – 3 seconds. Subsequent clicks from that page will lead to around 600 ms load time which is much faster. If I close the browser and enter the same page, load time will be back to 2 – 3 seconds but subsequents clicks to other pages will be as low as 600 ms. Why does the first load take so long even if a page is cached?

I should be able to watch it live on your site. Do you have a link to your page?

Can I send my link to you privately?

Why does the first load take so long even if a page is cached?

This depends on status header. The LiteSpeed web server has a special feature for that, but you must be a technical to understand what this feature is about.

Can I send my link to you privately?

This forum doesn’t support private messaging.

Here:

https://expiry.link/0ncja

What is your problem? It fits! If your site is slower the first time it is accessed, it is due to DNS resolution.

Sorry, what do you mean by it fits?

I was just saying the crawling time takes way too long to complete crawling the entire site.

What I mean is that I don’t see anything that could cause your site to be slow.

I was just saying the crawling time takes way too long to complete crawling the entire site.

Do you purge your cache often?

Normally I only purge my cache once a week when new products are added or plugins are deactivated/updated.

Re: DNS resolution

I added a bunch of external DNS URL under html prefetch. For localization, do I add URLs with query string at the end?

Example:

https://www.googletagmanager.com/gtm.js?id=GTM-FAKE1234

No worries, the GTM ID is fake.

I suppose these settings may improve dns lookup for external resources.

I added a bunch of external DNS URL under html prefetch. For localization, do I add URLs with query string at the end?

That’s not what I meant by DNS resolution. Before your preload settings can take effect, the server must first be contacted and DNS resolution is necessary for this. If the request takes longer after starting your browser, it is because your browser does not (yet) know the IP of your server.

You also use the Google Tag Manager and the Facebook Pixel. These external requests will block your page from rendering.

Thank you for all the details.

Do I add the query string after gtm.js for localization

1) https://www.googletagmanager.com/gtm.js

2) https://www.googletagmanager.com/gtm.js?id=GTM-FAKE1234

If these files are added onto localization tab, do I still need to add them under DNS prefetch?