Crawler Threads & Server Load Limit

[webarkitekterne]

Hi,

Our webserver is a very fast 4 core (AMD) machine with plenty or RAM, but when a crawler runs, the frontend becomes very unresponsive (from less than an sec serving a page to 30-50 sec).

Settings for Treads are 4 and Server Load Limit is 3.

We have thousands of pages and as you can see in this image, the runtime for all 8 crawlers with 4 treads goes up to 75513 sec (about 21 hours), meaning the frontend are quite slugish a lot of the time over a day. We have daily updates to many pages, so we need to run the crawlers daily.

How do we setup litespeed better?

See all crawler settings here.

Thanks!

[serpentdriver]

We will switch to a dedicated 12 core server soon. Still would like to dial in the perfect crawler settings before this happens.

There is nothing you can configure to make the crawler really faster. The only value that matters is the number of threads.

[webarkitekterne]

I have developed an alternative crawler

Yeah I saw the KITT site. Once your new version is out, we might be interested.

• This reply was modified 1 year, 12 months ago by webarkitekterne.

[serpentdriver]

Expected release date: 2022-12-06

[webarkitekterne]

Would KITT fix our problem?

[serpentdriver]

Kitt can’t break gravity because no matter what crawler it can only be as fast as the server allows. In any case, Kitt is faster than the built-in crawler and uses about 50% less resources, i.e. 50% less load.

[webarkitekterne]

Say we ran Kitt from a different server, would it be able to crawl 5000 pages in under 1 hour? (on our current server/a new dedicated server)

[serpentdriver]

To prevent abuse, Kitt can only be run from the origin server.

…would it be able to crawl 5000 pages in under 1 hour?

Kitt website is running on shared hosting where we crawl 500 URLs * 14 cache varies = 7,000 URLs in 22 minutes, but Kitt doesn’t use different crawlers. Kitt has only 1 crawler, but crawls all cache varies at the same time and not one after the other like the built-in crawler.

• This reply was modified 1 year, 12 months ago by serpentdriver.

[webarkitekterne]

Ok, thanks.

[serpentdriver]

Did you check the demo for WP?

[webarkitekterne]

Yes, saw the demo. If we use Kitt, can we then stop using our local crawlers?

[serpentdriver]

Yes. It makes no sense to you use more than 1 crawler. Kitt does and can the same as the built-in one, but much faster and with half of load.

[webarkitekterne]

Sweet. Tomorrow then.

[serpentdriver]

We won’t be able to do it until next Wednesday, I hope…

[webarkitekterne]

Auch, you had my hopes up. Will keep an eye on it.

[serpentdriver]

Kitt is ready since weeks, but we have to change our website to make it ready for Download.

• This reply was modified 1 year, 11 months ago by serpentdriver.