crawler, Page optimization not working

Current server load:?0.99

your server load limit is 1 , if you have load like this, it’s very easily to hit the limit , please raise it to a higher value, and then set crawler interval to , for example , 61 seconds, then manual start it , see how it goes.

i didn’t limit it , it was left on default value , now i raised the limit to 100 , crwaling still not working, freezing to the date above, but i noticed that my rankmath sitemap is not accepted for some reason. it’s on:

myDomainName/sitemap_index.xml

i keep getting this when i try to crawl it

No valid sitemap parsed for crawler.

is checked the validation of sitemap by some tools and it’s fine

i hope there isn’t a bug or something

• This reply was modified 10 months ago by MohamedMaher. Reason: more info

<?php
require( './wp-load.php' );

$response = wp_remote_get( 'https://www.ruoaa.com/sitemap_index.xml' );

echo '<pre>';
var_dump($response);
echo '</pre>';

please create a php file with above code , put it at same dir as your wp-config.php , open it in browser , see what it returns.

I already tested that, for some reason it only return 404 error with some data

array(6) { [“headers”]=> object(WpOrg\Requests\Utility\CaseInsensitiveDictionary)#4465 (1) { [“data”:protected]=> array(7) { [“content-type”]=> string(9) “text/html” [“cache-control”]=> string(28) “private, no-cache, max-age=0” [“pragma”]=> string(8) “no-cache” [“date”]=> string(29) “Thu, 02 May 2024 18:10:18 GMT” [“server”]=> string(9) “LiteSpeed” [“content-encoding”]=> string(4) “gzip” [“vary”]=> string(15) “Accept-Encoding” } } [“body”]=> string(1249) “

by the way i use cloudflare , could that be the problem, i use the ip suggested by litespeed which is the real server ip not the cloudflare one .

hm , that’s the issue

https://www.ruoaa.com/wp-admin/admin-ajax.php it should not return 404 , it works on browser but 404 on PHP , and the body ?[“body”]=> string(1249) “ should not be that big

It seems that the issue was caused by a security plugin i disabled it and crawler now works fine grabbing sitemap, yet /wp-admin/admin-ajax.php is still showing 0 and 400 error in console, not sure if this affect litespeed ?

0 and 400 is okay , that is expected result

Thank you, i really appreciate your help

• This reply was modified 10 months ago by MohamedMaher.