• Resolved Argentum

    (@argentum)


    Hi have a similar problem to these closed topics.

    https://www.ads-software.com/support/topic/crawler-exit-with-stopped_reset/
    https://www.ads-software.com/support/topic/crawler-issues-blacklists-every-page-and-wont-crawl-stopped-reset/

    This morning I found all the pages in the blacklist. I removed them, tried to crawl, but crawler behaves strangely. Sometimes it just ends up with all the pages as grey dots. Sometimes it finds the pages (blue dot). But no pages are cached. The only way to cache a page is to visit it as guest. Sometimes when I click Manually run, the page just flickers, like a quick reload but nothing changes.

    I have several client sites at this particular hosting company and all run Litespeed cache and working fine. But not this one.

    I have enabled debug.log. IP’s and urls are obfuscated.

    10/13/21 10:34:02.454 [99.99.99.999:55880 1 Yz1] e?’“ ------GET HTTP/1.1 (HTTPS)  /wp-admin/admin.php
    10/13/21 10:34:02.454 [99.99.99.999:55880 1 Yz1] Query String: page=litespeed-toolbox&LSCWP_CTRL=PURGE_EMPTYCACHE&LSCWP_NONCE=b1846333a0
    10/13/21 10:34:02.454 [99.99.99.999:55880 1 Yz1] HTTP_REFERER: https://myownsite.se/wp-admin/admin.php?page=litespeed-toolbox
    10/13/21 10:34:02.454 [99.99.99.999:55880 1 Yz1] Cookie _lscache_vary: 57d879048edace9989ace55b92584efb
    10/13/21 10:34:02.454 [99.99.99.999:55880 1 Yz1] LSCACHE_VARY_VALUE: +webp
    10/13/21 10:34:02.758 [99.99.99.999:55880 1 Yz1] [Ctrl] X Cache_control -> no Cache ( logged in user )
    10/13/21 10:34:02.759 [99.99.99.999:55880 1 Yz1] [Router] LSCWP_CTRL: PURGE_EMPTYCACHE
    10/13/21 10:34:02.759 [99.99.99.999:55880 1 Yz1] [Router] LSCWP_CTRL verified: 'PURGE_EMPTYCACHE'
    10/13/21 10:34:02.760 [99.99.99.999:55880 1 Yz1] [Purge] added *
    10/13/21 10:34:02.760 [99.99.99.999:55880 1 Yz1] e??? Init
    10/13/21 10:34:02.766 [99.99.99.999:55880 1 Yz1] X-LiteSpeed-Purge: public,*
    10/13/21 10:34:02.771 [99.99.99.999:55880 1 Yz1] [Purge] added _MIN
    10/13/21 10:34:02.772 [99.99.99.999:55880 1 Yz1] X-LiteSpeed-Purge: public,*
    10/13/21 10:34:02.797 [99.99.99.999:55880 1 Yz1] [Object] flush!
    10/13/21 10:34:02.798 [99.99.99.999:55880 1 Yz1] [Purge] Flushed object cache
    10/13/21 10:34:02.798 [99.99.99.999:55880 1 Yz1] [Purge] Reset opcode cache
    10/13/21 10:34:02.799 [99.99.99.999:55880 1 Yz1] [Purge] Purge all 
    10/13/21 10:34:02.817 [99.99.99.999:55880 1 Yz1] [Ctrl] 301 from https://myownsite.se/wp-admin/admin.php
    10/13/21 10:34:02.817 [99.99.99.999:55880 1 Yz1] [Ctrl] 301 to https://myownsite.se/wp-admin/admin.php?page=litespeed-toolbox
    10/13/21 10:34:02.817 [99.99.99.999:55880 1 Yz1] [Ctrl] X Cache_control -> no Cache ( 301 to same url )
    10/13/21 10:34:02.818 [99.99.99.999:55880 1 Yz1] [Ctrl] X Cache_control -> public forced ( Setting: / )
    10/13/21 10:34:02.818 [99.99.99.999:55880 1 Yz1] [Ctrl] Forced public cacheable due to setting: /
    10/13/21 10:34:02.818 [99.99.99.999:55880 1 Yz1] [Ctrl] X Cache_control TTL -> 300
    10/13/21 10:34:02.819 [99.99.99.999:55880 1 Yz1] [Router] get_role: administrator
    10/13/21 10:34:02.819 [99.99.99.999:55880 1 Yz1] [Vary] set_cookie ---> admin_bar:1;logged-in:1;role:99
    10/13/21 10:34:02.820 [99.99.99.999:55880 1 Yz1] e?’° X-LiteSpeed-Cache-Control: public,max-age=300
    10/13/21 10:34:02.821 [99.99.99.999:55880 1 Yz1] e?’° X-LiteSpeed-Purge: public,*
    10/13/21 10:34:02.821 [99.99.99.999:55880 1 Yz1] e?’° X-LiteSpeed-Tag: e49_default,e49_URL./wp-admin/admin.php/,e49_
    10/13/21 10:34:02.821 [99.99.99.999:55880 1 Yz1] End response
    --------------------------------------------------------------------------------
    
    10/13/21 10:34:03.375 [99.99.99.999:55880 1 wHu] e?’“ ------GET HTTP/1.1 (HTTPS)  /wp-admin/admin.php
    10/13/21 10:34:03.375 [99.99.99.999:55880 1 wHu] Query String: page=litespeed-toolbox
    10/13/21 10:34:03.375 [99.99.99.999:55880 1 wHu] HTTP_REFERER: https://myownsite.se/wp-admin/admin.php?page=litespeed-toolbox
    10/13/21 10:34:03.375 [99.99.99.999:55880 1 wHu] Cookie _lscache_vary: admin_bar:1;logged-in:1;role:99
    10/13/21 10:34:03.375 [99.99.99.999:55880 1 wHu] LSCACHE_VARY_VALUE: +webp
    10/13/21 10:34:04.487 [99.99.99.999:55880 1 wHu] [Ctrl] X Cache_control -> no Cache ( logged in user )
    10/13/21 10:34:04.802 [88.888.88.88:50490 1 9yy] e?’“ ------POST HTTP/1.1 (HTTPS)  /wp-cron.php
    10/13/21 10:34:04.802 [88.888.88.88:50490 1 9yy] Query String: doing_wp_cron=1634114044.7045419216156005859375
    10/13/21 10:34:04.802 [88.888.88.88:50490 1 9yy] HTTP_REFERER: https://myownsite.se/wp-cron.php?doing_wp_cron=1634114044.7045419216156005859375
    10/13/21 10:34:05.122 [88.888.88.88:50490 1 9yy] [Router] get_role: 
    10/13/21 10:34:05.127 [88.888.88.88:50490 1 9yy] [Optm] init
    10/13/21 10:34:05.200 [88.888.88.88:50490 1 9yy] [Img_Optm] allowance_max 200 wet_limit 
    10/13/21 10:34:05.200 [88.888.88.88:50490 1 9yy] [Img_Optm] preparing images to push
    10/13/21 10:34:05.202 [88.888.88.88:50490 1 9yy] [Img_Optm] need gather due to no new raw image found
    10/13/21 10:34:05.203 [88.888.88.88:50490 1 9yy] [Img_Optm] gather_images bypass: no new image found
    10/13/21 10:34:05.225 [88.888.88.88:50490 1 9yy] [Ctrl] X Cache_control -> public forced ( Setting: / )
    10/13/21 10:34:05.225 [88.888.88.88:50490 1 9yy] [Ctrl] Forced public cacheable due to setting: /
    10/13/21 10:34:05.226 [88.888.88.88:50490 1 9yy] [Ctrl] X Cache_control TTL -> 300
    10/13/21 10:34:05.226 [88.888.88.88:50490 1 9yy] [Router] get_role: 
    10/13/21 10:34:05.226 [88.888.88.88:50490 1 9yy] [Vary] role id: failed, guest
    10/13/21 10:34:05.226 [88.888.88.88:50490 1 9yy] [Router] get_role: 
    10/13/21 10:34:05.226 [88.888.88.88:50490 1 9yy] [Router] get_role: 
    10/13/21 10:34:05.226 [88.888.88.88:50490 1 9yy] e?’° X-LiteSpeed-Cache-Control: public,max-age=300
    10/13/21 10:34:05.227 [88.888.88.88:50490 1 9yy] e?’° X-LiteSpeed-Tag: e49_default,e49_URL./wp-cron.php/,e49_
    10/13/21 10:34:05.227 [88.888.88.88:50490 1 9yy] [Optm] bypass: Not frontend HTML type
    10/13/21 10:34:05.227 [88.888.88.88:50490 1 9yy] End response
    --------------------------------------------------------------------------------
    
    10/13/21 10:34:05.333 [99.99.99.999:55880 1 wHu] [Ctrl] X Cache_control -> no Cache ( Admin page )
    10/13/21 10:34:05.466 [99.99.99.999:55880 1 wHu] [Ctrl] X Cache_control -> public forced ( Setting: / )
    10/13/21 10:34:05.466 [99.99.99.999:55880 1 wHu] [Ctrl] Forced public cacheable due to setting: /
    10/13/21 10:34:05.466 [99.99.99.999:55880 1 wHu] [Ctrl] X Cache_control TTL -> 300
    10/13/21 10:34:05.466 [99.99.99.999:55880 1 wHu] [Router] get_role: administrator
    10/13/21 10:34:05.466 [99.99.99.999:55880 1 wHu] e?’° X-LiteSpeed-Cache-Control: public,max-age=300
    10/13/21 10:34:05.467 [99.99.99.999:55880 1 wHu] e?’° X-LiteSpeed-Tag: e49_default,e49_URL./wp-admin/admin.php/,e49_
    10/13/21 10:34:05.467 [99.99.99.999:55880 1 wHu] End response
    --------------------------------------------------------------------------------
    
    10/13/21 10:34:05.706 [99.99.99.999:55880 1 wRm] e?’“ ------GET HTTP/1.1 (HTTPS)  /wp-admin/admin-ajax.php
    10/13/21 10:34:05.706 [99.99.99.999:55880 1 wRm] Query String: action=cff_ppca_token_check_flag
    10/13/21 10:34:05.706 [99.99.99.999:55880 1 wRm] HTTP_REFERER: https://myownsite.se/wp-admin/admin.php?page=litespeed-toolbox
    10/13/21 10:34:05.706 [99.99.99.999:55880 1 wRm] Cookie _lscache_vary: admin_bar:1;logged-in:1;role:99
    10/13/21 10:34:05.999 [99.99.99.999:55880 1 wRm] [Ctrl] X Cache_control -> no Cache ( logged in user )
    10/13/21 10:34:06.135 [99.99.99.999:55880 1 wRm] [Ctrl] X Cache_control -> no Cache ( Admin page )
    10/13/21 10:34:06.143 [99.99.99.999:55880 1 wRm] [Ctrl] X Cache_control -> public forced ( Setting: / )
    10/13/21 10:34:06.143 [99.99.99.999:55880 1 wRm] [Ctrl] Forced public cacheable due to setting: /
    10/13/21 10:34:06.143 [99.99.99.999:55880 1 wRm] [Ctrl] X Cache_control TTL -> 300
    10/13/21 10:34:06.143 [99.99.99.999:55880 1 wRm] [Router] get_role: administrator
    10/13/21 10:34:06.144 [99.99.99.999:55880 1 wRm] [Core] Silence Comment due to REST/AJAX
    10/13/21 10:34:06.144 [99.99.99.999:55880 1 wRm] e?’° X-LiteSpeed-Cache-Control: public,max-age=300
    10/13/21 10:34:06.144 [99.99.99.999:55880 1 wRm] e?’° X-LiteSpeed-Tag: e49_default,e49_URL./wp-admin/admin-ajax.php/,e49_
    10/13/21 10:34:06.144 [99.99.99.999:55880 1 wRm] End response
    --------------------------------------------------------------------------------
    
    
    • This topic was modified 3 years, 5 months ago by Argentum.
Viewing 3 replies - 1 through 3 (of 3 total)
  • Thread Starter Argentum

    (@argentum)

    I created a report number.

    Report number: LMZYOFHS

    Report date: 10/13/2021 15:32:41

    Hi,

    If all of your pages end up on the blacklist, this is usually an indication of a configuration problem, but I’ve noticed from reading your LS report a few things I’d like to bring up to your attention.

    1) I see you’ve entered ” / 300 ” in the “Force Public Cache URIs” box together with ” /dagens-status 300 “. You might have intended to force cache on “/dagens-status” only, but the ” / 300 ” entry forces the public cache TTL to be set for 5 minutes max. for your entire site.
    2) You’ve also entered ” / ” in the ” Scheduled Purge URLs ” box together with ” /digital-kartguide/ ” Again, you might have intended to schedule purge for ” /digital-kartguide/ ” only because the ” / ” covers your entire site. The schedule time is set at 05:30, which means a Purge All action happens at this time every day and the crawling process gets triggered.
    3) Your crawler interval is set at 12 hours.

    My impression of the above is that your site almost never stays in cache and that your crawler would never be able to serve a site like that.

    Thread Starter Argentum

    (@argentum)

    Hmmm, site’s been working for a long time prior to this. But much obliged, I will take a look.

    Yes, that was it! A heartfelt thank you!

    M

    • This reply was modified 3 years, 5 months ago by Argentum.
Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘Crawler not caching pages’ is closed to new replies.