How to get constant memcached preload pages

  • Resolved T4ng

    (@schwipps)


    5 years, 10 months ago

    Hi,

    I’m currently testing W3 Total Cache, mostly aiming to serve my pages through memcached on a Lightsail/Cloudfront/S3 setup. It works pretty well so far on a test website, though I’m still not sure about how to set it to keep my pages in cache in the long run.

    There are a couple of settings I could use, but their name are not always that explicit, so that I’m not sure how I’m supposed to set them.

    What I’d like, is basically to launch a precache ‘warm up’ of my pages every 3000 sec (like a cron task), per batches of 20 pages, every 10 secs (not to put the server down), and set pages expiry delay to 3600 sec.

    Finally, I understand W3TC needs a sitemap to know which page it should cache, but can I declare a couple of non-indexed pages?
    Sometimes, I have pages I don’t need browsers to take care of, such as contact forms. So I no-index them, so that they will get rightfully kicked out from the sitemap. But still, I’d like them to have them cached! ??

    How can I achieve this? Or am I basically on a wrong track?

    Thanks for your help.

    • This topic was modified 5 years, 10 months ago by T4ng. Reason: Title mistyped
Viewing 6 replies - 1 through 6 (of 6 total)
  • Plugin Contributor Marko Vasiljevic

    (@vmarko)

    Hello,

    You can use Cache preload option for this in Performance>Page cache, Page preload sub-box. There you can set the update interval and the Pages per interval:
    As for the non-indexed pages, you need to make other non-public sitemap file and feed w3tc with it. OR – call those pages via crontab: curl https://my-website.com/mypage/
    I hope this helps

    Thread Starter T4ng

    (@schwipps)

    Hi,

    I found out that the wp-cron.php was failing because of an htpasswd auth. Once I set that off, wp-cron get fired, every minute.

    With the basic settings(900 / 10), it primes 10 pages… Every 900 sec, and that’s about it.

    However, I also noticed the server wasn’t crawling all the pages in the same second. Therefore, I tried these settings : 100 pages, every 5 minutes.

    It seems just much better now.

    Regarding the sitemap, can I submit several sitemaos, with à comma for example?

    Thanks for your support!

    Plugin Contributor Marko Vasiljevic

    (@vmarko)

    Hello,

    Only one Sitemap can be used for Cache Preload.

    Thread Starter T4ng

    (@schwipps)

    OK thanks…
    Finally, is your plugin really able to run a periodic pages prime?
    Here it seems to work right after a flush, but once it’s done crawling the pages, and the pages get expired so that they won’t be served by memcached anymore (after 3600 seconds, by default), no further crawl will occur.

    Is it a normal behavior? Or am I just supposed to entrust a custom cron job with this task?

    Thanks

    Thread Starter T4ng

    (@schwipps)

    Sorry Marko, we’re kind of stuck here…

    wp-cron.php is called very frequently (almost every minute), but it doesn’t trigger the preload as requested in the settings. May be only twice a day, when settings ask for every hour, and object cache is 1 hour as well.

    Any idea what could cause this?

    Thread Starter T4ng

    (@schwipps)

    Hi,

    We digged it a bit, and it seems part of the problem was that CloudFront was catching some requests (wp-cron lauching + page visits), making the preload process very inconsistent.

    It’s a bit better now, but I still notice lots of pages non cached when I submit my whole sitemap, but not when I submit only the pages sitemap (the exact number of pages get preloaded by the server in a 3600 seconds period = Maximum lifetime of cache objects).

    I also noticed that, if batches are consistently fired every 120 sec as expected, it doesn’t visit 20 pages, and the number of visited pages is not consistent.

    Do you see any reason explaining why all the pages wouldn’t get consistently preloaded, and the number of pages visited not observed?

    Thanks

Viewing 6 replies - 1 through 6 (of 6 total)
  • The topic ‘How to get constant memcached preload pages’ is closed to new replies.