preloading, lifetime and cache policy

  • Resolved lawix

    (@lawix)


    1 year, 5 months ago

    Hello,

    Here’s how I think the plugin works and my issue. I set a cache lifetime of one day, and it gets cleared every day. I have 1000 pages and articles on my website. I perform preloading of pages with a ratio of 20 pages every 5 minutes, so in 4 hours, my site is fully preloaded. However, does the cache get entirely cleared every 24 hours, starting the preloading cycle anew?

    What I would like firstly is for the cache to regenerate page by page gradually over time without entirely clearing the old cache.

    Secondly, I have around 30 very heavy pages with many external API calls. Sometimes, it takes up to 40 seconds to load these pages, and they need to be regenerated once a day.

    As for my other static pages, I’d like them never to be cleared from the cache. Is it possible to have multiple cache policies based on content types?

    Thank you in advance.

    Best regards.

Viewing 7 replies - 1 through 7 (of 7 total)
  • Plugin Contributor Marko Vasiljevic

    (@vmarko)

    Hello @lawix

    Thank you for reaching out and I am happy to help!

    What the preload feature does is this:
    – Check the last offset
    – Check how many URLs are allowed to be processed per run
    – Fetch all URLs from sitemap (even nested sitemaps work)
    – Loop through a set of URLs to visit them

    If the end of the list is reached, it will start from the beginning in the next run.
    So after 4 hours, the preload starts from the top, and again and again.

    There is a Page Cache factor and the Cache control factor (Browser Cache)
    W3 Total Cache does not clear the cache itself. So you may have some theme or plugin that may be calling w3tc_flush_all and you can check this in the purge log.

    I hope this helps!

    Thanks!

    Thread Starter lawix

    (@lawix)

    Thank you very much for these details; I was not aware of this cache purge mechanism. I will enable it to check what’s happening. Because it seemed to me that setting pgcache_lifetime to 86400 would clear the cache, but if the preload keeps running continuously, I should never have a period where all the pages reload when the crawlers visit. This is causing my server to crash every time despite having 8 CPUs and 10GB of RAM.

    Thread Starter lawix

    (@lawix)

    ps : I hadn’t seen it; it’s a premium feature. :/

    Plugin Contributor Marko Vasiljevic

    (@vmarko)

    Hello @lawix

    Thank you for your feedback.

    With this kind of website as you described, you may want to consider not using disk-based caching and use memory-based caching like Redis or Memcached.
    Can you please share the website URL?

    Thanks!

    Thread Starter lawix

    (@lawix)

    I am indeed using Redis for all caches. My website is https://www.irit.fr. I could share the configuration file with you tomorrow if you’d like.

    Plugin Contributor Marko Vasiljevic

    (@vmarko)

    Hello @lawix

    Ah, I see, the problem may be that all the pages are preloaded which means that all 1000+ files are utilized, if you purge the cache all at once, the entire page cache is purged. We are working on a feature that will remedy this, and it will allow for the cached pages to be cached until explicitly purged by the user.

    I’ll get back to you with more info.

    Thanks!

    Thread Starter lawix

    (@lawix)

    Nice, thanks you so much !

    • This reply was modified 1 year, 5 months ago by lawix.
Viewing 7 replies - 1 through 7 (of 7 total)
  • The topic ‘preloading, lifetime and cache policy’ is closed to new replies.