Prime pages before they expire

Hello @1fabi1

Thank you for reaching out and I am happy to help!
The information you shared suggests that the website is preloaded every 15 mins, meaning that the new cache is created regulary. Custom cron helps with trigering the wp_cron, and the expiration time of the cache should not have any effect in this case.
Can you confirm this happens in the specific time of day? There could be some temporary network issue or you are hitting the website at the specific moment when the cache is invalidated, Maximum lifetime of cache objects expires.
Is there a chance you could share your website URL?

Thanks!

Hi @vmarko

I did some tests with different cache expiration times and different refresh times. I don’t remember the exact numbers but I think I tried setting the cache expiration to something like 10 minutes and the refresh to 1 minute. I ran the cron job manually (very often) for 9 minutes, waited another 2 minutes and then refreshed the page -> the page was not loaded from cache but generated on the fly resulting in > 1 second response time.

Next, I waited for 10 minutes (until the cache was expired) and ran cron again. This time, the page cache got primed.

This leads me to believe that running cache refresh multiple times WITHIN the expiration window will NOT acutally refresh the cache since it is not expired at the time of the cron run.

This however leads to the situation I’m experiencing as described above.

The response time spikes do occur precisely every 24 hours whicht is consistent with my cache expiration of 24 hours. See attached image: Screenshot 2024-08-02 083609.png (the app is running on Azure App Service and I’m taking the measurements from App Service HTTP access log). Querying the redis cache for TTL shows that the expiration timestamp fits into this time range:

127.0.0.1:6379> ttl "w3tc_353338000__0_pgcache_173e510187ce98631d575a985640921a_ssl"
(integer) 19182

PS C:\Users\xxxxxx> (Get-Date).AddSeconds(19182)
Freitag, 2. August 2024 13:51:14

To double check, I set the cache expiration to 1 hour, purged the cache and waited for a few hours. As you can see, the response time spikes are now happening every hour insted of every 24 hours (timing may be a bit off since the website does receive that much traffic for now): Screenshot 2024-08-02 132204.png

The URL for the webpage is https://heuberg-brass-festival.de/

Edit: Is it possible to log when and why (by cron/cache prime or by requesting noncached site) a page got cached?

Thanks ??

• This reply was modified 3 months, 3 weeks ago by 1fabi1.

Hello @1fabi1

Thank you for your feedback.
Ok, now I can see what you are referring to. However, this shows the spike because the cache is cleared and not the TTFB in this case. Naturally, the cache purge does consume some of the resources on the server depending on the size of the cache.
Do you see any downtime in website performance when you visite the website in that time period?
Thanks!

Hello @vmarko,

thanks for the quick reply. I’m not sure if I understand you correctly. Obviously purging the cache does consume some resources but I don’t think thats the problem here.

As I stated, I think the culprit is that the automatic priming of the page cache does not work correctly. Can you perhaps describe how that mechanism works precisely?

My best guess is that it works like this:

1. Prime cache cron job is called
2. Download the sitemap
3. Do the following steps recursively for each link in the sitemap
   • Check if a cached version of the site is present
   • If a cached version is present, continue with the next link
   • If NO cached version is found, create that one and continue with the next link

If my assumptions are correct, this would imply that the page cache only gets primed after the current cached version expired. Obviously there is some time gap between the old version expiration and the next cron job run because the cron job is not called event-based but rather time-based (eg. every 15 minutes). That would result in a time gap somewhere in the range of close to 0 (best case) to 15 (worst case) minutes where we don’t have a cached version of the page available. Now whenever a “real” request occurs within this window, the request can not be delivered from the page cache.

To fix this issue and make the “prime cache” feature work as intended, I think we would need some kind of grace period like “re-prime the page cache if current cache item’s expiration date is less than 20 minutes in the future” where these 20 minutes should obviously be the gap between individual cron runs plus a few minutes for safety margin.

Again, this is based on my assumptions which I got from testing with various timings and checking the cached state of the page. Please correct me if I’m wrong here ??

-Fabian

Hello @1fabi1

Thank you for yoru feedback.

What the feature does is this:
– Check the last offset
– Check how many URLs are allowed to be processed per run
– Fetch all URLs from sitemap (even nested sitemaps work)
– Loop through a set of URLs to visit them
If the end of the list is reached, it will start from the beginning in the next run

So the priming starts from the start onnce the end of the list is reached, meaning the cache priming is constant, and it’s always priming the cache.
If the settings are: Update interval: 900s (15 mins) Pages per interval: 10 it means that every 15 mins 10 pages will be cached untill the end of the list based on the sitemap is reached, and then, it starts from the top, no matter if the cache is purged or not.

What you can do is to use wp-cli instead and call wp w3-total-cache pgcache_prime [–start=<start>] [–limit=<limit>] using manual cron

Thanks!

Hi @vmarko

thanks for clarifying how the mechanism should work. I have set the update interval to 900 seconds and 10 pages per interval (but as I wrote earlier, it is a single page website so there is essentially only one page).

Given this settings, the pages delivered from cache should never be older than 15 minutes, correct?

I just did look at the source code of the site (view-source:https://heuberg-brass-festival.de/) and the W3TC summary at the bottom of the page states that the page was created yesterday, arund 23 hours ago:

<!--
Performance optimized by W3 Total Cache. Learn more: https://www.boldgrid.com/w3-total-cache/

Object Caching 99/260 objects using Redis
Page Caching using Redis 
Content Delivery Network via cdn.heuberg-brass-festival.de
Database Caching 7/26 queries in 0.037 seconds using Redis

Served from: _ @ 2024-08-05 13:23:49 by W3 Total Cache
-->

I did run the command you quoted above, this was the output:

/home/site/wwwroot# wp w3-total-cache pgcache_prime --allow-root
Priming from sitemap https://heuberg-brass-festival.de/sitemap.xml entries 1..10
Priming https://heuberg-brass-festival.de/sitemap.xml
Priming https://heuberg-brass-festival.de/wp-sitemap-posts-page-1.xml
Priming https://heuberg-brass-festival.de/
Success: PageCache Priming triggered successfully.

But even after this, the cached date in the page source did not change.

So I manually purged the page cache, entered the same command again (with the same output) and now the page source shows the current date and time:

Served from: _ @ 2024-08-06 12:11:26 by W3 Total Cache

Maybe W3TC does regenerate the cache every 15 minutes like it’s supposed to be but fetches the wrong record from the cache when returning it? Is there any way to verify this?

• This reply was modified 3 months, 2 weeks ago by 1fabi1.

Hello @1fabi1

Thank you for your feedback.
Well, exactly, this is when the Cache expiration hits, after 24 hrs. The Cache is created with preload, however, yoru settings are also stating that the cache should be retained 24 hrs. Meaning that even when the cache is preloaded, the cache is still not cleared becuase of the TTL settings.

Please let me know if I need to clarify this more.
I understand that you have certain expectations, and I am just trying to say how this works. If the TTL is set to 24 hrs, new cache will be preloaded but not served becuase of those settings.

Thanks!

Hi @vmarko,

although we talked about this in depth, the problem is still not fixed.

Again, assuming the cache preload works as intended and creates a new cached page every 15 minutes. Since my TTL ist set to 24 hours, there should be plenty of overlap since we ALWAYS have a cached page that is less then 15 minutes old.

Why am I getting served a page that is 23 hours old as shown in the example above??

And why do my users still have to wait for the uncached page once every 24 hours??

Regards,
Fabian

Hello @1fabi1

That is correct. So what you are seeing in the statistics you shared is the moment when the cache is purged. This does not indicate the slow response time, only the consumption of server resources.
You are getting the served page that is 23 hours old because the cache is not purged. So the “Old cache ” is presented to the users, because if you do not purge the cache, there is no reason to serve the new one. So if you want to do this, set the cache preload to 1 hour, and set the TTL of the cache to less than an hour. This way, the cache will be purged and the preload will hit every hour to serve the new cache.

Thanks!

This does not indicate the slow response time, only the consumption of server resources.

The graph I shared shows the response time when hitting the web page (response times of GET requests to “/”). Resource utilization is NOT shown in the graph.

You are getting the served page that is 23 hours old because the cache is not purged. So the “Old cache ” is presented to the users, because if you do not purge the cache, there is no reason to serve the new one.?

I believe, every cache would by default serve the “newest” cached entry, and not the oldest one, just because it was not purged. When caching data, a newer version of the cache is always better than the older one.

So if you want to do this, set the cache preload to 1 hour, and set the TTL of the cache to less than an hour. This way, the cache will be purged and the preload will hit every hour to serve the new cache.

This would not fix my initial problem.

Again: My problem is not that the cache preload feature does not work at all. When the corresponding cron job runs, the cached item will be create – we’ve already checked that.

The problem is, that there is a natural gap between the time the cached page expires and the next preload cache cron job run since it is called every 15 and not immediately after expiration of the cached page. If a normal request to the page occurs in between this gap, there is no cached page available so it has to be created on the fly slowing down the response time for the user. Additionally, this cached page is stored, which means that the preload cache job running a few minutes later will have nothing to do, as the cached page is available now.

Hello @1fabi1

Thank you for your feedback.
The preload works when initiaded, and it depends on the wp-cron.
You can bypass this by setting a custom server cron and use wp-cli call wp w3-total-cache pgcache_prime [–start=<start>] [–limit=<limit>] as I mentioned before. This will not change the time output as it depends on the cache TTL, however, you can set the wp-cli to preload the cache once the cache is purged, therefore (every 24 hrs) therefore preloading the cache at that point

Thanks!