Preloading more URLs

Looking at the log this morning, it seems that the preloader did load some more pages on my cron job at 3am, and then a few times this morning when someone made an order.

Each time it preloaded 50 urls, but most of them are not URLs that we’d want preloaded, and I’d tried to exclude them from being cached.

Eg:
/?taxonomy=product_type&term=simplepage/2/
/?taxonomy=product_type&term=simplepage/3/
/?taxonomy=product_type&term=simplepage/4/
It’s also tried to preload some paginated categories like

/cat/miniatures-and-painting/fantasy-historical-miniatures/page/2/
/cat/miniatures-and-painting/fantasy-historical-miniatures/page/3/ etc etc

Nobody navigates this site via taxonomy and we don’t use category pagination, we have infinite scroll.
I have ticked don’t cache: Pages with query args, Tags (is_tag) , and I set pagination to 500 in the hope of stopping it caching paginated pages.

I’ve added to my do not cache:

/?product_type=*
/author/*
/condition/*
/?taxonomy=*

But I’m not sure why the preloader is preloading these URLs at all, and I’m not sure how to tell it not to cache anything with /page/ in it, or how to tell it to cache more of my actual products, which are the things that I would like preloaded.

I’ve also noticed that where I had listed my sitemaps on the /Advanced tab, it’s rewritten my sitemap paths to /product_cat-sitemap.xml?taxonomy=*

I can’t take that ?taxonomy=* out, it just puts it back in when I do.

Yes in the plugin settings under FAQ tab under the advanced questions section, you can see that the there is a constant that you can set to overwrite the default 50 limit.

Thank you! I had missed that. I added:

define ('SWCFPC_PRELOADER_MAX_POST_NUMBER','100');

to my wp-config.php file and it’s now getting 100 from the queue rather than 50, I’ll experiment to see how many we can reasonably preload.

I think I’ve found a minor bug.
If you put something like /?attachment_id* or /?taxonomy=* as the last item in the ‘Prevent the following URIs to be cached’ box on the cache tab, it appends that string to all the sitemap filenames you list in the Advanced tab in the ‘Preload all URLs into the following sitemaps’ box.

I stopped this being a problem temporarily by re-ordering the items in the ‘Prevent the following URIs to be cached’ box so that the last item wouldnt’ have a ? in it, but you’ll probably want to fix that.

It’s still preloading strings like:
[2024-08-22 15:35:23] [preloader::task] Preloading URL https://shopontheborderlands.co.uk/?product_type=simplepage/6/
[2024-08-22 15:36:22] [preloader::task] Preloading URL https://shopontheborderlands.co.uk/cat/other-role-playing-games/page/2/

Even though I listed /?product_type* in ‘do not cache’.

How can I stop it preloading URLs that don’t exist? Where’s it even getting them from?

Ah, OK, I think I’ve got it. I had set ‘Automatically preload the pages you have purged from Cloudflare cache with this plugin’ to ‘yes’.

So it was preloading not just the product page, but everything that the changed product reported as being related, eg taxonomies, author page, etc etc, even if those had otherwise been excluded form caching. It would be nice if this respected the exclusions set elsewhere but I guess it’s not that important.

Great to hear that you have found the solution and thanks a lot for the feedback. I’m sure the team will look into it.