New media images not uploaded unless “Only purge CDN manually” is disabled

Hello @nickwp161

Thank you for your inquiry and I am happy to assist you with this.
The answer to your question is in the name of the option mentioned “Only purge CDN manually”. What you need to do when that option is enabled is to use the “purge all” button. It’s a manual operation so it’s performed under “Only purge CDN manually” mode. We built that because of the problems with invalidation requests on CloudFront causing huge bills. If it’s enabled, you can click the purge CDN button in W3 Total Cache to send invalidation requests, otherwise, all changes will be detected automatically and an invalidation request will be sent immediately.
Let’s say you have myimage.jpg. at point (1) of time request to /myimage.jpg cause 404 response. when you upload a new image – content changes, now it responds with 200 response and image contents. so content is effectively changed. that’s why there’s a purge request sent.
So attempts to optimize purge requests to save a couple of bucks will definitely have side effects. Again – for websites with a lot of purge requests it’s much safer to change the purging policy. switch it off at plugin and perform purge from time to time.
I hope this answers your question.
Thanks!

Hi Marko,

Thank you for your prompt reply, but unfortunately I’m not sure that I’m really much clearer on this strategy.

I am talking about a single, new image being uploaded via the media control panel. It will not be uploaded to the CDN unless the “purge manually” is unchecked. But it’s new image so would not require a cache invalidation.

This means if the CDN is enabled, all the website images are rewritten for serving by the CDN, but will be missing because they haven’t been uploaded. It would then be a manual task to purge each relevant image.

Whilst this may prevent huge invalidation bills, and does neatly re-enforce responsibility for the invalidation bills to the user, I’m sure it is very confusing for those who wouldn’t even consider that this setting was relevant for new images (i.e. me!)

As the CDN should perform its own invalidations based on cache control configuration, wouldn’t it be better to always upload new/changed files but only send invalidations/purge requests based on the “purge manually” setting? That way at least the website will continue to show the full content (especially with regards new images), and will either A) show updated images when the CDN eventually invalidates or B) show updated images when the user sends manual purge requests.

There are frequent recommendations for enabling the “purge manually” setting, but this is at the expense of a partially visible website which seems wrong and counter-intuitive.

Perhaps an additional note against “Purge CDN only if explicit purge button is clicked.” to also read “Enabling this setting will also prevent uploads of new images/files. These will be uploaded only by submitting purge requests.”

At least I understand the mechanism now, but it’s taken me much of two days debugging to work out how and why it would behave like this.

Yes I want to avoid huge invalidation bills but No I don’t want to have to manually purge(upload) images in order for them to actually appear on the website. Can three be a mechanism to meet both of these requirements?

Hello @nickwp161

Sorry for the late reply. This is unfortunately how this works now since the users were complaining about the huge invalidation bills. This is the side-effect of the setting. So you can use the settings and purge the cache manually, you can disable the settings and hope for the best regarding the invalidation bills, or you can use some other CDN provider like for example Stackpath or Cloudflare (Its free but better results are with better plans)
Thanks!

Hi @vmarko,

I’m experiencing this same scenario; exactly as @nickwp161 has described I find it quite confusing. Nonetheless, there are things that are not clear to me.

1. Is the purge function a dual function? It uploads the files to S3 & then also purges the files from cloudfront?
2. If I run cdn_purge from commandline; I do not see corresponding cloudfront invalidations nor is this process running for all my network sites.
3. I have tried cdn_purging 1 site at a time from the command line & the result is the same as aforementioned.
4. Does cdn_purge only work by clicking the purge button backend?
5. Is there a function I can call to only upload files to s3 without dependency on cdn_purge?

What I am trying to do is create a cronjob that invokes the cdn purge manually after a couple of mins; doing it this way will run a “pseudo bulk” invalidation of images + possibly upload the missing assets to S3.

Achieving this would be great because then I can manage exactly how many invalidations happen and as such calculate the projected cost.

My setup is a multisite currently with 12 distinct domains & growing. All my sites are heavy visual blog publishers / authors. Having a broken interface due to non-availability of images on the cdn is not an option.

Cheers,

Munya