Purge policy request pooling speedup

Purging with varnish is very slow – 2 seconds per request. Purge requests are sent immediately on update/publish which can cause insanely slow post updates/publishes when W3TC is configured with one or more reverse proxies (varnish) with a purge policy in the Page Cache settings.

Let’s say you have 2 varnish servers, and your page cache purge policy is set to 4. Here’s what the purge log looks like (I’ve cut parts out and minified it for readability):

[03:26:15] [/] Connecting to 192.168.0.1 …
[03:26:15] [/] PURGE / HTTP/1.1
[03:26:15] [/] PURGE OK
[03:26:15] [/] Connecting to 192.168.0.2 …
[03:26:15] [/] PURGE / HTTP/1.1
[03:26:15] [/] PURGE OK
[03:26:15] [/page/2/] Connecting to 192.168.0.1 …
[03:26:15] [/page/2/] PURGE /page/2/ HTTP/1.1
[03:26:18] [/page/2/] PURGE OK
[03:26:18] [/page/2/] Connecting to 192.168.0.2 …
[03:26:18] [/page/2/] PURGE /page/2/ HTTP/1.1
[03:26:21] [/page/2/] PURGE OK
[03:26:21] [/page/3/] Connecting to 192.168.0.1 …
[03:26:21] [/page/3/] PURGE /page/3/ HTTP/1.1
[03:26:23] [/page/3/] PURGE OK
[03:26:23] [/page/3/] Connecting to 192.168.0.2 …
[03:26:23] [/page/3/] PURGE /page/3/ HTTP/1.1
[03:26:25] [/page/3/] PURGE OK
[03:26:25] [/page/4/] Connecting to 192.168.0.1 …
[03:26:25] [/page/4/] PURGE /page/4/ HTTP/1.1
[03:26:28] [/page/4/] PURGE OK
[03:26:28] [/page/4/] Connecting to 192.168.0.2 …
[03:26:28] [/page/4/] PURGE /page/4/ HTTP/1.1
[03:26:30] [/page/4/] PURGE OK
[03:26:58] [/2018/my-post] Connecting to 192.168.0.1 …
[03:26:58] [/2018/my-post] PURGE /2018/my-post HTTP/1.1
[03:26:58] [/2018/my-post] PURGE OK
[03:26:58] [/2018/my-post] Connecting to 192.168.0.2 …
[03:26:58] [/2018/my-post] PURGE /2018/my-post HTTP/1.1
[03:26:58] [/2018/my-post] PURGE OK
[03:26:58] [/([a-z0-9_\-]*?)sitemap([a-z0-9_\-]*)?\.xml] Connecting to 192.168.0.1 …
[03:26:58] [/([a-z0-9_\-]*?)sitemap([a-z0-9_\-]*)?\.xml] PURGE /([a-z0-9_\-]*?)sitemap([a-z0-9_\-]*)?\.xml HTTP/1.1
[03:26:59] [/([a-z0-9_\-]*?)sitemap([a-z0-9_\-]*)?\.xml] Bad response: HTTP/1.1 404 Not Found
[03:26:59] [/([a-z0-9_\-]*?)sitemap([a-z0-9_\-]*)?\.xml] Connecting to 192.168.0.2 …
[03:26:59] [/([a-z0-9_\-]*?)sitemap([a-z0-9_\-]*)?\.xml] PURGE /([a-z0-9_\-]*?)sitemap([a-z0-9_\-]*)?\.xml HTTP/1.1
[03:26:59] [/([a-z0-9_\-]*?)sitemap([a-z0-9_\-]*)?\.xml] Bad response: HTTP/1.1 404 Not Found

Here is an example stack trace of our /wp-admin/post.php with our original W3TC configuration before we knew exactly what was causing the slowdown:

View post on imgur.com

That’s 2*2secs for home page + 2*2secs for sitemaps + 2*2 secs for post page + 8*2 secs for archive pages = 28 seconds of delay on post publish on top of the usual wordpress operation time. Depending on the configuration of your load balancer, it’s very easy to trigger a load balancer timeout and be presented with an error page every time you publish or update a post.

I have two proposed solutions to this issue:

1. Pool the PURGE requests. Even if we sent our list of requests to each configured server synchronously in parallel, that’d be a 50% speedup without tripping any rate limits. An example of a pooled request implementation can be seen in Guzzle Concurrent Requests.
2. Create a job queue to run the PURGE requests when publishing posts instead of doing it as part of the publish action. This can be done in many ways such as a wp-cron or even sending a CURL request to an AJAX URL that fires the requests off immediately as per TechCrunch’s WP Asynchronous Tasks plugin.

Hell, these two methods above could even be combined for a huge speedup.