Waiting for queue file to be writable slows down admin actions significantly

  • Hi,

    On a website with high amount of concurrent edits (e.g. add/edit posts, upload media, etc), I noticed that WP Cloudflare Super Page Cache spends a lot of time sleeping while waiting for the queue file to be writable. Here’s a snippet of the plugin’s debug log file where this is happening:

    [2021-09-10 02:48:36] [cache_controller::purge_cache_on_post_edit] Purge Cloudflare cache for only post id 171781 and related contents - Fired action: edit_post
[2021-09-10 02:48:36] [cache_controller::purge_cache_queue_job] I'm the purge cache cronjob
[2021-09-10 02:48:36] [cache_controller::purge_cache_queue_job] Queue file not writable. Sleep 1 second
[2021-09-10 02:48:37] [cache_controller::purge_cache_queue_job] Queue file not writable. Sleep 1 second
[2021-09-10 02:48:38] [cache_controller::purge_cache_queue_job] Queue file not writable. Sleep 1 second
[2021-09-10 02:48:39] [cache_controller::purge_cache_queue_job] Queue file not writable. Sleep 1 second
[2021-09-10 02:48:40] [cache_controller::purge_cache_queue_job] Queue file not writable. Sleep 1 second
[2021-09-10 02:48:41] [cache_controller::purge_cache_queue_job] Queue file not writable. Sleep 1 second
[2021-09-10 02:48:42] [cache_controller::purge_cache_queue_job] Queue file not writable. Sleep 1 second
[2021-09-10 02:48:43] [cache_controller::purge_cache_queue_job] Queue file not writable. Sleep 1 second
[2021-09-10 02:48:44] [cache_controller::purge_cache_queue_job] Queue file not writable. Sleep 1 second
[2021-09-10 02:48:45] [cache_controller::purge_cache_queue_job] Queue file not writable. Sleep 1 second
[2021-09-10 02:48:46] [cache_controller::purge_cache_queue_job] Queue file not writable. Sleep 1 second
[2021-09-10 02:48:47] [cache_controller::purge_cache_queue_job] Queue file not writable. Sleep 1 second
[2021-09-10 02:48:48] [cache_controller::purge_cache_queue_job] Queue file not writable. Sleep 1 second
[2021-09-10 02:48:49] [cache_controller::purge_cache_queue_job] Queue file not writable. Sleep 1 second
[2021-09-10 02:48:50] [cache_controller::purge_cache_queue_job] Queue file not writable. Sleep 1 second
[2021-09-10 02:48:51] [cache_controller::purge_cache_queue_job] Queue file not writable. Sleep 1 second

    The consequence of this is that WordPress dashboard appears to be loading for a long time (anywhere between 30 – 60 seconds), as the PHP functions being called by WP CSPC (e.g. purge_cache_queue_write()) is being held up waiting for the queue file to be available.

    This isn’t a server performance issue as the server is in fact doing no processing, but the CPU and memory resources remain held up to that PHP process nonetheless, at times leading the server to exhaust its memory.

    I would suggest to implement the purge queue in the database instead of using a file on disk. This should allow you to queue multiple cache purge actions without having to wait for the previous item in the queue to complete.

Viewing 5 replies - 1 through 5 (of 5 total)
  • Plugin Contributor iSaumya

    (@isaumya)

    Hi @silverks,
    When it comes to data manipulation/operation there are mainly two ways to do it. Either at the database level or at the file level.
    Database operations are the slowest mode of operation and will take a much longer period of time to execute especially when you will simultaneous operation where chances of deadlock are higher.
    Though file operations are much faster, but it also has the deadlock condition that is two people cannot edit the same file at the same time and they needs to be performed sequentially.
    If you have a side that has multiple admin users making multiple edits simultaneously which may lead constant modification of the cache queue this problem that you are facing is quite common.
    In situations like these what I will recommend is inside the plugin settings:
    1. Disable the Disable cache purging using queue option [under cache tab]

    Then check if you are still facing slowness try disabling the Purge HTML pages only option.

    Thread Starter silverks

    (@silverks)

    Yes, I have tried disabling cache purging using queue, but this has the consequence of making every cache-affecting actions taking longer to complete as remote connection needs to be made to Cloudflare to purge the cache.

    The issue was not that writing to file or database was slow – however slow those actions are, they take much less than a second. It was that only one purging action can be queued at any one time. Until that queued purge action is completed (and file lock released), other actions which require writing to the queue has to wait, and that waiting is what’s artificially slowing the process. In my sample, it took 15 – 30 seconds of waiting.

    I would suggest to allow for multiple actions to be queued, with another process (possibly through cron) that will then pickup and process the queued actions sequentially, independent of user actions. This is why I was suggesting to implement the queue through the database since it’s trivial to add more rows to a table compared to writing serialized arrays to file.

    • This reply was modified 3 years, 5 months ago by silverks.
    Plugin Contributor iSaumya

    (@isaumya)

    We have tried that idea but the major problem with multiple actions to be queued is that it is almost impossible to keep track of what has been purged and what has not been purged. I have tried the database route as well as it was significantly slower.

    Thread Starter silverks

    (@silverks)

    If it takes more than a few milliseconds to write to database, something’s probably not right there. You can keep track of what has been purged by adding a column that identifies each purgeable page, e.g. the URL.

    Plugin Contributor iSaumya

    (@isaumya)

    Will talk about this internally. Will let you know if we have a beta version with this feature to test.

Viewing 5 replies - 1 through 5 (of 5 total)
  • The topic ‘Waiting for queue file to be writable slows down admin actions significantly’ is closed to new replies.