Ideas for Improvement/Scalability

• WordPress 5.3.2 (single site)
• PHP 7.4.4
• Ubuntu 18.04.04 (fairly stock), nginx 1.16.1, MySQL 8.0.19
• ImageMagick 6.9.7

Greetings. This plugin is wonderful and has saved me massive time over the years (RIP Alex Mills). However, I have been struggling with it on a huge site that I’m currently working on. The site has ~123k images*, and naturally/sadly, many authors upload massive, unoptimized images (even though it takes seconds, but I digress; also yes, image optimization is in the works, but I’m dealing with these cards right now).

Scalability

Due to a site rewrite, I’ve been trying to regenerate thumbnails for this site for the past 1.5 weeks. For various reasons, I haven’t been able to get the web UI to complete all images (I was able to get it to finish attached featured images only, about ~36k, which took 16 hours; I got as high as about 65k when generating all, but never past that) – Internet or VPN connection drops, imagick cache capacity/configuration errors, etc, etc. I’ll get it to go eventually (probably by tweaking it to allow some sort of paging – I only need it to complete once and then the stock plugin will be fine), but I came here with some ideas.

Naturally, the CLI tool was my first choice since it is faster and will continue to run if my connection drops, but ImageMagic kept hitting a fatality on a certain, unknown image:

13732/122925 No thumbnail regeneration needed for "some-filename" (ID 1234). php: ../../magick/quantum.c:216: DestroyQuantumInfo: Assertion 'quantum_info != (QuantumInfo *) NULL' failed. Aborted (core dumped)

I imagine that this is an issue with imagick/config, but I haven’t been able to figure it out. For such purposes:

1. It would be nice if there was some sort of way to resume from the last successful file that was processed, both in the web UI and CLI.
2. If I get a connection drop, I tried hitting the “Pause” button then “Resume” in the web UI, but it doesn’t continue (presumably due to the JavaScript errors created by the connection fail). It would be nice if this was resumable in this case.
3. It might be nice if there was a way (even if undocumented) to have it begin at a certain image ID, when desperate. Perhaps something as simply as a querystring argument added to the page for the tool, like ?start=100000, or whatever makes sense. For the CLI, perhaps an --offset= switch or similar?
4. It isn’t clear to me which image is causing the above core dump – if it was, I’m just delete or replace it (I suppose that it’s the next one in the database?). Since I’m not familar with the incatracies, I’m not sure if you can add a switch like --ignore-errors or something to have it continue to the next image when imagick returns a fatal exception?

I’m not suggesting that all or any of these ideas are necessarily or ideal, just brainstorming for those of us with a huge amount of crappy images to deal with.

Documentation Ambiguity

The documentation for the media regenerate command is ambiguous to me (?):

The default for the web UI appears to have the “Delete thumbnail files for old unregistered sizes” checkbox unticked. However, with the CLI --skip-delete, it says, “Defaults to false.” What does “false” mean in this context? I would personally read this default as “do not skip deleting unregistered sizes” (which is a different default than the web UI). The (sort of) double negative is also a bit confusing, too. At minimum, it seems like the defaults (and ideally phrasing) should match (unless they do and I’m interpreting incorrectly)?

Also, the docs say that --skip-delete “Defaults to false,” but it is not clear from the wording how to set it to true. For example, to “enable skip deleting” (an ambiguous thought in itself), do I simply add --skip-delete as an argument (which could be interpreted my me as setting it to a null value, which would presumably default to false), or does it require a boolean value such as --skip-delete=true or --skip-delete=1, etc? In other words, it appears that --skip-delete does the opposite of what --skip-delete=false would (the documented default), which is confusing for both reasons mentioned.

I understand that changing defaults and argument names is perilous to those currently using it (which may be via automated scripts), but perhaps some clarification example(s) could be added to ensure that the user knows exactly what they are asking for? (In my case, I don’t wish to accidentally delete unregistered sizes for ~123k images!)

*Anyway, just some ideas. I’m aware that:

1. This tool probably wasn’t created for this kind of scale, and:
2. There are better ways to handle this, such as an image proxy/resizer (such as Proton or similar), however, due to a very tight/typical deadline and bureaucracy, that notion is a work-in-progress (further complicated by the fact that this site is headless, but that is an unrelated bit of massive frustration).

I know that most users do not operate on this scale, however, I feel like some minor/simple tweaks might be handy for those who do.

Thank you again,
Daniel

EDIT (since it isn’t letting me reply to this topic):

Another idea to help with the command line switch ambiguity might be to tell the user more verbosely in the confirmation what is about to happen. As an example, this command:
wp media regenerate --skip-delete --only-missing

…currently displays
“Do you really want to regenerate all images? [y/n]”

Perhaps it could be tweaked to include the argument interpretations, such as this or similar:
“Do you really want to regenerate all images (only missing, do not delete original thumbnails)? [y/n]”

• This topic was modified 4 years, 8 months ago by Daniel Hendricks. Reason: Attempt #1 to fix formatting issues
• This topic was modified 4 years, 8 months ago by Daniel Hendricks. Reason: Attempt #2 to fix formatting issues
• This topic was modified 4 years, 8 months ago by Daniel Hendricks. Reason: Attempt #3 to fix formatting issues
• This topic was modified 4 years, 8 months ago by Daniel Hendricks. Reason: Minor typo: WordPress version
• This topic was modified 4 years, 8 months ago by Daniel Hendricks. Reason: Added clarification note for CLI confirmation