• ResolvedPlugin Author Jordy Meow

    (@tigroumeow)


    Hello,

    This is a special support thread for all of you who can’t go beyond the “Read medias…” issue.

    I know why, this is basically I run a simple request to get all the IDs of all the media in the library. Normally, that SQL request should execute, but in the case you have a huge Media Library and not a powerful server, the request takes too much time and/or memory and get cut, so basically the scanning never starts.

    There is a way to fix this, but it’s not easy. I need basically to gather those IDs little by little. But that will work.

    If you have this issue, please write “+1” (or anything else for that matter) in this thread. That way I will keep you updated when this issue is fixed, and if it’s not, we can continue to talk about it here.

    Thank you ??

Viewing 8 replies - 1 through 8 (of 8 total)
  • +1

    I have run into a similar issue with various plugins that attempt a mass SQL query. I’m migrating and still importing images (having over 150k images right now, without counting the thumbnails generated..) so the import plugin would not run properly with this size of db, naturally. What I did was, instead of querying for all (-1) posts, to query a limited number of posts (i.e. 2500) and by manually setting an offset in the php file’s code to scan for a specific ‘group’ of posts each time.

    I haven’t looked at Media Cleaner’s code yet, but I wonder if something similar can be achieved, even manually by editing the relevant SQL request part.

    Thank you for looking into this.

    Plugin Author Jordy Meow

    (@tigroumeow)

    @techsmurfy You can have a look at the code, the problem is there are many requests to prepare all the data required to check if a media is used or not, for example parsing all the galleries in the post (resolving the IDs of each media in those galleries), resolving all the media in the library and their path, etc. It’s a very tricky and difficult update but I need to find a way to do it.

    I was able to get past ‘read medias’ by adding a “LIMIT x” to a couple of SQL queries of media-cleaner.php

    Line 174 (function wpmc_get_original_images):
    $posts = $wpdb->get_col( "SELECT id FROM $wpdb->posts WHERE post_type = 'attachment' LIMIT 1000" );

    Line 213 (function wpmc_get_galleries_images):
    $posts = $wpdb->get_col( "SELECT id FROM $wpdb->posts WHERE post_type != 'attachment' AND post_status != 'inherit' LIMIT 1000" );

    Line 276 (function wpmc_wp_ajax_wpmc_scan):
    $results = $wpdb->get_col( "SELECT p.ID FROM $wpdb->posts p WHERE p.post_status = 'inherit' AND p.post_type = 'attachment' LIMIT 1000" );

    (replace 1000 with whatever number you want; I suppose you can set an offset too, like LIMIT 1000, 2000)

    I don’t know if this actually works like it is supposed to, cause I get 0 issues (of course maybe I don’t have any issues, hopefully). I’m also not sure to which functions I should add ‘LIMIT’. By adding it to these 3 functions, I’m getting past ‘read medias’ and it scans the first 1000 medias with no issues.

    When I only edited line 174 and line 213 WITHOUT editing line 276, it still got past ‘read medias’ and started scanning my whole library (145.000 images) or so it says! It then got stuck at 13%, and I’m now restarting the scan to see how far it will get. But if it does indeed scan the whole library, I guess line 174 did the trick.

    Hopefully this will help someone explain the how’s and the why’s.
    I will report if I encounter anything interesting, while waiting for a more permanent fix.

    • This reply was modified 7 years, 10 months ago by techsmurfy.

    @techsmurfy

    Thank you!
    I’m goint to try it too, and see if it works ??

    Plugin Author Jordy Meow

    (@tigroumeow)

    Line 174 is the heavy one; this is the one I will try to break first in the process (of course I can’t do it in LIMIT directly like this in the code, the client side needs to control this, which is not easy you can guess). Line 276 is strangely similar, I need to check why I am doing this request as well but slightly differently.

    However, changing the Line 213 will not give you good results :/

    Thing is for the new version of the plugin I need to asynchronously create a temporary cache and THEN run the cleaner on every single media. Actually that might even be a faster process at the end that now. But it’s a lot of coding implied ?? I need to find myself a few days and do it very properly.

    Yes, maybe line 213 is completely irrelevant. I’m just not sure if by adding LIMIT only to line 174, will it still scan the whole library (it says it does!)?
    @chickenrun np, did it help you?

    A bit off-topic, but everything was seemingly going smoothly on my end, scanning was at 66% (around 100.000/145.000 media), and I couldn’t just wait another half a day to finish, so at the same time I installed a theme, and the plugin just stopped. I thought it had to do with me interrupting something with the installation, but no; I restarted the scan from the beginning and it was simply stuck at 0/145.000. So I’m thinking it is incompatible with the theme I’m going to use. But how can a plugin like this be incompatible with a theme? What would halt its scanning process? Should I contact the theme’s dev? I’m not sure what we do in these cases, reach out to the plugin’s dev or the theme’s dev?

    I have reverted back to wordpress’ default theme and restarted the scan from the beginning :/ but it will be a blow if I won’t be able to use the plugin in the future due to incompatibility.

    Plugin Author Jordy Meow

    (@tigroumeow)

    Yes, maybe line 213 is completely irrelevant. I’m just not sure if by adding LIMIT only to line 174, will it still scan the whole library (it says it does!)?

    No, it will only scan the first 1,000 ?? I mean, whatever LIMIT you used.

    Should I contact the theme’s dev? I’m not sure what we do in these cases, reach out to the plugin’s dev or the theme’s dev?

    That’s strange, yes, it should work exactly the same with any theme. Maybe that theme is adding some special shortcodes, and my plugin is also resolving shortcodes (can be disabled in the options). Maybe there is a problem linked to that? Did you check your PHP Error Logs?

    @tigroumeow

    No, it will only scan the first 1,000 ?? I mean, whatever LIMIT you used.

    Yeah but despite setting the LIMIT to 1000, it still says it is scanning through the whole library (/145408). Maybe this has to do with line 276 which I left untouched? Hence the confusion.

    Edit after further testing:
    Case 1. Adding LIMIT to line 174 AND line 213: as mentioned. It starts scanning (the whole library -or so it says) with speed at around 80 media per 30 seconds.
    Case 2. Adding LIMIT to line 174 ONLY: Usually stuck @ Read medias. A couple of times it started scanning (the whole library -or so it says) with ridiculously low speed, around 5 media per 30 seconds max.
    Case 3. Adding LIMIT to line 213 ONLY: Stuck @ Read media.
    Case 4. Adding LIMIT to line 174, line 213 AND line 276: as mentioned, it starts scanning (the LIMIT number, i.e. 1000) with speed at around 80 media per 30 seconds.

    I tried setting the limit to 10000, but it was too much I guess, so I reset it back to 1000. All this means, I guess, that line 213 is not irrelevant. I do have to edit both lines for it to start smoothly. Hope this helps.

    That’s strange, yes, it should work exactly the same with any theme. Maybe that theme is adding some special shortcodes, and my plugin is also resolving shortcodes (can be disabled in the options). Maybe there is a problem linked to that? Did you check your PHP Error Logs?

    Indeed it does add quite a number of shortcodes, but I have that option disabled (unchecked) in the options anyway. As for my error_log… for some strange reason it stopped logging since this afternoon! Have no idea why, I deleted it and hoping it will regenerate, if not I will contact my host.

    • This reply was modified 7 years, 10 months ago by techsmurfy.
    • This reply was modified 7 years, 10 months ago by techsmurfy.
    • This reply was modified 7 years, 10 months ago by techsmurfy. Reason: Further testing
Viewing 8 replies - 1 through 8 (of 8 total)
  • The topic ‘About issues with huge Media Library’ is closed to new replies.