CSV import

(@jagwaugh)

7 years, 7 months ago

Ok, thanks.

(That’ll take me a while to digest)

Thread Starter

(@jagwaugh)

Just as an aside for anyone else trying to import a heirarchy of albums.

If you create a tree of folders:

Level 0
    Level 1A
        Album 1a1
        Album 1a2
    Level 2A
        Album 2a1
        Album 2a1

and put your photos inside 1a1, 1a2, 2a1, 2a2 but no photos in Level 0, Level 1A or Level 2A

If you zip that structure into “Level 0.zip”, then upload and import the .zip, then wppa will create the parent albums Level 0, Level 1A and Level 2A and add the lowest levels with the parent albums accordingly.

If you include a correctly formatted .csv either a single, one per parent, or a master, then when you import the .zip wppa will apply the values in the .csv to the photos automatically.

Like magic.

This reply was modified 7 years, 6 months ago by jagwaugh.
This reply was modified 7 years, 6 months ago by jagwaugh. Reason: format

Plugin Author

(@opajaap)

It just does what i promised it would do…

Thread Starter

(@jagwaugh)

Naja… it happens so rarely that one doesn’t expect it.

Is there a way to (temporarily) suspend the wppa_remake_index_photos, and wppa_remake_index_albums during a mass import?

From the look of the logfile each individual image triggers these two jobs. I’m working with a mass upload of about 5K albums with about 5 images per album and the machine is balls to the wall so that it times out a lot.

I’d like to be able to suspend the individual remake_index jobs, then once the imports are done (i.e. moved out of the depot folder) I can trigger the clean, album, and photo index jobs with cron.

Plugin Author

(@opajaap)

Do not bother about the indexes. They will be updated by cron jobs automayicly. The cron jobs do only the changed/added photos and should not bother your import activity.

Thread Starter

(@jagwaugh)

I think I found why my machine was adding photos/albums so slowly.

Yesterday morning when I first started this import I ran into a disk full condition during the import. It’s an AWS EC2 so I just bumped the disk size and did a reboot.

For some reason wppa_cleanup_index was triggered (I can’t remember – I may have started it myself after expanding the disk), and it was apparently causing delays on the import.

I did delete a lot of albums before I started the import, so it may still have been dealing with that before I started the import.

I just left the machine overnight and started the import again now. performance is a LOT better. It is now adding about 40 photos/min, yesterday it was taking about 1min/photo.

Plugin Author

(@opajaap)

40 / min is GOOD!

Thread Starter

(@jagwaugh)

I would be more inclined to say “breathtaking” as it’s only a free tier t2.micro (Single vCPU and 1 GiB RAM).

But something has gone wrong again, the import has slowed down to a crawl again.
VIII>A>8.3 shows:

8.3 Clean Index Remove obsolete entries from index db table. Locked! Locked! Cron job 26329

The bottom of wppa-log.txt shows:

{b}Cron{/b}: on:May 7, 2017 6:51 am: cron-job: {b}wppa_remake_index_photos{/b} completed
{b}Cron{/b}: on:May 7, 2017 6:51 am: cron-job: {b}wppa_cleanup_index{/b} continued. Allowed runtime: 894s. Cron id = 1494139903.7865159511566162109375
{b}Cron{/b}: on:May 7, 2017 6:54 am: cron-job: Could not complete scan of index item # {b}403{/b}, slug = {b}am{/b}, count = {b}15189{/b}, photo id = {b}14460{/b}, next element # = {b}13197{/b},
{b}Cron{/b}: on:May 7, 2017 6:54 am: cron-job: {b}wppa_cleanup_index{/b} re-scheduled by {b}wppa_do_maintenance_proc( 'wppa_cleanup_index' ){/b} on line {b}1038{/b} of wppa-maintenance.php called by apply_filters
{b}Cron{/b}: on:May 7, 2017 6:54 am: cron-job: {b}wppa_remake_index_photos{/b} started. Allowed runtime: 894s. Cron id = 1494140076.2668220996856689453125
{b}Cron{/b}: on:May 7, 2017 6:54 am: cron-job: Indexed photo {b}19713{/b}
{b}Cron{/b}: on:May 7, 2017 6:54 am: cron-job: Adding index slug {b}12~1204097165{/b} for photo {b}19714{/b}
{b}Cron{/b}: on:May 7, 2017 6:54 am: cron-job: Indexed photo {b}19714{/b}
{b}Cron{/b}: on:May 7, 2017 6:54 am: cron-job: Adding index slug {b}1~1204097165{/b} for photo {b}19715{/b}
{b}Cron{/b}: on:May 7, 2017 6:54 am: cron-job: Indexed photo {b}19715{/b}
{b}Cron{/b}: on:May 7, 2017 6:54 am: cron-job: Adding index slug {b}2~1204097165{/b} for photo {b}19716{/b}
{b}Cron{/b}: on:May 7, 2017 6:54 am: cron-job: Indexed photo {b}19716{/b}
{b}Cron{/b}: on:May 7, 2017 6:54 am: cron-job: {b}wppa_remake_index_photos{/b} completed
{b}Cron{/b}: on:May 7, 2017 6:54 am: cron-job: {b}wppa_cleanup{/b} started.
{b}Cron{/b}: on:May 7, 2017 6:54 am: cron-job: {b}wppa_cleanup{/b} completed.
{b}Cron{/b}: on:May 7, 2017 6:55 am: cron-job: {b}wppa_cleanup_index{/b} continued. Allowed runtime: 895s. Cron id = 1494140105.1863238811492919921875
{b}Cron{/b}: on:May 7, 2017 6:55 am: cron-job: Continuing cleanup index at slug = {b}am{/b}, element # = {b}13197{/b}
{b}Cron{/b}: on:May 7, 2017 6:56 am: cron-job: Could not complete scan of index item # {b}403{/b}, slug = {b}am{/b}, count = {b}15189{/b}, photo id = {b}8073{/b}, next element # = {b}6810{/b},
{b}Cron{/b}: on:May 7, 2017 6:56 am: cron-job: {b}wppa_cleanup_index{/b} re-scheduled by {b}wppa_do_maintenance_proc( 'wppa_cleanup_index' ){/b} on line {b}1038{/b} of wppa-maintenance.php called by apply_filters
{b}Cron{/b}: on:May 7, 2017 6:56 am: cron-job: {b}wppa_remake_index_photos{/b} started. Allowed runtime: 895s. Cron id = 1494140194.3013091087341308593750
{b}Cron{/b}: on:May 7, 2017 6:56 am: cron-job: {b}wppa_remake_index_photos{/b} completed
{b}Cron{/b}: on:May 7, 2017 6:56 am: cron-job: Cron fixed treecounts for 505
{b}Cron{/b}: on:May 7, 2017 6:56 am: cron-job: {b}wppa_cleanup{/b} started.
{b}Cron{/b}: on:May 7, 2017 6:56 am: cron-job: {b}wppa_cleanup{/b} completed.
{b}Cron{/b}: on:May 7, 2017 6:56 am: cron-job: {b}wppa_remake_index_albums{/b} started. Allowed runtime: 884s. Cron id = 1494140194.3013091087341308593750
{b}Cron{/b}: on:May 7, 2017 6:56 am: cron-job: Indexed album {b}4819{/b}
{b}Cron{/b}: on:May 7, 2017 6:56 am: cron-job: {b}wppa_remake_index_albums{/b} completed
{b}Cron{/b}: on:May 7, 2017 6:57 am: cron-job: {b}wppa_cleanup_index{/b} continued. Allowed runtime: 894s. Cron id = 1494140225.4872140884399414062500
{b}Cron{/b}: on:May 7, 2017 6:57 am: cron-job: Continuing cleanup index at slug = {b}am{/b}, element # = {b}6810{/b}

Am I just expecting too much from the machine?

Plugin Author

(@opajaap)

The log shows correct.
However, try this:
You can temporary skip the cron jobs as follows: (just try, and revert when done):
Edit file wppa-cron.php:


<?php
/* wppa-cron.php
* Package: wp-photo-album-plus
*
* Contains all cron functions
* Version 6.6.24
*
*
*/

// Are we in a cron job?
function wppa_is_cron() {

	if ( isset( $_GET['doing_wp_cron'] ) ) {
		return $_GET['doing_wp_cron'];
	}
	if ( defined( 'DOING_CRON' ) ) {
		return DOING_CRON;
	}
	return false;
}

// Activate our maintenance hook
add_action( 'wppa_cron_event', 'wppa_do_maintenance_proc', 10, 1 );

// Schedule maintenance proc
function wppa_schedule_maintenance_proc( $slug, $from_settings_page = false ) {
global $is_reschedule;

Add right below global $is_reschedule;:
return;

This will prevent the majority of cron jobs. Afterwards, revert the edit, and just edit one photo desc and one album desc to trigger the index mechanism. Do not run it from Table VIII.

Thread Starter

(@jagwaugh)

Just to be clear, Do I edit the file while the current import is running and the changes are effected on the fly, or do I need to restart the machine, edit the file, and re run the import?

Plugin Author

(@opajaap)

Just edit on the fly.

Just for your information:

8.3 Clean Index Remove obsolete entries from index db table. Locked! Locked! Cron job 26329
means Clean index runs as a cron job, do not manually interfere. You can overrule by clicking the first button, but you should not.
Remake index albums/photos works very fast as cron job. It only processes items where the datetame last indexed is prior to datetime last modified.
Do not run remake index interactively ( the Doit! button ) beause this will first clear the index completely (remake(!)), if needed, you can run it as cron job but this should not be needed.
I think the cleanup index job kills you, because it scans the entire index and you have a very large max_execution_time (900 s.), so the edit will most likely help you out.

Plugin Author

(@opajaap)

Did you disable cron jobs and, if so, did it work? If usefull, I could make it a setting…

Thread Starter

(@jagwaugh)

I did, and it made a big difference, roughly 3x faster.

I also changed the AWS instance to a 2G one, which also helped.

It was reading a LOT of EXIF data during the import, as far as I could tell by watching phpMyAdmin.

I’m a edge case, but it would be nice to have a “Mass import” setting to hold off on any tasks other than just creating the Albums and copying the photos. Once the copy is complete then the rest could be pushed manually, or run as a cron (In my case the cron took about 10 hours once I’d finished the raw import.

Plugin Author

(@opajaap)