cwjordan

Thanks. I did also mention this on the buddypress “requests & feedback” forum. We’ll see if anything comes of that: https://buddypress.org/support/topic/request-for-way-to-disable-caching-of-xprofile-data-for-get_all_for_user/

get_all_for_user doesn’t look pluggable to me – it would have to have if ( ! function_exists( in it right? However it is also nice and simple. All I did to test was just copy the old version of the function onto the end of export-user-data.php like below and then called it instead of the buddypress one $bp_data = $this->get_all_for_user( $user->ID );:

/**
         * BP_XProfile_ProfileData::get_all_for_user()
         *
         * Get all of the profile information for a specific user.
         */
        private static function get_all_for_user( $user_id ) {
                global $wpdb, $bp;

                $results      = $wpdb->get_results( $wpdb->prepare( "SELECT g.id as field_group_id, g.name as field_group_name, f.id as field_id, f.name as field_name, f.type as field_type, d.value as field_data, u.user_login, u.user_nicename, u.user_email FROM {$bp->profile->table_name_groups} g LEFT JOIN {$bp->profile->table_name_fields} f ON g.id = f.group_id INNER JOIN {$bp->profile->table_name_data} d ON f.id = d.field_id LEFT JOIN {$wpdb->users} u ON d.user_id = u.ID WHERE d.user_id = %d AND d.value != ''", $user_id ) );
                $profile_data = array();

                if ( !empty( $results ) ) {
                        $profile_data['user_login']    = $results[0]->user_login;
                        $profile_data['user_nicename'] = $results[0]->user_nicename;
                        $profile_data['user_email']    = $results[0]->user_email;

                        foreach( (array) $results as $field ) {
                                $profile_data[$field->field_name] = array(
                                        'field_group_id'   => $field->field_group_id,
                                        'field_group_name' => $field->field_group_name,
                                        'field_id'         => $field->field_id,
                                        'field_type'       => $field->field_type,
                                        'field_data'       => $field->field_data
                                );
                        }
                }

                return $profile_data;
        }

Yes, I based it on what was in the github version.

Not sure I want to take it up with BP or not. I feel like they’ll just tell me no. I’ll think about it.

Thanks Q,

I’ve tried it out and that seems to work okay for me.

I believe that sanitize code you were asking about should changed to be:

// update_option sanitizes the option name but not the option value ##
                foreach ( $options as $field_name => $field_value ) {

                    // so do that here. ##
                    if ( is_array( $field_value ) ) {

                        foreach ( $field_value as $field_array_key => $field_array_value ) {

                            $options[$field_name][$field_array_key] = sanitize_text_field( $field_array_value );

                        }

                    } else {

                        $options[$field_name] = sanitize_text_field( $field_value );

                    }

                }

Oh, and as an experiment I tried pulling the version of BP_XProfile_ProfileData::get_all_for_user from Buddypress 1.9.2 and putting it in as a function in your code, and calling that instead of the Buddypress 2.0.1 version. Result was I saw a speed improvement on a database of 13302 users on my slow and ancient development server from about 23 minutes to about 3 minutes. So I was right, it did get much slower when Buddypress 2 came out. Pity.

Q,

It gives me an error that it can’t load css/export-user-data.css. That looks like a new file, could you please add it to github also?

Regarding that way the sanitize works, I can see where that is unclear. I think I did it that way because I first wrote it to save just one report, then I changed it to do multiple reports and that was an easy way.

The get_option( 'q_eud_options',array() ); in the constructor is going to pull an array of report names, and each report name is itself an array. It puts that into q_eud_options. When I update options I just need to update the one key in q_eud_options for a given report name, and then save the whole thing. So when it came to updating and sanitizing that data it was easier to have an array, $options of one element, whose key was the report name to be updated, and use key($options) (which will return the only key in a one element array) to get that name instead of adding a separate argument for $report_name when I called the function.

Looks like you aren’t planning to support multiple report names anyway, so you don’t need that top level of array at all.

Q,

Thanks! I’ll take a look.

Regarding using wp_usermeta instead of wp_options, that’s up to you. The use case I had in mind was that one person might want to create a report, and then hand it off to someone else to run. In that case having the same report visible to more than one person would be useful. I can see why some people might want the reports to be only visible to them though.

Yes, those changes are in the copy on dropbox.

I looked a bit into the memory usage issue and I don’t see a better solution than the one Eli proposed. If I understand what Buddypress did correctly for 2.0, they rewrote BP_XProfile_ProfileData::get_all_for_user so that instead of making it’s own SQL queries it calls BP_XProfile_ProfileData::get which calls BP_XProfile_ProfileData::get_data_for_user. get_data_for_user in turn checks if the data is in the cache already and if not it makes the SQL query, populates the cache with it, queries the cache for it and returns the data it got from the cache.

I guess I can see why they did it that way, but that doesn’t work well for exporting most or all of the data from a site, it just fills up the cache. I suppose an alternative would be to do our own SQL query, but it’s better to use the Buddypress functions to access the Buddypress data. I’m not using a cache plugin, I don’t know if that would make a difference.

Okay, I’ve fixed the problem where my Buddpress extended profile fields that had serialized data in them were showing up as “Array”. I’ve also fixed the problem where selecting an offset and total number of users wasn’t working. URL is the same as above.

I’m going to take a little look at what is up with memory usage & Buddypress but I fear that may be beyond me. Looks like it’s just caching everything very aggressively. I don’t see a way to disable that for a particular query.

Okay, I see the problem with the offset. Line 278 of export-user-data.php has:

$args['number'] = $limit_total - $limit_offset;

So when you have an offset of 100 and a total of 100 $args[‘number’] is set to 0, so no users are found.

That line needs to be changed to

$args['number'] = $limit_total;

Okay, I’ve updated it to the latest plugin version, 0.9.5. I also included -Eli’s code for clearing the cache to work around the out of memory problem (I didn’t include that part very prettily, and I feel like the underlying cause could use more research, so I don’t know if you’d want to release it like that, but it does work for me. I really think there is something in the Buddypress code that is not playing nicely with us).

I just noticed when I exported something with serialized data in one of the fields it showed up as “Array”, instead of as the serialized data. I wonder if I broke something with my change, I’ll have to look through that more carefully tomorrow – be wary.

URL is the same as before: https://www.dropbox.com/s/7falfqdoojo7mse/export-user-data.php?dl=0

Thanks for posting this, I was running into this problem and this helps. I _think_ I never saw it before Buddypress 2.0 came out. The release notes say they changed some things about how caching works with that release. Anyway, nice fix.

About 9 weeks ago I did a version of a way to save which fields you’ve selected. It’s based on version 0.9.4, so it’s a release back. I’ve got it up at https://www.dropbox.com/s/7falfqdoojo7mse/export-user-data.php
for the moment. I’m not intending a fork of the code or anything, but it might help someone.

I just tested pie-register version 1.31.3 and this problem has been fixed. Thanks! I’m marking this topic as being resolved.

Thanks for taking a look. Hiding that by default is fine with me.

I have not run into a problem with foreign characters in exports, but I won’t be likely to have foreign characters in my data.

Q, I have put the modified export-user-data.php file here: https://pastebin.com/QyM7aRrh