ndp

Well, I’ve partially got some information on this;

When I view-source on the scripts, WordPress is mung-ing the code with spurious <p> and </p> tags. Even though I’m using the html editor, I’ve got the visual editor turned off, I’m using ExecPHP.

So I download the “text-control” plugin, and try that out, and guess what – it only works for posts. Not pages. Unless I set the setting site-wide.

How inconvenient.

Yep. That was it.

1. Open PHP.INI in a text editor of your choice (normally you can find php.ini in your php install dir)
2. Change the recursion limit to 200x normal, that is, set: pcre.recursion_limit=20000000
3. Change the backtrack limit to 100x normal, that is, set: pcre.backtrack_limit=10000000
4. Stop and start the Apache (or IIS) service

Lucky me – I have access to my php.ini!

I think I might be hitting this:
https://www.ads-software.com/support/topic/256910?replies=28

Probably why my test program displays so much more data, and why I can get away with more if I run my code straight in a template: the shortcodes make this problem worse somehow.

Sorry I forgot to include the link to the public-facing site – this is the page which I believe will have a problem at a certain point:
https://edtechfuture.org/?page_id=9787

Here’s a version of the same page, where I have raised the limit on the number of characters displayed in the “Brief Description” field. (All I’m doing here, is changing substr($post_excerpt, 0, 250) to substr($post_excerpt, 0, 500)). This version crashes.
https://edtechfuture.org/?page_id=9971

Whether I hit this limit seems to depend on two factors:
The number of items (rows) displayed, and the amount of data per row.
Right now, since I only have a few rows, I am limiting the amount of data per row by abridging the description.

To simplify this, I wrote some test code that can reproduce the error:

<div id="reslist">
<table id="reslist" cellspacing="0" border="0" width="100%" summary="A table to see what Exec PHP's apparent rendering limit is">
<tr>
<th scope="col" abbr="Column 1">Column 1</th>
<th scope="col" abbr="Column 2">Column 2</th>
<th scope="col" abbr="Column 3">Column 3</th>
<th scope="col" abbr="Column 4">Column 4</th>
</tr>

<?php
for ( $i = 0; $i < 100 ; $i++) {
		$x = "abcdefghij";
		$y = $x;
		for( $j = 0 ; $j < 10 ; $j++ ) {
			$y .= $y . " " ;
		}

		?>
		<tr>
			<td><?php print($x); ?></td>
			<td><?php print($x); ?></td>
			<td><?php print($y); ?></td>
			<td><?php print($x); ?></td>
		</tr>

<?php } ?>
<tr><td><?php print($i); ?> items submitted.</td>
</tr></table>
</div>

So when my $i loop is set to a max of 987 (much higher than I thought), and $j to 10, I can reproduce the crash – so this problem has absolutely nothing at all to do with the sql query.

(and I could simply omit the third-column print statement and go to 10,000 rows, and it would chug along just fine; it’s not allocation of memory for this data, it’s the amount that is getting rendered).

When I set $i’s limit to 100, $j craps out at about 13 (because the size of that $y field increments geometrically).

So here’s the kicker. If I render something up near the limit – view source, copy the html, then paste that into the editor of a new page, I have just static html in this page now, no php – then I do an Update Page, and when the screen refreshes, that window is blank. WordPress no-likey. I’m trying to identify where, exactly that limit is.

My wp-config.php has the memory set to 96MB. Changing this to 128 or 256 did not appear to make any difference.

Resolved in post: https://www.ads-software.com/support/topic/314907?replies=4

Solution for this post:
https://www.ads-software.com/support/topic/314907?replies=4#post-1249255

also fixes this problem.

Problem solved. (for me).

Turns out that we just plain don’t want to mess with serialized data stored in database fields, without unserializing it first.

So I wrote a little script to do this:
Instead of using mysqldump | sed | mysql to move a WordPress database from one instance of WordPress to another, I now use this script.

In a nutshell, it makes a “scratch” copy of all the specified (hard-coded) tables. Then it scans through every field in the database for instances of the source-site’s URL. If it’s a “flat” database field (normal, ie. wasn’t serialized), then it does a simple str_ireplace to update the URL to the target-site’s URL. If it’s a serialized field, it unserializes the data, then does str_ireplace on the resulting array, then re-serializes the data, which ensures that the correct lengths on string values are recorded in the serialized data.

The “scratch” tables are then all copied to the target site.

There’s still some problems with this:
I was not able to reliably “fix” serialized fields of the “Object” type. (MagpieRSS, and GD-star rating both persist their data as Objects). This is because when you unserialize() objects, you have to have the class definition available; otherwise you lose the class definition when you re-serialize it, and that object is now useless to whatever class tries to read it back out of the database.
However – none of the stored serialized objects in MY site have any impact on how my site works. If someone else wants to take a stab at this, go ahead.
The other problem is that there is one serialized array, wp_options.option_name=dashboard_widgets_options just can’t be touched by str_ireplace(). I have no idea why. It just returns my original string. All the other serialized arrays (like Register_Plus and TDOMF’s stored form data) convert over just fine.

It’s been a huge relief for me to be able to transfer my content from my staging/development server to my production server without the hassle of manually re-entering this configuration data. Hopefully, this helps other people.

The preconditions and procedure are documented in comments. Basically, you need the execPHP plugin, and you copy this code into a (Private) WordPress page, update the page, then view it – which executes the code. I’m sure there are other ways to do this, but this was the simplest for me.

<?php
/*
Migrate WordPress Site - Assumptions:
1. Source system is running phpExec (or similar) plugin, allowing php code to be embedded in a Page.
2. Tested only on WordPress version 2.8.4
3. Source and Target sites are both hosted in the same MySQL instance.
4. Files have already been synchronized (including correct ownership and permissions issues)
5 (wp-config.php and .htaccess are two files that should not be synchronized)
6. Both systems' Database names, username, password, and URLs are known, and entered into the assignments at the start of this script.
7. All of the tables that you need to transfer are in the $export_tables_list array.
8. Source system's cache is either disabled or cleared.  Target system's cache is disabled.
9. Physical (mysqldump) backups have been taken of both source and target system.
10. Minimal to no user-activity on either system.

Procedure:
1. On Source site, in the Dashboard, use Pages->Add New.
2. Mark the new page as Private, and also disable comments.
3. Give the page a Title (doesn't really matter what).
4. "Update Page".
5. Paste THIS script into the page contents.
6. "Update Page"
7. Click "View Page". (this executes the script)
8. Under NO circumstances should this page be re-loaded.
9. Examine the output to ensure there were no unrecoverable errors.
10. Test both the Source and Target systems to ensure all desired functionality is retained.
11. Edit the page, and delete all the (code) contents of the page.
12. On the TARGET system, UNDER NO CIRCUMSTANCES should the same page_id be loaded.
(the script should not run, but don't load it, just to be safe).
13. Locate the Page in the Pages list, and DELETE this page.
*/

global $wpdb;
/*List all of the tables you would like to migrate to your new site.
Some tables like the bsuite stats tables (etc) - you don't really want
to migrate those, because statistics counters should remain with a given
site.
Your list of tables depends on what plugins you have installed.
Also - it's better practice not to hard-code the table prefix "wp_"
in scripts like this; but use $wpdb->prefix to determine the
table prefix dynamically. But I'm not doing that in this implementation.*/
$export_tables_list = array('wp_comments',
							'wp_gdpt_posts_views',
							'wp_gdpt_users_tracking',
							'wp_links',
							'wp_login_redirects',
							'wp_options',
							'wp_pageblocks_pages_config',
							'wp_postmeta',
							'wp_posts',
							'wp_pt_templates',
							'wp_ratings',
							'wp_tdomf_table_edits',
							'wp_tdomf_table_forms',
							'wp_tdomf_table_sessions',
							'wp_tdomf_table_widgets',
							'wp_terms',
							'wp_term_relationships',
							'wp_term_taxonomy',
							'wp_usermeta',
							'wp_users'
							);

/*SOURCE SITE PARAMETERS*/
/*instead of hard-coding the source url, I'm using the get_bloginfo()
function so that the assignment doesn't get str_replaced in the database
field that contains the post_content of this script in wp_posts. */
$srcURL = get_bloginfo('url');
$srcDBHost = '127.0.0.1';
$srcDBName = '(source site database name goes here)';
$srcCredUN = '(source site database user name goes here)';
$srcCredPW = '(source site database user password goes here)';
/*TARGET SITE PARAMETERS*/
$trgDBName = '(target site database name goes here)';
$trgDBHost = 'localhost';
$trgCredUN = '(target site database user name goes here)';
$trgCredPW = '(target site database user password goes here)';
/*I use the concatenation, here, to prevent this value from being
modified in the database record that stores this code */
$trgURL = 'https://' . 'netp-dev' . '.esd.sri.com';

/*INSTRUMENTATION ARRAY*/
$instr = array(  'num_tables' => 0, 			//number of tables
				'num_columns_o' => 0, 		//overall number of columns
				'num_columns_i' => 0, 		//number of columns, this table
				'num_rows_o' => 0,		 	//overall number of rows selected
				'num_rows_i' => 0, 			//number of rows selected, this table
				'num_fields_o' => 0, 		//overall number of fields checked
				'num_fields_i' => 0,	 	//number of fields checked, this table
				'num_viable_fields_o' => 0, //overall number of fields meeting processing criteria
				'num_viable_fields_i' => 0,	//number of fields meeting processing criteria, this table
				'num_flat_fields_o' => 0, 	//overall number of "flat" (normal) database fields
				'num_flat_fields_i' => 0, 	//number of "flat" database fields, this table
				'num_ser_fields_o' => 0, 	//overall number of serialized fields
				'num_ser_fields_i' => 0, 	//number of serialized fields, this table
				'num_ser_obj_o' => 0, 		//overall number of serialized objects
				'num_ser_obj_i' => 0, 		//number of serialized objects, this table
				'num_ser_arr_o' => 0,		//overall number of serialized arrays
				'num_ser_arr_i' => 0, 		//number of serialized arrays, this table
				'num_elems_upd_o' => 0, 	//overall number of unserialized elements (total).
				'num_elems_upd_i' => 0, 	//number of unserialized elements (total), this table
				'num_unser_fail_o' => 0, 	//overall number of failed unserializations
				'num_ser_upd_fail_o' => 0,	//overall number of serialized records matching after update
				'num_ser_upd_fail_i' => 0,	//number of serialized records matching after update, this table
				'current_table' => 0, 		//current working table (by number)
				'current_column' => 0, 		//current working column (by number)
				'current_rownum' => 0, 		//current working row (by number)
				'num_rows_updated_o' => 0, 	//overall number of database rows updated
				'num_rows_updated_i' => 0, 	//number of database rows updated, this table
				'num_ser_arr_repl_i' => 0,	//number of serialized array updates (not number of elements) , this table
				'num_ser_obj_repl_i' => 0,	//number of serialized array updates (not number of elements) , this table
				'num_flat_repl_i' => 0,		//number of serialized array updates (not number of elements) , this table
				'num_ser_arr_repl_o' => 0,	//number of serialized array updates (not number of elements) , overall
				'num_ser_obj_repl_o' => 0,	//number of serialized array updates (not number of elements) , overall
				'num_flat_repl_o' => 0,		//number of serialized array updates (not number of elements) , overall
			);
$wpdb->flush();   //flush database cache

/*Main Loop:
After a basic safety check, this program basically just takes each table in
the export tables list array, and runs a series of functions on it*/
if($srcURL==$trgURL) {
		/*Then we should skip processing, because this unsafe condition may occur
		during the migration process: the page running this code is a field in the database,
		in wp_posts.post_content. . .  and the $srcURL assignment, above, may be changed
		to that of the target URL.
		This renders the script useless; and dangerous: because the database name
		and credentials variables aren't changed.
		So this fail-safe has been put in place.
		Two additional fail-safes are that I assign $srcURL based on get_bloginfo(),
		and I use concatenation in $trgURL so that that pattern can't be matched
		either.*/
} else {
	foreach ($export_tables_list as $this_table) {
		$instr['num_tables'] += 1 ;
		print('::::::::::::::::::::::::::::::::::::::::::::::::::::<br />');
		print('processing: ' . $this_table . '<br />');
		copy_table($this_table);
		update_fields($this_table, $srcURL, $trgURL, &$instr);
		intermediate_instrumentation_report(&$instr);
		/*make a copy in the target database; overwrites original tables in target*/
		migrate_scratch_table($this_table, $srcDBName, $trgDBName);
		/*retaining source scratch_table as a failsafe backup - these can be easily removed by hand.*/
		drop_scratch_table($this_table); //copy will still exist on target database
	}
}
final_instrumentation_report(&$instr);
/*End*/

/* copy_table()
Makes a copy of the table whose name was passed as a parameter.
The name of the copy has "scratch_" prepended, it is a scratch copy
where we can safely work; modifying the scratch data without touching
the original data.
Does not return anything.
*/
function copy_table($tablename) {
	global $wpdb;
	$scratchTableName = 'scratch_' . $tablename ;
	$mkTableQry = "CREATE TABLE {$scratchTableName} LIKE {$tablename};";
	$popTableQry = "INSERT INTO {$scratchTableName} SELECT * FROM {$tablename};";
	$wpdb->query($mkTableQry);
	$wpdb->query($popTableQry);
} 

/* drop_scratch_table()
As the name suggests, this functions as a cleanup after the modified data
has been transferred to the target database.  However, I leave this
function disabled, so the scratch tables can remain as a backup until
I complete my testing of the new site. I remove the scratch tables by hand.
*/
function drop_scratch_table($tablename) {
	global $wpdb;
	$drpTableQry = "DROP TABLE scratch_{$tablename};";
	$wpdb->query($drpTableQry);
}

/*update_fields()
Takes three string parameters: the name of the current working table, the old URL, and the new URL.
I've also added some additional "i_" parameters for tracking the activity.
This function works exclusively on the scratch copy of the table, and does not access the original copy of the table.
This function processes all of the table's fields, row by row, column by column.
When a serialized field is found, that data is handed off to a function to handle serialized data.
When a flat-field is found, and the old URL is contained in that field, it is replaced with the new
URL.
*/
function update_fields($tablename, $oldURL, $newURL, $instruments){
	global $wpdb;
	$scratchTableName = "scratch_{$tablename}";
	$result_rows = $wpdb->get_results("SELECT * FROM {$scratchTableName};", ARRAY_A);
	$col_names = $wpdb->get_col_info('name');
	$instruments['num_columns_i'] = 0;
	$instruments['num_rows_i']  = 0;
	$instruments['num_fields_i']  = 0;
	$instruments['num_viable_fields_i']  = 0;
	$instruments['num_flat_fields_i']  = 0;
	$instruments['num_ser_fields_i']   = 0;
	$instruments['num_ser_obj_i']  = 0;
	$instruments['num_ser_arr_i']  = 0;
	$instruments['num_elems_upd_i']  = 0;
	$instruments['current_table']  = $instruments['num_tables'];
	$instruments['current_column']  = 0;
	$instruments['current_rownum']  = 0;
	$instruments['num_rows_updated_i']  = 0;
	$instruments['num_ser_upd_fail_i'] = 0;
	$instruments['num_ser_arr_repl_i'] = 0;
	$instruments['num_ser_obj_repl_i'] = 0;
	$instruments['num_flat_repl_i'] = 0;
	$i = 0; //(instantaneous counter for replacements)
	foreach($col_names as $column) {
		$instruments['num_columns_i'] += 1 ;
		$instruments['current_column'] = $instruments['num_columns_i'];
		foreach($result_rows as $result_row) {
			$instruments['current_rownum'] += 1 ;
			$instruments['num_fields_i'] += 1 ;
			//if ((is_string($result_row[$column]))&&(strlen($result_row[$column])>=strlen($oldURL))&&(substr_count($result_row[$column], $oldURL)>0)) {
			if (substr_count($result_row[$column], $oldURL) > 0){
				$instruments['num_viable_fields_i'] += 1 ;
				/*we will only process a "viable" field;  contains the URL*/
				$original_field = $result_row[$column];
				$working_field = $original_field;
				if (is_serialized( $working_field )) {
					$instruments['num_ser_fields_i'] += 1 ;
					if(substr($working_field, 0, 1)=="O") {
						$instruments['num_ser_obj_i'] += 1 ;
						$working_field = fix_serialized_field_o($original_field, $oldURL, $newURL, &$instruments);
					} else {
						$instruments['num_ser_arr_i'] += 1 ;
						$working_field = fix_serialized_field_a($original_field, $oldURL, $newURL, &$instruments);
					}
					if($working_field == $original_field){
							$instruments['num_ser_upd_fail_i'] += 1 ;
							print('Serialized field was not updated:<br />');
							print('- in table #: ' . $instruments['current_table'] . '<br />');
							print('- at column #: ' . $instruments['current_column'] . '<br />');
							print('- at row #: ' . $instruments['current_rownum'] . '<br />');
							print('- original data: ' . strip_tags(substr($original_field, 0, 90)) . '<br />');
							/*if we got here; we detected a serialized field that had our old URL in it.
							But, when we unserialized it, searched all of the elements, that old URL
							wasn't found - which means one of two things:
							1. Either str_replace() failed to enumerate all of the elements of that
							unserialized object, in which case, we'll only see THIS error message.
							OR
							2. unserialize() failed for this string; which will also generate an errror
							message similar to this one in the fix_serialized_field_.() function, so you'll
							see a double-error.

							unserialize() will fail if the data was corrupted at some point in the past
							(example: if a flat search/replace was done on the string, and the length of
							one of the string elements was changed; without changing the length-value).
							unserialize() will succeed with Object-type serialized strings, but older versions
							of PHP will not preserve the object type, and unless the class that defined that
							object is included, PHP wont know what to do with it.  get_object_vars() was
							my attempt to fix that problem. I'm not sure if it works - but I assume that
							this whole issue is moot with Object fields, because WordPress will simply
							generate a new object.  The real issue is when a plugin saves options as a
							serialized array, and expects them to remain unchanged.
							If corruption of serialized Objects *IS* a problem, and if get_object_vars(),
							in fact, isn't giving us a good representation of the data that we can re-serialize
							in a useful way, then we could try this function instead?:
							function toObject($data){ return is_array($data) ? (object) array_map(__FUNCTION__, $data) : $data;}
							. . . don't know.*/
					} else {
						/*update was successful*/
					}
				} else {
					/*update the flat field*/
					$instruments['num_flat_fields_i'] += 1 ;
					$working_field = str_ireplace($oldURL, $newURL, $original_field, $i);
					$instruments['num_flat_repl_i'] += $i ;
				}
				/*Update the database field*/
				if ($working_field!=$original_field) {
					$new_data = array( $column => $working_field );
					$where_data = array( $column => $original_field );
					// UPDATE DATABASE ROW
					$wpdb->update( $scratchTableName, $new_data, $where_data );
					$instruments['num_rows_updated_i'] += 1 ;
				}
			} else {
				/*was not a "viable" field.*/
			}
		}
		$instruments['num_rows_i'] = $instruments['current_rownum'];
		$instruments['current_rownum'] = 0;
	}
	/*update totals*/
	$instruments['num_columns_o'] 		= $instruments['num_columns_o'] 		+ $instruments['num_columns_i'];
	$instruments['num_rows_o']  		= $instruments['num_rows_o'] 			+ $instruments['num_rows_i'];
	$instruments['num_fields_o']  		= $instruments['num_fields_o'] 			+ $instruments['num_fields_i'];
	$instruments['num_viable_fields_o']	= $instruments['num_viable_fields_o'] 	+ $instruments['num_viable_fields_i'];
	$instruments['num_flat_fields_o']  	= $instruments['num_flat_fields_o'] 	+ $instruments['num_flat_fields_i'];
	$instruments['num_ser_fields_o']   	= $instruments['num_ser_fields_o'] 		+ $instruments['num_ser_fields_i'];
	$instruments['num_ser_obj_o']  		= $instruments['num_ser_obj_o'] 		+ $instruments['num_ser_obj_i'];
	$instruments['num_ser_arr_o']  		= $instruments['num_ser_arr_o'] 		+ $instruments['num_ser_arr_i'];
	$instruments['num_elems_upd_o'] 	= $instruments['num_elems_upd_o'] 		+ $instruments['num_elems_upd_i'];
	$instruments['num_rows_updated_o'] 	= $instruments['num_rows_updated_o'] 	+ $instruments['num_rows_updated_i'];
	$instruments['num_ser_upd_fail_o'] 	= $instruments['num_ser_upd_fail_o'] 	+ $instruments['num_ser_upd_fail_i'];
	$instruments['num_ser_arr_repl_o'] 	= $instruments['num_ser_arr_repl_oi']	+ $instruments['num_ser_arr_repl_i'];
	$instruments['num_ser_obj_repl_o']	= $instruments['num_ser_obj_repl_o']	+ $instruments['num_ser_obj_repl_i'];
	$instruments['num_flat_repl_o'] 	= $instruments['num_flat_repl_o']		+ $instruments['num_flat_repl_i'];

}

/*fix_serialized_field_o()
accepts four parameters, $serfield is a string, selected from the database, containing serialized data.
$srcURL and $trgURL are self-explanatory, as is the $instruments instrumentation array.
"_o" is the Object version - to deal with serialized objects.
This function unserializes the data, copying it into a (usually) multidimensional array, then each element of that
structure is searched for the source URL, and instances are replaced with the target URL.
The array is then re-serialized, so that string elements whose field-lengths were changed by the
search/replace operation, can be stored with their correct length, so the data can be
correctly unserialized.
Returns the modified, re-serialized string if successful, and if unserialize() failed, it returns the
original serialized data string.*/
function fix_serialized_field_a($serfield, $srcURL, $trgURL, $counters){
	$unserialized_data = "";
	$unserialized_data = unserialize($serfield);
	if($unserialized_data){
		$j = 0;
		$counters['num_elems_upd_i'] = count($unserialized_data, 1); //not a count of elements updated
		$updated_unserialized_data = str_ireplace($srcURL, $trgURL, $unserialized_data, $j);

		$counters['num_ser_arr_repl_i'] += $j ;
		if ($updated_unserialized_data==$unserialized_data){
			/*the field unserialized - but str_replace didn't work.  Why not?  I don't know.
			This seems to happen every time for wp_options.option_name = dashboard_widgets_options
			*/
			print('<br />');
			print('str_replace() failed for this field.<br />');
			print('unserialized field was not updated:<br />');
			print('- in table #: ' . $counters['current_table'] . '<br />');
			print('- at column #: ' . $counters['current_column'] . '<br />');
			print('- at row #: ' . $counters['current_rownum'] . '<br />');
			print('- number of replacements counted: ' . $j . '<br />');
			print('- original data: ' . strip_tags(substr($serfield, 0, 90)) . '<br />');
			print('DUMP of updated_unserialized_data: <br />');
			var_dump($updated_unserialized_data);
		}
		$results = serialize($updated_unserialized_data);
	} else {
		$counters['num_unser_fail_o'] += 1 ;
		/*Sometimes, a field just fails to unserialize, often this is because there's previous "corruption":
		a string may have been updated while it was in serialized state, so the length no longer matches.
		in these cases; if your site is working, with this data in the database; then don't worry about it,
		you can probably just delete this row, because a new row was probably already created to replace
		it when WordPress tried to unserialize() it to read the options. */
		print('<br />');
		print('unserialize() failed for this field.<br />');
		print('Serialized field was not updated:<br />');
		print('- in table #: ' . $counters['current_table'] . '<br />');
		print('- at column #: ' . $counters['current_column'] . '<br />');
		print('- at row #: ' . $counters['current_rownum'] . '<br />');
		print('- original data: ' . strip_tags(substr($serfield, 0, 90)) . '<br />');
		$results = $serfield;
	}
	return $results;
}

/*fix_serialized_field_o()
accepts four parameters, $serfield is a string, selected from the database, containing serialized data.
$srcURL and $trgURL are self-explanatory, as is the $instruments instrumentation array.
"_o" is the Object version - to deal with serialized objects.
This function unserializes the data, copying it into a (usually) multidimensional array, then each element of that
structure is searched for the source URL, and instances are replaced with the target URL.
The array is then re-serialized, so that string elements whose field-lengths were changed by the
search/replace operation, can be stored with their correct length, so the data can be
correctly unserialized.
Returns the modified, re-serialized string if successful, and if unserialize() failed, it returns the
original serialized data string.*/
function fix_serialized_field_o($serfield, $srcURL, $trgURL, $counters){
	$unserialized_data = "";
	$unserialized_data = unserialize($serfield);
	$test_reserialization = serialize($unserialized_data);
	if (substr_count($test_reserialization, '__PHP_Incomplete_Class_Name') > 0) {
		/*unserialize could not successfully  define the class of this object; reserialization would corrupt it.
		. . . and we wouldn't want that.  We can't predict which objects (whether they're WordPress objects, or
		if they belong to a plugin) that we'll need to define at this point, so setting up class includes wouldn't
		be of much help.*/
		unset($unserialized_data);
		print('Serialized Object could not be unserialized():<br />');
	}
	if($unserialized_data){
		$k = 0;
		$counters['num_elems_upd_i'] = count($unserialized_data, 1); //not a count of elements updated
		$arrayified_object = get_object_vars($unserialized_data);
		$updated_unserialized_data = str_ireplace($srcURL, $trgURL, $arrayified_object, $k);
		$counters['num_ser_obj_repl_i'] += $k ;
		$results = serialize($updated_unserialized_data);
	} else {
		$counters['num_unser_fail_o'] += 1 ;
		/*Sometimes, a field just fails to unserialize, often this is because there's previous "corruption":
		a string may have been updated while it was in serialized state, so the length no longer matches.
		in these cases; if your site is working, with this data in the database; then don't worry about it,
		you can probably just delete this row, because a new row was probably already created to replace
		it when WordPress tried to unserialize() it to read the options.
		If the field was an Object - and if unserialize() can't define the class, we'll aslo abort our process.*/
		print('<br />');
		print('unserialize() failed for this field.<br />');
		print('Serialized field was not updated:<br />');
		print('- in table #: ' . $counters['current_table'] . '<br />');
		print('- at column #: ' . $counters['current_column'] . '<br />');
		print('- at row #: ' . $counters['current_rownum'] . '<br />');
		print('- original data: ' . strip_tags(substr($serfield, 0, 90)) . '<br />');
		$results = $serfield;
	}
	return $results;
}
/*Prints a report of tallies for each individual table*/
function intermediate_instrumentation_report($tallies) {
	print('<br />');
	print('Current table #: ' . $tallies['current_table']  . ' finished processing.<br />');
	print('Number of columns in this table: ' . $tallies['num_columns_i'] . '<br />');
	print('Number of rows selected in this table: ' . $tallies['num_rows_i'] . '<br />');
	//fields should equal rows multiplied by columns, by the way :)
	print('Number of fields processed in this table: ' . $tallies['num_fields_i'] . '<br />');
	//number of fields meeting criteria should equal number of flat fields plus number of serialized fields.
	print('Number of fields meeting criteria in this table: ' . $tallies['num_viable_fields_i'] . '<br />');
	print('Number of flat database fields in this table: ' . $tallies['num_flat_fields_i'] . '<br />');
	print('Number of replacements made in flat fields in this table: ' . $tallies['num_flat_repl_i'] . '<br />');		//number of serialized array updates (not number of elements) , this table
	//number of serialized fields should equal number of serialized objects plus serialized arrays
	print('Number of serialized fields in this table: ' . $tallies['num_ser_fields_i'] . '<br />');
	print('Number of serialized objects in this table: ' . $tallies['num_ser_obj_i'] . '<br />');
	print('Number of replacements made in serialized objects in this table: ' . $tallies['num_ser_obj_repl_i'] . '<br />');
	print('Number of serialized arrays in this table: ' . $tallies['num_ser_arr_i'] . '<br />');
	print('Number of replacements made in serialized arrays in this table: ' . $tallies['num_ser_arr_repl_i'] . '<br />');
	//I originally planned to be able to dig into individual elements but the str_replace doesn't provide that information.
	//. . . so number of unserialized elements just totals how many elements were in all of the serialized fields.
	print('Number of unserialized elements processed in this table: ' . $tallies['num_elems_upd_i'] . '<br />');
	// Number of database rows should equal number of fields meeting criteria. . .  unless there were serialized record update failures.
	print('Number of database rows updated in this table: ' . $tallies['num_rows_updated_i'] . '<br />');
	// number of failures would be subtracted from number of database rows, to yeild number of fields meeting criteria
	print('Number of serialized record update failures in this table: ' . $tallies['num_ser_upd_fail_i'] . '<br />');
	print('::::::::::::::::::::::::::::::::::::::::::::::::::::<br />');
	print('<br />');
}
/*Prints a final report of totalled tallies kept during execution*/
function final_instrumentation_report($tallies) {
	print('<br />');
	//should be the number of tables in the $export_tables_list
	print('Total number of tables processed: ' . $tallies['num_tables'] . '<br />');
	print('Total number of columns processed: ' . $tallies['num_columns_o'] . '<br />');
	print('Total number of rows processed: ' . $tallies['num_rows_o'] . '<br />');
	//number of fields should be total rows multiplied by total columns multiplied by number of tables
	print('Total number of fields processed: ' . $tallies['num_fields_o'] . '<br />');
	print('Total number of fields meeting criteria: ' . $tallies['num_viable_fields_o'] . '<br />');
	print('Total number of flat fields fixed: ' . $tallies['num_flat_fields_o'] . '<br />');
	print('Total number of replacements made in flat fields: ' . $tallies['num_flat_repl_o'] . '<br />');
	print('Total number of serialized fields processed: ' . $tallies['num_ser_fields_o'] . '<br />');
	print('Total number of serialized objects processed: ' . $tallies['num_ser_obj_o'] . '<br />');
	print('Total number of replacements made in serialized objects: ' . $tallies['num_ser_obj_repl_o'] . '<br />');
	print('Total number of serialized arrays processed: ' . $tallies['num_ser_arr_o'] . '<br />');
	print('Total number of replacements made in serialized arrays: ' . $tallies['num_ser_arr_repl_o'] . '<br />');
	print('Total number of unserialized elements processed: ' . $tallies['num_elems_upd_o'] . '<br />');
	print('Total number of failed unserializations: ' . $tallies['num_unser_fail_o'] . '<br />');
	print('Total number of database rows updated: ' . $tallies['num_rows_updated_o'] . '<br />');
	print('Total number of serialized record update failures: ' . $tallies['num_ser_upd_fail_o'] . '<br />');
	print('Migration Completed. <br />');
	print('<br />');
}

/*migrate_scratch_table()
accepts 3 parameters; the name of the table, the name of the source database, and the name of the target database.
This function drops (deletes) the live, functioning, WordPress table, on the target system, and creates a new one,
populating it with the modified rows from the corresponding scratch table on the source system.
Returns nothing.*/
function migrate_scratch_table($tablename, $source_database, $target_database){
	global $wpdb;
	print('migrating: scratch_' . $tablename . '<br />from: ' . $source_database . '<br />to: ' . $target_database . '.<br />::::::::::::::::::::::::::::::::::::::::::::::::::::<br />');
	$scratchTableName = 'scratch_' . $tablename ;
	$killTableQry = "DROP TABLE {$target_database}.{$tablename};";
	$mkTableQry = "CREATE TABLE {$target_database}.{$tablename} LIKE {$source_database}.{$scratchTableName};";
	$popTableQry = "INSERT INTO {$target_database}.{$tablename} SELECT * FROM {$source_database}.{$scratchTableName};";
	$wpdb->query($killTableQry);
	$wpdb->query($mkTableQry);
	$wpdb->query($popTableQry);
}

?>

By the way: What this all means is that THIS procedure:
https://codex.www.ads-software.com/Changing_The_Site_URL

… will NOT work for anyone who has plugins who write your site’s URL to a variable length record, and stores it in a field in the database. (unless the old URL is exactly the same length, in characters, as your new URL).

Okay – I think I have figured out the CAUSE of this problem!

Both TDOMF and Register Plus use SERIALIZED fields to store their data in the database.

In other words, they’ve got a data-structure they’re using in memory, they store it in the database as one long string, instead of breaking that structure down into its component values and giving them each their own columns in the database.

I can see why they do this – so that they can send that data across the wire serialized, without having to unserialize/reserialize.

One thing about these serialized data fields is: each sub-component starts with a data-type (a=array, s=string, etc) followed by a colon, followed by a number followed by a colon, followed by the contents of that sub-field. Example – s:8:”password”; . . . get it?

So the problem occurs when I use “sed” on my data stream from:

mysqldump --opt --compress --default-character-set=utf8 --user=<my_db_user> --password=<my_db_password> --host=localhost <my_staging_db_name> | sed 's%https:\/\/<my_staging_server's_url>%https://<my_production_server's_url>%g' | mysql --default-character-set=utf8 --user=<my_db_user> --password=<my_db_password> --host localhost -D <my_production_db_name> -C <my_production_db_name>

I end up converting my URLs, say, “https://myurl.org” to “https://mytestserver.site.com”. If one (or more) instances of my URL swap occurs in a SERIALIZED data field, I end up changing the LENGTH. And when the LENGTH value does not match the actual length, get_option() pukes.

mysqldump, sed, and mysql are all smart enough to deal with any data that comes through there, as long as it doesn’t have to worry about logic embedded in that data. get_option() contains the logic to deal with serializing/deserializing my data.

So what I guess I need to do, is figure out a way to reverse-engineer the serialize/deserialize logic – already done in WP’s php api, and re-implement it in some form of sed-awk in my data stream.

It would also be possible to export to a .sql file using phpMyAdmin or mysqldump.

Then use your handy-dandy unix text-processing tools to go into the wp_posts section (I actually export my tables to separate files) remove the rows that include posts with the earlier dates (you could use sed, or even perl, if you’re a perl ninja). Or, if we’re talking about a small dataset, you could do it by hand.

If you use categories or tags, you’ll need to go into the wp_term_relationships table, and remove any rows with object_id matching your removed wp_posts.id’s. Same with wp_postmeta.post_id. Oh yeah. wp-comments too. Could be a nightmare.

I’ve had no luck with exporting via the xml interface. Import always seems to puke. But I’ve been able to do exports with SQL.

Problem is, you’ve got to change your post_id numbers in wp_posts, and wp_postmeta, and you’ve got to make sure there are no collisions, which also means you have to change the URLs that link to that post.

I’m sure a database whiz could program it. I transferred about 6 posts from one site to another, and it was an extremely tedious manual process.

Looks fine to me.

Okay – so wp_head() is getting called in my start.php file, language settings in my config file are:
define(‘DB_CHARSET’, ‘utf8’);
define(‘DB_COLLATE’, ‘utf8_unicode_ci’);

and phpMyAdmin is set to English.

Do you think it could be my “iconv” settings?

Which header.php file?

There are two, in my theme (modified version of Generic Designer 1.0)

There’s themes/mytheme/header.php, and themes/mytheme/templates/header.php.

neither of those have a <?php wp_head(); ?>.
Nor do the original Generic-Plus theme files.

I see that statement in my theme’s 404.php file. . . ?
it’s also in theme/mytheme/templates/start.php;
theme/mytheme/header.php has a
<?php include(‘templates/start.php’); ?>

So I have to conclude that wp_head() is getting called.

With a little more work, I managed to isolate PART of the problem.

With the assumptions of my first post;

I have a page with a Post and Comments.
This is that page on our production site – I have manually cleaned all the funky characters:
https://edtechfuture.org/?page_id=56

The same page on my development server, I cleaned as well.
When I exported and imported ONLY the wp_comments and wp_posts tables, the funky characters re-appeared – in both the comments, and the post.

Then I manually cleaned the funky characters from that page – again.

Then I deactivated the “UTF8-Sanitize” plugin.

And I did another export/import of just the wp_comments and wp_posts tables.
THIS time, the comments remained “clean” – but the Post picked up the ? characters at the end of sentences.

I cleaned them manually, again.
And I again, performed the export/import (fresh export) from wp_comments and wp_posts – and again, the ? characters appeared in my post. (but the “, ‘, and – characters in my comments all stayed clean).

When I check the wp_posts.sql file, I check the file (after the import) in Notepad++, and it’s viewing the file as “ANSI as UTF-8”. I can’t see the ? characters – they’re displayed as a “space” in this encoding. But when I look in Hex Edit mode, these characters consist of two-bytes (all the other characters are SINGLE-BYTE), and the value is 0xc2a0. The subsequent ACTUAL space is 0x20.

So, I go into phpMyAdmin, and check that post’s record in wp_posts, and the ? character is clearly visible in post_content.

Then I clean up the post in WordPress’s editor – check the record again in phpMyAdmin, the ?’s are gone, as you’d expect. Okey dokey?

This time, I export the table with mysqldump. I re-import it using the same import method I’ve always used (mysql. . . source dumpfile.sql;) . . . and this time, no weird characters are in my post.

By process of elimination – this appears to be a bug in phpMyAdmin. . . ?
(MySQL version 5.0.22, phpMyAdmin version 2.10.1, php version 5.2.2)

When I look at my environment info via GD Press Tools, under the “PHP Info” tab, in the ICONV section, all the ICONV settings are set to ISO-8859-1. . . . is that a problem?