Too much data?

  • Resolved ndp

    (@ndp)


    15 years, 1 month ago

    To put this one very briefly;

    I was asked to have a page on my site that generates a list of all the items submitted by users.

    I do a database query, and I render the results into a styled table.

    This is accomplished with some php code embedded in the page using the ExecPHP plugin.

    This worked fine for datasets up to about 100 items/rows.

    But after about 110 or so, the code began to just hang. The page would render, but the contents of the eval() would return nothing, so I’d have a blank page.

    By shortening the contents of the Description string from a limit of 1000 characters to 250, I was able to get it to work again, but further testing revealed that when I add enough items, this code is going to die again.

    It is as if ExecPHP is going to allow me to add so many characters to a page, and if I exceed that limit, it decides to give up.

    My query returns all the data fine, and the data in the variables is fine. The problem happens when I try to echo or print the formatted contents to the screen.

    By putting this code in a template file, outside of the page contents, I was able to circumvent this limitation (I think) – but then the wp-postratings info isn’t available from that context, so I get the tag markup and post-id instead.

    This site relies heavily on such “list-views” so I guess I’m fortunate that we didn’t end up scaling to a whole lot of feedback. But it would be nice if I could find out the cause of this seemingly weird limitation. 1000 characters times 100 items = 100,000, that’s not a lot. I can write posts with ten times that amount of data that publish just fine.

Viewing 5 replies - 1 through 5 (of 5 total)

  • is there a site to look at.

    php memory in WordPress is usually set to 32MB. It could be, BUT usually you would get some type of error about it saying memory exceeded.

    Be better to see the code you’re using, can’t think why you’d want to eval the code…

    Thought about paging the results?, so you can grab a set amount per page, limiting how intensive the query will be..

    Again, i’d like to see the code you use for the query.. ?? please..

    Thread Starter ndp

    (@ndp)

    Sorry I forgot to include the link to the public-facing site – this is the page which I believe will have a problem at a certain point:
    https://edtechfuture.org/?page_id=9787

    Here’s a version of the same page, where I have raised the limit on the number of characters displayed in the “Brief Description” field. (All I’m doing here, is changing substr($post_excerpt, 0, 250) to substr($post_excerpt, 0, 500)). This version crashes.
    https://edtechfuture.org/?page_id=9971

    Whether I hit this limit seems to depend on two factors:
    The number of items (rows) displayed, and the amount of data per row.
    Right now, since I only have a few rows, I am limiting the amount of data per row by abridging the description.

    To simplify this, I wrote some test code that can reproduce the error:

    <div id="reslist">
<table id="reslist" cellspacing="0" border="0" width="100%" summary="A table to see what Exec PHP's apparent rendering limit is">
<tr>
<th scope="col" abbr="Column 1">Column 1</th>
<th scope="col" abbr="Column 2">Column 2</th>
<th scope="col" abbr="Column 3">Column 3</th>
<th scope="col" abbr="Column 4">Column 4</th>
</tr>

<?php
for ( $i = 0; $i < 100 ; $i++) {
		$x = "abcdefghij";
		$y = $x;
		for( $j = 0 ; $j < 10 ; $j++ ) {
			$y .= $y . " " ;
		}

		?>
		<tr>
			<td><?php print($x); ?></td>
			<td><?php print($x); ?></td>
			<td><?php print($y); ?></td>
			<td><?php print($x); ?></td>
		</tr>

<?php } ?>
<tr><td><?php print($i); ?> items submitted.</td>
</tr></table>
</div>

    So when my $i loop is set to a max of 987 (much higher than I thought), and $j to 10, I can reproduce the crash – so this problem has absolutely nothing at all to do with the sql query.

    (and I could simply omit the third-column print statement and go to 10,000 rows, and it would chug along just fine; it’s not allocation of memory for this data, it’s the amount that is getting rendered).

    When I set $i’s limit to 100, $j craps out at about 13 (because the size of that $y field increments geometrically).

    So here’s the kicker. If I render something up near the limit – view source, copy the html, then paste that into the editor of a new page, I have just static html in this page now, no php – then I do an Update Page, and when the screen refreshes, that window is blank. WordPress no-likey. I’m trying to identify where, exactly that limit is.

    My wp-config.php has the memory set to 96MB. Changing this to 128 or 256 did not appear to make any difference.

    Thread Starter ndp

    (@ndp)

    I think I might be hitting this:
    https://www.ads-software.com/support/topic/256910?replies=28

    Probably why my test program displays so much more data, and why I can get away with more if I run my code straight in a template: the shortcodes make this problem worse somehow.

    Thread Starter ndp

    (@ndp)

    Yep. That was it.

    1. Open PHP.INI in a text editor of your choice (normally you can find php.ini in your php install dir)
    2. Change the recursion limit to 200x normal, that is, set: pcre.recursion_limit=20000000
    3. Change the backtrack limit to 100x normal, that is, set: pcre.backtrack_limit=10000000
    4. Stop and start the Apache (or IIS) service

    Lucky me – I have access to my php.ini!

Viewing 5 replies - 1 through 5 (of 5 total)
  • The topic ‘Too much data?’ is closed to new replies.