Automatic Periodic Table Import

Hi,

thanks for your post, and sorry for the trouble.

Unfortunately, I have to disappoint you here. To be honest, I don’t think that the Extension will work for that many tables, or at least that the chances are very low. Basically, your are running into problems with the form submission handling of PHP and your server configuration.
My guess is that some variables in the configuration prevent that the code of the Extension actually gets all the information – which is why it looks like it “forgets” it. This could be PHP config variables like “MAX_INPUT_VARS” and similar and related. You could maybe try increasing these in files like php.ini on the server (maybe with the help of your webhost company).

Regards,
Tobias

I’m seeing a problem that looks similar. what I’m noticing is that it intermittently succeeds and fails. The outbound HTTP network call sometimes times out.

I added some debug logging to the auto update plugin file, and when it fails, it’s fails when calling wp_remote_fopen() to get the file. (I’m using a URL as my source which is a google doc) I copied the source code for wp_remote_fopen() from WP core into the plugin file (as mywp_remote_fopen()) and added logging. It’s actually the wp_safe_remote_get() which times out after 10001 ms (10 sec).

From my logs.. the FAIL case:

[13-Nov-2019 20:32:11 UTC] response from wp_safe_remote_get
[13-Nov-2019 20:32:11 UTC] WP_Error Object
(
    [errors] => Array
        (
            [http_request_failed] => Array
                (
                    [0] => cURL error 28: Operation timed out after 10001 milliseconds with 0 bytes received
                )

        )

    [error_data] => Array
        (
        )

)

The SUCCESS case:

[13-Nov-2019 20:47:03 UTC] Array
(
    [headers] => Requests_Utility_CaseInsensitiveDictionary Object
        (
            [data:protected] => Array
                (
                    [content-type] => text/csv
                    [x-robots-tag] => noindex, nofollow, nosnippet
                    [cache-control] => no-cache, no-store, max-age=0, must-revalidate
                    [pragma] => no-cache
                    [expires] => Mon, 01 Jan 1990 00:00:00 GMT
                    [date] => Wed, 13 Nov 2019 20:47:03 GMT
....snip....

This may be something people are commonly running into. It’s also a fairly common problem in WP. Googling shows lots of it. Now I’m looking into how to solve that

https://www.google.com/search?q=10001+milliseconds&rlz=1C1CHBF_enUS708US708&oq=10001+milliseconds&aqs=chrome..69i57.3847j0j4&sourceid=chrome&ie=UTF-8

Oh and also, the 10 second limit is hardcoded into the WP code. argh
https://developer.www.ads-software.com/reference/functions/wp_remote_fopen/#source

function wp_remote_fopen( $uri ) {
    $parsed_url = @parse_url( $uri );
 
    if ( ! $parsed_url || ! is_array( $parsed_url ) ) {
        return false;
    }
 
    $options            = array();
    $options['timeout'] = 10;
 
    $response = wp_safe_remote_get( $uri, $options );
 
    if ( is_wp_error( $response ) ) {
        return false;
    }
 
    return wp_remote_retrieve_body( $response );
}

Hi,

thanks for your deeper analysis here! Indeed, the timeout likely means that the Google server is slow to respond to your server. Increasing the timeout could be a possibility. And even though that value of 10 is hard-coded in that function, it should be possible to override it via the http_request_timeout filter hook.

TablePress itself (not the Extension) uses download_url, which sets a timeout of 300s, but has an intermediate step (writing the downloaded file to a temporary file on the server first).

Regards,
Tobias

I’ll try adding the filter to my theme’s functions.php and see if that fixes it as well and let you know.

Hi,

thanks, that would be great! If so, I’ll probably just add a filter hook to the Extension itself, to activate this for everybody…

Regards,
Tobias

Yes, that’s a good idea. However, the hook you mentioned doesn’t fix the problem.
I think that’s because the wp_remote_fopen() actually sets the timeout option explicitly. (see above).

Alternatively, using http_request_args hook does work. That explicitly overrides the option that wp_remote_fopen() sets.

//Set HTTP Request Timeout
  add_filter('http_request_args', 'my_http_request_args', 100, 1);
  function my_http_request_args( $r ) {
    $r['timeout'] = 30;
    return $r;
  }

I have the http_request_args on in place on my server. I’ll let it run overnight and check that there are no failures and I’ll report back.

An additional thing to consider is modifying the plugin code to call wp_safe_remote_get() and wp_remote_retrieve_body() directly rather than calling wp_remote_fopen().

In addition to setting the timeout, it also would let the plugin see the WP_Error object if an error occurs. The wp_remote_fopen() swallows it and just returns false. This leaves the plugin unable to tell what really happened.

If you made the calls direct, you could add the message from the WP_Error object to your “Failed” string so the user can see what type of error occurred. That would be very helpful in debugging these kinds of problems. Especially, given that WP_Cron can be so finicky and hard to debug.

Once you implement a solution, I’d also suggest adding it to the FAQ. Lol, I spent a lot of time thinking this was a “real Cron” problem.

Hi,

yep, you are right. http_request_timeout only filters the default value, so http_request_args is better here.

And yes, replacing wp_remote_fopen() sounds reasonable here. I’ll add it to my list ??

Best wishes,
Tobias

The autoimport has been running all night, interval every 15min, with no failures. Using http_request_args hook.
If you have a repo and want me to make a PR, I can do that.
Otherwise, post a comment or ping me if you’d like the new version tested.

Cheers
Josh

Hi Josh,

thanks for testing this and the update!

Great to hear that this helped! I don’t have a public repo with the Extension right now, but maybe you can email me your version or a patch? (The email address is in the main plugin file “tablepress.php”.) Thanks!

Regards,
Tobias

I created a patch and readme here:
https://gist.github.com/joshco/089944027b2c04090978524fdc6b77c5

This patch adds:

* a function to wrap the WP calls to get a URL based source and sets a 30sec timeout.
* More detailed status information in the plugin’s WP Dashboard pane on why an autoupdate failed.
* Logging information, and a logging function, which outputs debug information when WP_DEBUG is true to help troubleshoot.

Hi,

that’s just awesome! Thanks a lot for this, I really appreciate it! I’ll incorporate that into the next version of the Extension!

Best wishes,
Tobias

You’re welcome. This is a pretty cool plugin. On a previous project I wrote client side JS to go direct to google. But then under high traffic, that got painfully slow. So then I built a simple cacher in PHP to proxy the google request. That got painful. Your plugin can be a better solution.

Also, for the FAQ, turns out you can run the auto-update on demand with the WP CLI:
$ wp eval 'do_action("tablepress_table_auto_import_hook");'

So people can set WP_DEBUG and WP_DEBUG_LOG to ‘true’ and then run it from the command line to troubleshoot/verify.
I suppose it could also be put in a real cron job, if the hosting service supports real cron. (mine doesn’t)

Hi,

thanks, that’s also a nice idea! I’ve previously been recommending plugins like Core Control or WP-Crontrol to trigger the WP Cron jobs.
But even that required using a log file to debug things, as no output was shown with that method. So using WP-CLI is indeed a smart trick here!

Regarding real cron jobs: Yes, those are always a good idea here, especially in high traffic environments – but of course the hosting service has to offer it.

Best wishes,
Tobias