cronjob fail when publishing 500 news per day

  • Hello,
    I need some help to understand how Postie works. I will start with background on what our problem is.

    We publish more than 500 news some days. Sometimes in intervals of 2-3 minutes. We have setup the service setcronjob.com to hit the Postie URL: https://www.domain.com/?postie=get-mail every 10 minutes. Sometimes there are 5-10-15 news for publishing. Postie gets the articles from our email. It usually works when the news load is low but when we have many news, Setcronjob starts to failure at executing the postie=get-mail giving erros: “Service Unavailable”.

    We know that our website works as we check from other computers. It works very well, quick loading …

    So once this happen, we try to hit the /?postie=get-mail manually, and after we hit, it start loading very slow … and then we also get error and we are blocked from accessing the site for a minute from the same browser.

    The settings in Postie was to publish 25 emails per hit. Maximum number of emails to process. We tried with 10 also.
    The cronjob was set at 10 minutes. We tried with 3 minutes also.

    What we don’t understand is, the connection between the cronjob hit of the URL of postie plugin and the publishing process. What settings are the optimal for publishing huge number of posts. Do we have to decrease the Cronjob time to 1 minute or it must be higher like 15 minutes? Do we have to let Postie publish max 1-2-3 emails and the cronjob to hit in like 30 sec, or do we have to make it publish 20-30-40 articles and make the cronjob hit at 5-10-15 minutes. What’s the optimal one? We plan to publish more than 2000 news per day and in some periods more than 40 per 10 minutes. What we have to do to make postie work with that load?

    We believe that Postie /?postie=get-mail somehow crashes or can’t load , and freezes itself. Another probably important detail is that when the Cronjob can’t hit the url , it usually shows Time: 200-300 seconds. When there are no emails its 1-2-3 seconds.

    I really hope you can help us understand what we have to do to make publishing of many articles possible with postie. We use WPEngine and our hosting is really good. But we wonder, is it some blocking? or what happens so the cronjob and we manually can’t get normal message from /?postie=get-mail sometimes … after waiting 1-2 minutes. What we saw though is even though the message is not returned after we hit /?postie=get-mail and we get service unavailable or the cronjob, the posts are deleted from email and some are published (not sure if all). We have to check more.

    Thank you in advance!

    https://www.ads-software.com/plugins/postie/

Viewing 6 replies - 1 through 6 (of 6 total)
  • Thread Starter occ821

    (@occ821)

    One more thing. Could the problem be because, Postie has not finished publishing and giving message and at the same time, the cronjob service hit again? Te cronjob failed 14 times last night and was automatically disable because of that … We wonder can Postie handle publishing of 20-30 news at once? According to check with our host, the website was working very fast and all was fine with speed.

    Plugin Author Wayne Allen

    (@wayneallen-1)

    Generally speaking what you are trying to do should be entirely doable. I know of one user who had 15,000 emails stacked up and although it took awhile it was able to process them all using a setup like you describe.

    I think the first step is to do the following:
    In your wp-config.php file look for:

    define(‘WP_DEBUG’, false);

    Replace it with:

    define('WP_DEBUG', true);
define('WP_DEBUG_LOG', true);
define('POSTIE_DEBUG', true);

    Then access https://www.domain.com/?postie=get-mail through a browser or get the info from setcronjob.com and send the resulting output to [email protected] and I’ll see if there is anything obvious.

    Thread Starter occ821

    (@occ821)

    I find the reason is connected with high load and traffic. When the USA visitors wake up, the issue starts somewhere around then. The cron starts to more regularly failure and later it fails all the time. When the traffic decrease after the US session, it starts to work successfully again.

    I will do what you do me, but I think we tried putting this in wp-config.php and breaked the website. Any workaround?

    Thread Starter occ821

    (@occ821)

    I managed to add these without breaking the website. The only problem is that when these are there, our Jetpack custom CSS doesn’t work for some reason … and this breaks some of our design style. We will add the debug as soon as the cron starts failing and give you the error log by email.

    Plugin Author Wayne Allen

    (@wayneallen-1)

    You may be able to just have this line

    define('POSTIE_DEBUG', true);

    and reset this one

    define('WP_DEBUG', false);

    to keep your site working.

    Thread Starter occ821

    (@occ821)

    Email sent.

Viewing 6 replies - 1 through 6 (of 6 total)
  • The topic ‘cronjob fail when publishing 500 news per day’ is closed to new replies.