Wordfence database issue crashes server

  • Resolved squasher

    (@squasher)


    2 years, 9 months ago

    Hi,

    Today my complete server went down due to a Wordfence database issue. I noticed similar posts about this today. Is this a known issue?

    It seems slow queries can pull down the complete server.

    This is what my hosting provider found out and which lead to a complete crash:

    === TOP 10 of 90 (total) Slow Queries for the past 4 hours =======================================================
 1. Executed 24m 5s ago for 427.521232 sec on Database --> xx 
Date: 2022-02-13 11:27:04 Query_time: 427.521232 Rows_examined: 0: Rows_sent 0 Lock_time: 5.101989 Query_chars: 672
SELECT *, CASE WHEN <code>type</code> = 3 THEN 0 WHEN <code>type</code> = 4 THEN 1 WHEN <code>type</code> = 7 THEN 2 WHEN <code>type</code> = 6 THEN 3 WHEN <code>type</code> = 5 THEN 4 WHEN <code>type</code> = 9 THEN 5 WHEN <code>type</code> = 8 THEN 6 WHEN <code>type</code> = 2 THEN 7 WHEN <code>type</code> = 1 THEN 8 ELSE 9999 END AS <code>typeSort</code>,
 ...... 
 <code>IP</code> WHEN <code>type</code> = 7 THEN <code>IP</code> WHEN <code>type</code> = 2 THEN <code>IP</code> WHEN <code>type</code> = 8 THEN <code>IP</code> ELSE 9999 END AS <code>detailSort</code> FROM <code>hggy_wfBlocks7</code> WHERE <code>type</code> IN (3) AND (<code>expiration</code> = 0 OR <code>expiration</code> > UNIX_TIMESTAMP()) ORDER BY <code>typeSort</code> ASC, <code>id</code> DESC; 
-------------------------------------------------------------------------------------------------------------------
 2. Executed 24m 5s ago for 427.520877 sec on Database --> xx
Date: 2022-02-13 11:27:04 Query_time: 427.520877 Rows_examined: 0: Rows_sent 0 Lock_time: 5.105281 Query_chars: 672
SELECT *, CASE WHEN <code>type</code> = 3 THEN 0 WHEN <code>type</code> = 4 THEN 1 WHEN <code>type</code> = 7 THEN 2 WHEN <code>type</code> = 6 THEN 3 WHEN <code>type</code> = 5 THEN 4 WHEN <code>type</code> = 9 THEN 5 WHEN <code>type</code> = 8 THEN 6 WHEN <code>type</code> = 2 THEN 7 WHEN <code>type</code> = 1 THEN 8 ELSE 9999 END AS <code>typeSort</code>,
 ...... 
 <code>IP</code> WHEN <code>type</code> = 7 THEN <code>IP</code> WHEN <code>type</code> = 2 THEN <code>IP</code> WHEN <code>type</code> = 8 THEN <code>IP</code> ELSE 9999 END AS <code>detailSort</code> FROM <code>hggy_wfBlocks7</code> WHERE <code>type</code> IN (3) AND (<code>expiration</code> = 0 OR <code>expiration</code> > UNIX_TIMESTAMP()) ORDER BY <code>typeSort</code> ASC, <code>id</code> DESC; 
-------------------------------------------------------------------------------------------------------------------
 3. Executed 24m 5s ago for 427.520773 sec on Database --> xx
Date: 2022-02-13 11:27:04 Query_time: 427.520773 Rows_examined: 0: Rows_sent 0 Lock_time: 5.101828 Query_chars: 672
SELECT *, CASE WHEN <code>type</code> = 3 THEN 0 WHEN <code>type</code> = 4 THEN 1 WHEN <code>type</code> = 7 THEN 2 WHEN <code>type</code> = 6 THEN 3 WHEN <code>type</code> = 5 THEN 4 WHEN <code>type</code> = 9 THEN 5 WHEN <code>type</code> = 8 THEN 6 WHEN <code>type</code> = 2 THEN 7 WHEN <code>type</code> = 1 THEN 8 ELSE 9999 END AS <code>typeSort</code>,
 ...... 
 <code>IP</code> WHEN <code>type</code> = 7 THEN <code>IP</code> WHEN <code>type</code> = 2 THEN <code>IP</code> WHEN <code>type</code> = 8 THEN <code>IP</code> ELSE 9999 END AS <code>detailSort</code> FROM <code>hggy_wfBlocks7</code> WHERE <code>type</code> IN (3) AND (<code>expiration</code> = 0 OR <code>expiration</code> > UNIX_TIMESTAMP()) ORDER BY <code>typeSort</code> ASC, <code>id</code> DESC; 
-------------------------------------------------------------------------------------------------------------------
 4. Executed 24m 5s ago for 427.520753 sec on Database --> xx
Date: 2022-02-13 11:27:04 Query_time: 427.520753 Rows_examined: 0: Rows_sent 0 Lock_time: 5.101839 Query_chars: 672
SELECT *, CASE WHEN <code>type</code> = 3 THEN 0 WHEN <code>type</code> = 4 THEN 1 WHEN <code>type</code> = 7 THEN 2 WHEN <code>type</code> = 6 THEN 3 WHEN <code>type</code> = 5 THEN 4 WHEN <code>type</code> = 9 THEN 5 WHEN <code>type</code> = 8 THEN 6 WHEN <code>type</code> = 2 THEN 7 WHEN <code>type</code> = 1 THEN 8 ELSE 9999 END AS <code>typeSort</code>,
 ...... 
 <code>IP</code> WHEN <code>type</code> = 7 THEN <code>IP</code> WHEN <code>type</code> = 2 THEN <code>IP</code> WHEN <code>type</code> = 8 THEN <code>IP</code> ELSE 9999 END AS <code>detailSort</code> FROM <code>hggy_wfBlocks7</code> WHERE <code>type</code> IN (3) AND (<code>expiration</code> = 0 OR <code>expiration</code> > UNIX_TIMESTAMP()) ORDER BY <code>typeSort</code> ASC, <code>id</code> DESC; 
-------------------------------------------------------------------------------------------------------------------
 5. Executed 24m 5s ago for 427.520729 sec on Database --> xx
Date: 2022-02-13 11:27:04 Query_time: 427.520729 Rows_examined: 0: Rows_sent 0 Lock_time: 5.101896 Query_chars: 672
SELECT *, CASE WHEN <code>type</code> = 3 THEN 0 WHEN <code>type</code> = 4 THEN 1 WHEN <code>type</code> = 7 THEN 2 WHEN <code>type</code> = 6 THEN 3 WHEN <code>type</code> = 5 THEN 4 WHEN <code>type</code> = 9 THEN 5 WHEN <code>type</code> = 8 THEN 6 WHEN <code>type</code> = 2 THEN 7 WHEN <code>type</code> = 1 THEN 8 ELSE 9999 END AS <code>typeSort</code>,
 ...... 
 <code>IP</code> WHEN <code>type</code> = 7 THEN <code>IP</code> WHEN <code>type</code> = 2 THEN <code>IP</code> WHEN <code>type</code> = 8 THEN <code>IP</code> ELSE 9999 END AS <code>detailSort</code> FROM <code>hggy_wfBlocks7</code> WHERE <code>type</code> IN (3) AND (<code>expiration</code> = 0 OR <code>expiration</code> > UNIX_TIMESTAMP()) ORDER BY <code>typeSort</code> ASC, <code>id</code> DESC; 
-------------------------------------------------------------------------------------------------------------------
 6. Executed 24m 5s ago for 427.520719 sec on Database --> xx
Date: 2022-02-13 11:27:04 Query_time: 427.520719 Rows_examined: 0: Rows_sent 0 Lock_time: 5.101793 Query_chars: 672
SELECT *, CASE WHEN <code>type</code> = 3 THEN 0 WHEN <code>type</code> = 4 THEN 1 WHEN <code>type</code> = 7 THEN 2 WHEN <code>type</code> = 6 THEN 3 WHEN <code>type</code> = 5 THEN 4 WHEN <code>type</code> = 9 THEN 5 WHEN <code>type</code> = 8 THEN 6 WHEN <code>type</code> = 2 THEN 7 WHEN <code>type</code> = 1 THEN 8 ELSE 9999 END AS <code>typeSort</code>,
 ...... 
 <code>IP</code> WHEN <code>type</code> = 7 THEN <code>IP</code> WHEN <code>type</code> = 2 THEN <code>IP</code> WHEN <code>type</code> = 8 THEN <code>IP</code> ELSE 9999 END AS <code>detailSort</code> FROM <code>hggy_wfBlocks7</code> WHERE <code>type</code> IN (3) AND (<code>expiration</code> = 0 OR <code>expiration</code> > UNIX_TIMESTAMP()) ORDER BY <code>typeSort</code> ASC, <code>id</code> DESC; 
-------------------------------------------------------------------------------------------------------------------
 7. Executed 24m 5s ago for 427.520707 sec on Database --> xx
Date: 2022-02-13 11:27:04 Query_time: 427.520707 Rows_examined: 1: Rows_sent 1 Lock_time: 3.900691 Query_chars: 85
SELECT option_value FROM hggy_options WHERE option_name = 'disallowed_keys' LIMIT 1; 
-------------------------------------------------------------------------------------------------------------------
 8. Executed 24m 5s ago for 427.520699 sec on Database --> xx
Date: 2022-02-13 11:27:04 Query_time: 427.520699 Rows_examined: 0: Rows_sent 0 Lock_time: 5.101878 Query_chars: 672
SELECT *, CASE WHEN <code>type</code> = 3 THEN 0 WHEN <code>type</code> = 4 THEN 1 WHEN <code>type</code> = 7 THEN 2 WHEN <code>type</code> = 6 THEN 3 WHEN <code>type</code> = 5 THEN 4 WHEN <code>type</code> = 9 THEN 5 WHEN <code>type</code> = 8 THEN 6 WHEN <code>type</code> = 2 THEN 7 WHEN <code>type</code> = 1 THEN 8 ELSE 9999 END AS <code>typeSort</code>,
 ...... 
 <code>IP</code> WHEN <code>type</code> = 7 THEN <code>IP</code> WHEN <code>type</code> = 2 THEN <code>IP</code> WHEN <code>type</code> = 8 THEN <code>IP</code> ELSE 9999 END AS <code>detailSort</code> FROM <code>hggy_wfBlocks7</code> WHERE <code>type</code> IN (3) AND (<code>expiration</code> = 0 OR <code>expiration</code> > UNIX_TIMESTAMP()) ORDER BY <code>typeSort</code> ASC, <code>id</code> DESC; 
-------------------------------------------------------------------------------------------------------------------
 9. Executed 24m 5s ago for 427.520695 sec on Database --> xx
Date: 2022-02-13 11:27:04 Query_time: 427.520695 Rows_examined: 0: Rows_sent 0 Lock_time: 5.101813 Query_chars: 672
SELECT *, CASE WHEN <code>type</code> = 3 THEN 0 WHEN <code>type</code> = 4 THEN 1 WHEN <code>type</code> = 7 THEN 2 WHEN <code>type</code> = 6 THEN 3 WHEN <code>type</code> = 5 THEN 4 WHEN <code>type</code> = 9 THEN 5 WHEN <code>type</code> = 8 THEN 6 WHEN <code>type</code> = 2 THEN 7 WHEN <code>type</code> = 1 THEN 8 ELSE 9999 END AS <code>typeSort</code>,
 ...... 
 <code>IP</code> WHEN <code>type</code> = 7 THEN <code>IP</code> WHEN <code>type</code> = 2 THEN <code>IP</code> WHEN <code>type</code> = 8 THEN <code>IP</code> ELSE 9999 END AS <code>detailSort</code> FROM <code>hggy_wfBlocks7</code> WHERE <code>type</code> IN (3) AND (<code>expiration</code> = 0 OR <code>expiration</code> > UNIX_TIMESTAMP()) ORDER BY <code>typeSort</code> ASC, <code>id</code> DESC; 
-------------------------------------------------------------------------------------------------------------------
 10. Executed 24m 5s ago for 427.520686 sec on Database --> xx
Date: 2022-02-13 11:27:04 Query_time: 427.520686 Rows_examined: 0: Rows_sent 0 Lock_time: 5.101959 Query_chars: 672
SELECT *, CASE WHEN <code>type</code> = 3 THEN 0 WHEN <code>type</code> = 4 THEN 1 WHEN <code>type</code> = 7 THEN 2 WHEN <code>type</code> = 6 THEN 3 WHEN <code>type</code> = 5 THEN 4 WHEN <code>type</code> = 9 THEN 5 WHEN <code>type</code> = 8 THEN 6 WHEN <code>type</code> = 2 THEN 7 WHEN <code>type</code> = 1 THEN 8 ELSE 9999 END AS <code>typeSort</code>,
 ...... 
 <code>IP</code> WHEN <code>type</code> = 7 THEN <code>IP</code> WHEN <code>type</code> = 2 THEN <code>IP</code> WHEN <code>type</code> = 8 THEN <code>IP</code> ELSE 9999 END AS <code>detailSort</code> FROM <code>hggy_wfBlocks7</code> WHERE <code>type</code> IN (3) AND (<code>expiration</code> = 0 OR <code>expiration</code> > UNIX_TIMESTAMP()) ORDER BY <code>typeSort</code> ASC, <code>id</code> DESC; 
-------------------------------------------------------------------------------------------------------------------

    Can you advise how to prevent this?

    Thanks!

Viewing 3 replies - 1 through 3 (of 3 total)
  • Plugin Support wfpeter

    (@wfpeter)

    Hi @squasher, thanks for getting in touch with your detailed log information.

    We see posts about this exact issue from time-to-time and have been sent log information from our customers where the exact same error is also reporting for other WordPress plugins.

    For this reason, we believe that the issue is related to the database itself where MySQL/MariaDB is stopping, reconnecting and leaving the connection in a state that somehow causes this query error to appear.

    We usually request that you or your host check the database server logs and other logs for the website(s) on the server such as syslog etc. to see what else is going on with the server at the timestamp of these issues. I would expect the database to be unavailable temporarily, either because it is restarting or has crashed.

    This customer was having intermittent load issues on their server:
    https://www.ads-software.com/support/topic/commands-out-of-sync-can-we-get-a-resolution-please/

    Although investigation isn’t conclusive here, Elegant Themes looks to be experiencing similar “Commands out of sync” issues on a server also flagging these problems with wfblocks7: https://www.ads-software.com/support/topic/database-error-wfblocks7/

    To assist us see whether there’s a consistent element between your case and others, can you send a diagnostic report to wftest @ wordfence . com? You can find the link to do so at the top of the Wordfence > Tools > Diagnostics page. Then click on “Send Report by Email”. Please add your forum username where indicated and respond here after you have sent it.

    NOTE: It should look as follows – Screenshot of Tools > Diagnostic > Send by Email

    Thanks,

    Peter.

    Thread Starter squasher

    (@squasher)

    Hi Peter,

    Thanks for your reply.

    I have send the diagnostic report. I hope it gives some insights.

    Best,
    Jerome

    Plugin Support wfpeter

    (@wfpeter)

    Hi @squasher, thank-you for your diagnostic.

    There are no clear problems with connectivity being reported back so your host would still be the main point of contact for why your database might be stopping and causing these errors. I haven’t received diagnostics from all customers experiencing this issue but it’ll certainly help make a comparison if/when we get more to compare it with.

    By all means let us know if you make any significant progress identifying a problem with server resources, load-balancers, firewalls or anything else that results in resolving the problem as it may help others in future.

    Thanks,

    Peter.

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘Wordfence database issue crashes server’ is closed to new replies.

Tags