WordFence and ASG & EFS issues, had to remove

  • Resolved SoN9ne

    (@son9ne)


    4 years, 8 months ago

    Hello,

    Great plugin, I have used it on numerous sites with no issue until recently. I have a high traffic site using AWS ASG with EFS. I noticed random 504’s for static images and random white pages and daily server crashes. It took me about 3 days to identify that WordFence was the issue. The main issue was due to the wflogs directory that is on EFS so that it could be shared across the cluster. If I ran an ls -la on the directory, it would take about 10 mins and still not show any output other than files don’t exist. I can see every iteration of the config.php.cfww2 (not actual code as I have resolved the issue and do not remember the exact output) and ls is just saying the file does not exist. I was never able to get a directory listing to complete… mostly due to my impatience as this was crashing my PROD siteand after 10 mins I was more focused on uptime.

    EFS has a 100mb throughput and I have no issue with any other shares resource. I share cache, languages, uploads, w3tc-config, and wflogs. wflogs is the only one that seems to get corrupt. I am assuming this has to do with how often the files are be written to. With a high traffic site (over 1 million users) this was literally crashing a cluster of 30 servers. As soon as I disabled and removed WordFence, the entire site went back to normal and it’s been fine for about a week now with no issues. Before it was multiple times a day and I had to terminate instances and allow the ASG to spin up new ones… only to have them go down in a couple hours.

    I wish I had more data to give you but as this was PROD, I was focused on keeping uptime for the service. I am posting this here as a way to let you know about my experience using this on a larger project.

    The issue appeared to be with wflogs being corrupted. I first deleted the directory and rebuilt it and reconfigured WordFence but that lasted about a day before it was corrupted again. The issue could be due to the 100mb throughput and the frequency that the files are being modified (read that these files are modified often somewhere that eludes me right now). Other than that, this was my first time experiencing this issue. Is there a setting to save this to a DB? I assume that defeats the purpose of what you are doing but I am no expert in what WordFence is doing.

    I just wanted to leave this for you as a means to investigate and see if there is anything that can be done. To solve my issue, I just removed the plugin. Ideally, I would like to use this plugin but unfortunately this is not possible at this time. Thanks for your time.

    WordFence: 7.4.6
    WordPress: 3.5.2
    PHP-FPM: 7.2.26
    Apache: 2.4.41

Viewing 6 replies - 1 through 6 (of 6 total)
  • Plugin Support wfphil

    (@wfphil)

    Hi @son9ne,

    You can try switching to the MySQLi firewall data storage engine:

    https://www.wordfence.com/help/firewall/mysqli-storage-engine/

    Thread Starter SoN9ne

    (@son9ne)

    Thanks for the suggestion.

    I tried this for less than 12 hours and it is still crashing the server. Same issue as I posted above. The wflogs directory is not listable again.

    I immediately disabled WordFence once this started. This time, my site did not recover right away.

    With WordFence disabled, the log was filling with:

    There are numerous occurrences of:

    
[29-Mar-2020 20:03:57 UTC] PHP Warning:  file_get_contents(): open_basedir restriction in effect. File(/var/www/public/wp-content/wflogs//../../wp-config.php) is not within the allowed path(s): (/var/www/public/:/var/www/wordpress/:/usr/share/pear:/usr/share/php:/tmp/:/mnt/efs/) in /var/www/public/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/lib/utils.php on line 1081
[29-Mar-2020 20:03:57 UTC] PHP Warning:  file_get_contents(/var/www/public/wp-content/wflogs//../../wp-config.php): failed to open stream: Operation not permitted in /var/www/public/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/lib/utils.php on line 1081

    This is interesting because /var/www/public/wp-content/wflogs//../../wp-config.php is looking into /var/www/public which is allowed. This is only an issue with WordFence so I’m going to have to look for alternatives at this point. System worked great until I enabled it again. I wonder if the extra / would cause any issue but I doubt it.

    The interesting part is that these logs only happen after I disabled WordFence. For this to stop, I actually need to delete WordFence from the system.

    Unfortunately, it will be some time before I can look deeper into the issue.

    Plugin Support wfphil

    (@wfphil)

    Hi @son9ne

    Thank you for the update.

    To be able to switch to the MySQLi firewall data storage engine can you remove your open_basedir restrictions first and see if that fixes it.

    Thread Starter SoN9ne

    (@son9ne)

    That will not be possible. I will have to test this in a non-PROD environment but I have run into another issue with WordFence. Until I resolve that issue, this is still on the back-burner.

    I am currently in the process of purchasing a premium plan so I can escalate support for these issues.

    • This reply was modified 4 years, 7 months ago by SoN9ne.
    Plugin Support wfphil

    (@wfphil)

    Hi @son9ne

    Thank you for the update. As you are now seeking premium support then I will close this topic.

    Thread Starter SoN9ne

    (@son9ne)

    Further investigation shows this is due to an issue with using symlinks for the wflogs directory…

    The error log entries point out the issue:

    
[15-Apr-2020 17:50:24 UTC] PHP Warning:  file_get_contents(): open_basedir restriction in effect. File(/var/www/public/wp-content/wflogs//../../wp-config.php) is not within the allowed path(s): (/var/www/public/:/var/www/wordpress/:/usr/share/pear/:/usr/share/php/:/usr/share/httpd/:/tmp/:/dev/null:/dev/random:/dev/arandom:/dev/urandom:/mnt/efs/) in /var/www/public/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/lib/utils.php on line 1081

    While this would appear to be /var/www/public/ it is, in fact, /mnt as the wflogs directory is located in /mnt/efs/wflogs. Since using a symbolic link, this would be expected behavior.

    The correct approach would be to use bind mounts so it actually acts as if it were in the directory. This is an issue on my end and as such, was simple to address and all is working well now.

    Thanks for your time

    • This reply was modified 4 years, 7 months ago by SoN9ne.
Viewing 6 replies - 1 through 6 (of 6 total)
  • The topic ‘WordFence and ASG & EFS issues, had to remove’ is closed to new replies.