Exclude wildcards from scan not working

  • teamicm

    (@teamicm)


    10 years ago

    I’m having trouble with a multisite set up with lots of files.
    There are several interactive thing made up of a bunch of .swfs and .mp3s and many are duplicated across the sub sites (I did not set the site up and we don’t have time to restructure it just yet)

    The upshot is there are 62,000 files on the and the scan stops at 42,000 or so (I’m rounding the numbers) with another scan starting while the first scan is going, so the scanning never stops and the host kills the process or restarts the server. If let run to the end the scan terminates with an error.

    I’ve tried setting the Maximum execution time for each scan stage to 15, and the update interval to 3.

    I then figured I could exclude files from the scans thinking that by doing so fewer files would need to be scanned and analyzed and the scan could finish.
    In the “Exclude files from scan that match these wildcard patterns. Comma separated.” field I have: *.jpg,*.png,*.flv,*.mp3,*.mp4,*.pdf,*.swf

    That should eliminate 52,464 files from the scan leaving around 10k files to be scanned.

    When I actually ran the scan again, it went way past 10k files and eventually started another scan.

    [Nov 04 10:19:36] Analyzed 34300 files containing 6.82 GB of data so far
    [Nov 04 10:19:47] Scheduled Wordfence scan starting at Tuesday 4th of November 2014 10:19:47 AM
    [Nov 04 10:19:53] Analyzed 34400 files containing 6.84 GB of data so far
    [Nov 04 10:23:21] Scan kill request received.

    Am I missing something? Why are files that should be excluded being analyzed? Did I mess up the syntax?

    Thanks

    https://www.ads-software.com/plugins/wordfence/

Viewing 8 replies - 1 through 8 (of 8 total)
  • Thread Starter teamicm

    (@teamicm)

    I let the scan run until it died again with

    [Nov 04 22:48:28:1415170108.576888:2:error] Scan terminated with error: Wordfence file scanner detected a possible infinite loop. Exiting on file: wp-content/blogs.dir/23/files/2012/02/Ergonomics_NFP/data/a24x11x1.mp3

    So it’s failing in a possible infinite loop on file that’s supposed to be excluded from the scan.

    Also, I noticed that even though I have Login Security Settings set with “Immediately block the IP of users who try to sign in as these usernames:” [admin,administrator,icmoly] and when I look at the Live Traffic screen I have people trying to log in with ‘admin’ within the last 30 minutes and most don’t appear to have been blocked.

    WFSupport

    (@wfsupport)

    Were looking into the blocked usernames issue, I think.

    For the initial issue, I can’t see why it would see that file as a loop. Is that a symbolic link to somewhere else?

    Also on the options page do you have box next to “scan images as executable” checked?

    tim

    Thread Starter teamicm

    (@teamicm)

    Thank you for looking into this.

    As far as I know that is a real file in a real place that I can see via FTP.

    Scan image files as if they were executable is not checked.

    I got this from web host:

    Spotted these in the server logs:

    209.160.72.68 – – [04/Nov/2014:20:08:05 -0500] “POST /wp-admin/admin-ajax.php?action=wordfence_testAjax HTTP/1.0” 200 545 “-” “WordPress/4.0; https://icmoly.com”
    209.160.72.68 – – [04/Nov/2014:21:02:18 -0500] “POST /automotiveasa/wp-admin/admin-ajax.php?action=wordfence_testAjax HTTP/1.0” 200 546 “-” “WordPress/4.0; https://icmoly.com/automotiveasa”

    This tells me Wordfence may be scanning under two separate authorities, perhaps Wordfence was installed on one blog before being installed globally?

    I can definitely say that WF was not installed directly on the automotiveasa site before the main site. I did follow the web host’s suggestion of network deactivating the plugin then checking the individual sites to see if it was on. It was not. I currently only have it active on the main site (everyone logs in through that one), so there shouldn’t be separate instances of it running.

    While it’s scanning and reporting the number of files scanned and the amount of data (“Scan Detailed Activity” [Nov 07 13:20:41] Analyzed 700 files containing 51.71 MB of data so far) the ‘busy’ animations are running for “Comparing core WordPress files against originals in repository” and “Scanning for known malware files”.

    When I uncheck those options, the “Scan Detailed Activity” looks just the same as when they are checked.

    WFSupport

    (@wfsupport)

    how about scan files outside my wordpress installation? Is that checked?

    Thread Starter teamicm

    (@teamicm)

    No, sir.

    Last scan settings:

    Scans to include
    N Scan public facing site for vulnerabilities?(Paid members only)
    Y Scan for the HeartBleed vulnerability?
    N Scan core files against repository versions for changes
    N Scan theme files against repository versions for changes
    N Scan plugin files against repository versions for changes
    N Scan for signatures of known malicious files
    Y Scan file contents for backdoors, trojans and suspicious code
    Y Scan posts for known dangerous URLs and suspicious content
    Y Scan comments for known dangerous URLs and suspicious content
    Y Scan for out of date plugins, themes and WordPress versions
    Y Check the strength of passwords
    Y Scan options table
    N Monitor disk space
    Y Scan for unauthorized DNS changes
    N Scan files outside your WordPress installation
    N Scan image files as if they were executable
    N Enable HIGH SENSITIVITY scanning. May give false positives.
    Exclude files from scan that match these wildcard patterns. Comma separated.: *.jpg,*.png,*.flv,*.mp3,*.mp4,*.pdf,*.swf

    Last scan end result:

    Scan Summary:
    [Nov 07 18:19:17] Previous scan terminated with an error. See below. Scan Complete.

    Scan Detailed Activity:
    …
    [Nov 07 18:09:54] Analyzed 47500 files containing 9.7 GB of data so far
    [Nov 07 18:17:51] Analyzed 47600 files containing 9.7 GB of data so far
    [Nov 07 18:19:17] Scan terminated with error: Wordfence file scanner detected a possible infinite loop. Exiting on file: wp-content/blogs.dir/24/files/2012/02/How_to_implement_your_AP_Program/data/swf/engage_ac0c8/engage_content/imageZoom30.jpg

    Plugin Author Wordfence Security

    (@mmaunder)

    The infinite loop message was triggered because the number of times Wordfence “forked” which means the number of times it launched a new scan stage exceeded 1000.

    That’s too high and your scans shouldn’t take that long so we put that in as a safety measure to stop a scan that appears to just be going round and round.

    Please try setting the maximum time for each scan stage to 60, save and try another scan. See if it completes or at least gets past the first 2 minutes.

    If not, then lower it by 5 seconds, save and try again and keep doing that until it makes progress. This will make each scan stage as long as possible (and you can try larger than 60 if 60 works for you) which will increase performance and reduce the number of forks.

    Regarding excluding files. You can enable Wordfence debug mode to see which files are actually being scanned. however note that it will slow your scans drastically so just use it to check which files WF is actually looking at. Then kill the scan, disable debug mode and start a new scan.

    Regards,

    Mark.

    Thread Starter teamicm

    (@teamicm)

    Thank you for looking into this.
    I’ve followed your suggestions.

    I did also switch the settings up a bit and have it remotely starting a scan.

    Setting the max time to 60

    You said to set max time to 60 and see if it gets past the first two minutes. It always gets past the first two minutes. The last scan I did where it ended with 1,000th fork took nearly 5 hours, which is why the web host brought this to my attention.

    Here’s the end of the old scan and the start time. I just noticed the interval between the last couple of entries jumped from around 3 minutes to 8 minutes before it ended.

    [Nov 07 18:19:17:1415413157.522413:2:error] Scan terminated with error: Wordfence file scanner detected a possible infinite loop. Exiting on file: wp-content/blogs.dir/24/files/2012/02/How_to_implement_your_AP_Program/data/swf/engage_ac0c8/engage_content/imageZoom30.jpg

    [Nov 07 18:17:51:1415413071.126730:2:info] Analyzed 47600 files containing 9.7 GB of data so far
    [Nov 07 18:09:54:1415412594.608535:2:info] Analyzed 47500 files containing 9.7 GB of data so far
    [Nov 07 18:05:39:1415412339.448335:2:info] Analyzed 47400 files containing 9.67 GB of data so far
    [Nov 07 17:59:54:1415411994.279384:2:info] Analyzed 47300 files containing 9.6 GB of data so far
    [Nov 07 17:57:01:1415411821.733173:2:info] Analyzed 47200 files containing 9.59 GB of data so far
    …
    [Nov 07 13:23:32:1415395412.395183:10:info] SUM_PREP:Preparing a new scan.


    Excluded Files
    I turned on debug mode and let it run a little while before killing it.

    It definitely looks like files are not getting excluded to me.

    Here are some excerpts from the log:

    STARTING SCAN:

    [Nov 13 08:05:43:1415894743.823227:4:info] Scan engine received request.
    [Nov 13 08:05:43:1415894743.102145:4:info] Starting cron via proxy at URL https://noc1.wordfence.com/scanp/_________.com/wp-admin/admin-ajax.php?action=wordfence_doScan&isFork=0&cronKey=1eb6a4b913ccba35ff571e5
    [Nov 13 08:05:43:1415894743.099110:4:info] getMaxExecutionTime() returning config value: 60
    [Nov 13 08:05:43:1415894743.098143:4:info] Got value from wf config maxExecutionTime: 60
    [Nov 13 08:05:43:1415894743.094715:4:info] Entering start scan routine
    [Nov 13 08:05:43:1415894743.086786:4:info] Ajax request received to start scan.

    …
    AFTER THE MAIN WORDPRESS FILES:

    [Nov 13 08:06:00:1415894760.418928:4:info] Scanning: /var/www/vhosts/_________.com/httpdocs/wp-content/blogs.dir/11/files/2011/07/automotive4-300×201.jpg (Mem:11.8M)
    …
    [Nov 13 08:06:00:1415894760.533045:4:info] Scanning: /var/www/vhosts/_________.com/httpdocs/wp-content/blogs.dir/11/files/2012/02/APP/data/Slide1.swf (Mem:11.8M)
    …
    [Nov 13 08:06:00:1415894760.690046:4:info] Scanning: /var/www/vhosts/_________.com/httpdocs/wp-content/blogs.dir/11/files/2012/02/APP/data/a24x10x1.mp3 (Mem:11.8M)
    …
    [Nov 13 08:06:02:1415894762.802691:4:info] Scanning: /var/www/vhosts/_________.com/httpdocs/wp-content/blogs.dir/11/files/2012/02/AccidentReportingandInvestigation/AccidentReportingandInvestigation.pdf (Mem:11.8M)

    so it definitely appears that my wildcards of *.jpg,*.mp3,*.swf, and *.pdf are not working.

    Also from the debugged scan:
    At around the 1500 file mark (and indeed, 60 seconds after starting) it looks like a new fork was called. Is our server just really slow or are our files just larger than most? Either way, that’s why I’m trying to exclude the media files from the scan.

    [Nov 13 08:06:55:1415894815.523652:4:info] Scanning: /var/www/vhosts/_________.com/httpdocs/wp-content/blogs.dir/11/files/2012/02/ExpierenceRateCalculation/data/a24x6x1.mp3 (Mem:11.5M)
    [Nov 13 08:06:55:1415894815.462981:4:info] Scan process ended after forking.
    [Nov 13 08:06:55:1415894815.072723:4:info] Hashing item in base dir: /var/www/vhosts/_________.com/httpdocs/wp-content
    [Nov 13 08:06:54:1415894814.961941:4:info] Hashing item in base dir: /var/www/vhosts/_________.com/httpdocs/wp-config.php
    [Nov 13 08:06:54:1415894814.957784:4:info] Hashing item in base dir: /var/www/vhosts/_________.com/httpdocs/wp-config-sample.php
    [Nov 13 08:06:54:1415894814.902878:4:info] Hashing item in base dir: /var/www/vhosts/_________.com/httpdocs/wp-comments-post.php
    [Nov 13 08:06:54:1415894814.884197:4:info] Hashing item in base dir: /var/www/vhosts/_________.com/httpdocs/wp-blog-header.php
    [Nov 13 08:06:54:1415894814.785105:4:info] Hashing item in base dir: /var/www/vhosts/_________.com/httpdocs/wp-admin
    [Nov 13 08:06:54:1415894814.782795:4:info] Hashing item in base dir: /var/www/vhosts/_________.com/httpdocs/wp-activate.php
    [Nov 13 08:06:54:1415894814.770728:4:info] Hashing item in base dir: /var/www/vhosts/_________.com/httpdocs/sitemap.xml
    [Nov 13 08:06:54:1415894814.768640:4:info] Hashing item in base dir: /var/www/vhosts/_________.com/httpdocs/readme.html
    [Nov 13 08:06:54:1415894814.743038:4:info] Hashing item in base dir: /var/www/vhosts/_________.com/httpdocs/license.txt
    [Nov 13 08:06:54:1415894814.741781:4:info] Hashing item in base dir: /var/www/vhosts/_________.com/httpdocs/index.php
    [Nov 13 08:06:54:1415894814.710232:4:info] Hashing item in base dir: /var/www/vhosts/_________.com/httpdocs/index.html
    [Nov 13 08:06:54:1415894814.695065:4:info] Hashing item in base dir: /var/www/vhosts/_________.com/httpdocs/fsgf2e4a.txt
    [Nov 13 08:06:54:1415894814.692679:4:info] Hashing item in base dir: /var/www/vhosts/_________.com/httpdocs/__phpinfo.php
    [Nov 13 08:06:54:1415894814.689772:4:info] Hashing item in base dir: /var/www/vhosts/_________.com/httpdocs/BingSiteAuth.xml
    [Nov 13 08:06:54:1415894814.671738:4:info] Hashing item in base dir: /var/www/vhosts/_________.com/httpdocs/.htaccess
    [Nov 13 08:06:54:1415894814.640251:4:info] Got a true deserialized value back from ‘wfsd_engine’ with type: object
    [Nov 13 08:06:54:1415894814.542227:4:info] Loading serialized data from file /tmp/wordfence_tmpfile_wfsd_engine.php
    [Nov 13 08:06:54:1415894814.538954:4:info] Setting up scanRunning and starting scan
    [Nov 13 08:06:54:1415894814.493681:4:info] Setting up error handling environment
    [Nov 13 08:06:54:1415894814.479089:4:info] Requesting max memory
    [Nov 13 08:06:54:1415894814.438136:4:info] Done become admin
    [Nov 13 08:06:54:1415894814.402905:4:info] Scan authentication complete.
    [Nov 13 08:06:54:1415894814.382778:4:info] Scan will run as admin user ‘_________’ with ID ‘_________’ sourced from: multisite get_super_admins() function
    [Nov 13 08:06:54:1415894814.364912:4:info] Becoming admin for scan
    [Nov 13 08:06:54:1415894814.282882:4:info] Checking saved cronkey against cronkey param
    [Nov 13 08:06:54:1415894814.169616:4:info] Exploding stored cronkey
    [Nov 13 08:06:53:1415894813.964433:4:info] Fetching stored cronkey for comparison.
    [Nov 13 08:06:53:1415894813.907597:4:info] Checking cronkey
    [Nov 13 08:06:53:1415894813.861894:4:info] Scan engine received request.
    [Nov 13 08:06:53:1415894813.294533:4:info] Starting cron via proxy at URL https://noc1.wordfence.com/scanp/_________.com/wp-admin/admin-ajax.php?action=wordfence_doScan&isFork=1&cronKey=36f93e065c2be97b2454086f
    [Nov 13 08:06:52:1415894812.851223:4:info] getMaxExecutionTime() returning config value: 60
    [Nov 13 08:06:52:1415894812.711971:4:info] Got value from wf config maxExecutionTime: 60
    [Nov 13 08:06:52:1415894812.679404:4:info] Calling startScan(true)
    [Nov 13 08:06:52:1415894812.558375:4:info] Serialized data for wfsd_engine is 1373332 bytes and is greater than max_allowed packet so writing it to disk file: /tmp/wordfence_tmpfile_wfsd_engine.php
    [Nov 13 08:06:52:1415894812.055892:4:info] Entered fork()
    [Nov 13 08:06:51:1415894811.850443:4:info] Calling fork() from wordfenceHash::processFile with maxExecTime: 60
    [Nov 13 08:06:50:1415894810.972475:4:info] Scanning: /var/www/vhosts/_________.com/httpdocs/wp-content/blogs.dir/11/files/2012/02/ExpierenceRateCalculation/data/a24x5x1.mp3 (Mem:11.8M)

    Thread Starter teamicm

    (@teamicm)

    UPDATE
    The scan with the max time set to 60 appeared to work faster. It took a little less than 50 minutes to hit the 40700 files mark (8.38 GB data).

    The last log entry was recorded at [Nov 13 08:56:18] and it’s now past [Nov 13 09:35:00] with no further indication that it’s working.

    [Nov 13 08:55:54] Analyzed 40400 files containing 8.31 GB of data so far
    [Nov 13 08:55:59] Analyzed 40500 files containing 8.34 GB of data so far
    [Nov 13 08:56:10] Analyzed 40600 files containing 8.36 GB of data so far
    [Nov 13 08:56:18] Analyzed 40700 files containing 8.38 GB of data so far
    (It’s now been over 40 minutes since the last entry)

Viewing 8 replies - 1 through 8 (of 8 total)
  • The topic ‘Exclude wildcards from scan not working’ is closed to new replies.