URL Scanning Hits The Database Too Much

  • We have a site that makes a large number of database requests when Wordfence scans run. We suspect the problem is the Wordfence URL scanner. We are still confirming this.

    A few weeks ago we enabled bit.ly link shortening in our social sharing plugin. This caused the plugin to generate a bit.ly link per social network for every post in the site. In our case this is 6 bit.ly links per post.

    The last successful Wordfence scan checked over 700,000 URLs. We know the increase in scanned URLs coincides with our enabling bit.ly link shortening because we see the progression from just over 60,000 URL scans performed before we enabled link shortening through to 200,000 then 500,000 then 600,000.

    We also know the site does not contain 700,000 URLs. We checked the database.

    My immediate question is, which of the Wordfence options do I disable to disable URL scans in wp post meta? Currently I’ve disabled all URL scans.

    My next question: is there a bug in the URL scanner that causes Wordfence to scan the same URL multiple times?

    Final question: can Wordfence not grab the URLs that are in a post, including post meta, in one DB request?

    Lee

Viewing 4 replies - 1 through 4 (of 4 total)
  • Thread Starter Lee Hodson (VR51)

    (@leehodson)

    Just making sure you didn’t miss this over the Easter weekend. Do you have any feedback to provide?

    Your Hostname:
    p1341226-ipngn201007fukuokachu.fukuoka.ocn.ne.jp

    153.133.173.226

    Hey @leehodson,

    My apologies for the delayed response with this.

    The options you’ll want to disable is Wordfence > All Option > Scan Options > General Options > Scan posts for known dangerous URLs and suspicious content.

    As for why the URL scan count is so high, I’m going to speak with the developers for their thoughts on this. I’ll report back here when I know more.

    Thanks,

    Gerroald

    • This reply was modified 5 years, 6 months ago by WFGerroald.
    • This reply was modified 5 years, 6 months ago by WFGerroald.

    Hey @leehodson,

    I just spoke with the developers and they have some thoughts as to what might be happening.

    Do you have any caching on the site? If so, it’s possible the URLs are being grabbed for files that are currently cached as well. It will grab all URLs, even duplicates. If you do have any caching on the site, can you please clear it and let me know if it helps?

    As for the question regarding if we can grab the URLs in one request, we really can’t do it this way. If we were to try this, there almost certainly be issues due to running out of memory.

    Please let me know if this helps.

    Thanks,

    Gerroald

    • This reply was modified 5 years, 6 months ago by WFGerroald.
    • This reply was modified 5 years, 6 months ago by WFGerroald.
Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘URL Scanning Hits The Database Too Much’ is closed to new replies.