Shared Hosting Sites Crawling Each Other

  • Resolved daphnetalbot

    (@daphnetalbot)


    8 years, 10 months ago

    Hi,

    This is a bit odd and a bit convoluted. My hosting company says no worries but I should check the blogs. So here I am.

    It seems my websites are somehow crawling each other and they shouldn’t be.
    I have a shared hosting server using PLESK running with Apache.
    90% of my sites are using WordPress.
    All WordPress sites have the Wordfence Premium Plugin.
    Wordfence shows that my sites are being crawled every 15 minutes by my hosting server. No traffic should come from my server.
    Here is the confusing part:

    Domain1.com’s Wordfence logs show Domain2.com crawling it, about 10 pages in seconds, stopping and then doing it again every 15 minutes.

    Domain2.com’s PLESK Access logs show that at that exact same time as the Wordfence logs, IP xx.xxx.xxx.123 (just an example) having been blocked by Apache from bad behaviors.

    Domain2.com’s Wordfence logs show Domain3.com crawling it, about 10 pages in seconds, stopping and then doing it again every 15 minutes.

    Domain3.com’s PLESK Access logs show at that exact same time as the Wordfence logs, IP xx.xxx.xxx.456 (just an example) having been blocked by Apache from bad behaviors.

    This daisy chain of events cycles through all 67 domains on my shared IP. I just noticed this last week. All sites are have WordPress and plugins up-to-date. I have another shared IP and the same thing is happening on it as well.

    The shared hosting situation will stay as is. Other than that has anyone seen this? Is there a concern here and if so how do I stop it?

    Thanks,
    Daphne

    https://www.ads-software.com/plugins/wordfence/

Viewing 2 replies - 1 through 2 (of 2 total)
  • Plugin Author WFMattR

    (@wfmattr)

    Wordfence’s logs show the traffic to the site, but I don’t think this issue is likely to be caused by Wordfence.

    Are you using W3 Total Cache on the sites? That plugin has a feature to preload cached pages, which would appear as visits from your site’s own IP, and if I remember correctly, I think the default timing is to load 10 pages every 15 minutes.

    The strange chain of various domains appearing to crawl each other might be a DNS issue — if you have multiple PTR (reverse lookup) records for the host’s IP address, it could cause each lookup to show a different one of the domain names.

    -Matt R

    Thread Starter daphnetalbot

    (@daphnetalbot)

    Matt,

    Yes I have W3 total cache on the sites. My memory is slipping but I think this all started when I installed w3 on most of the sites. I’ll have to dig down that hole and see what I can uncover.

    I’ll got back to the company maintaining my server and asked them about the multiple PTR records and the loop you mentioned. It is a bit beyond what I know and understand. Hopefully my hosting company will be helpful on it.

    Thanks for the feedback!!!!

    Daphne

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Shared Hosting Sites Crawling Each Other’ is closed to new replies.