Spiders Ignore Addon Domain Robots.txt

  • My primary domain is LibertyResourceDirectory.com (LRD), I have a few addon domains including ethanglover.biz (EG). Recently, I decided to noindex LRD through the robots.txt.

    The problem is, what I put on the LRD robot.txt was applied to ALL my WordPress sites. I have a MediaWiki site that wasn’t effected at all.

    Here’s what I tried.

    1. Manually uploaded a robots.txt file to EG. When I went to EG.biz/robots.txt, it literally showed the primary domain robots.txt and ignored the EG.biz one. (This did NOT happen to the MediaWiki site, it worked as it’s supposed to.)

    2. Deleted the LRD robots.txt and used a meta robots tag while leaving the robots.txt on EG.biz. That robots file isn’t read at all, and I do want one on that site.

    3. This is my current fix, but it’s an odd band-aid patch that I’m not satisfied with. I disallowed all the LRD folders in the robots.txt. Those disallows show up on EG but they only work on LRD because the folders only exist in one place.

    More info: https://ethanglover.biz/using-robots-txt-with-addon-domains/

    Am I the only person this is happening to? I have another WordPress site and it’s having the same issues. I’m 100% sure this isn’t a plugin problem. I don’t really like plugins and use only the minimum. (Yoast, W3 Cache, that kind of thing.)

Viewing 1 replies (of 1 total)
  • Moderator James Huff

    (@macmanx)

    That sounds like maybe a hosting problem to me? I have a number of add-on domains, and there is no cross-contamination between them.

    When the server is properly configured, each domain should be treated as its own entity.

Viewing 1 replies (of 1 total)
  • The topic ‘Spiders Ignore Addon Domain Robots.txt’ is closed to new replies.