Indexed, though blocked by robots.txt

Viewing 3 replies - 1 through 3 (of 3 total)
  • MariusG

    (@marius_codeinwp)

    Hi @koolkatdesign,

    Google can still index URLs blocked by robots.txt if they are linked to from other pages which Google can crawl.

    If it is a single URL, then you can safely ignore it. If you are getting this warning for multiple URLs, then blocking them through robots.txt might not be the best solution. Instead, you might have to noindex them using the X-Robots-Tag.

    Thread Starter KoolKatDesign

    (@koolkatdesign)

    Hi @marius_codeinwp,

    I do understand that Google can still index URLs blocked by robots.txt if they are linked to from other pages which Google can crawl.

    The thing is that I am using the free version of Yoast and didn’t therefore didn’t create the robots.txt file or the url that is throwing the warning.

    This page doesn’t exist in the dashboard pages so I am assuming that the calendar app is auto generating it from its scripting?

    The good news is that this is the only page I am getting a warning for. Do I just ignore the warning or do I adjust a setting somewhere to stop this conflict from happening? If so, where as this page doesn’t exist in my WP dashboard.

    MariusG

    (@marius_codeinwp)

    You can edit the robots.txt file both with the free version too.

    As mentioned before, if the page cannot be accessed from your dashboard, then the only solution is to use X-Robots-Tag.

    Being a single warning, it is safe to ignore it. But if other pages like this start getting indexed, it’s best to use the X-Robots-Tag solution.

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘Indexed, though blocked by robots.txt’ is closed to new replies.