robots.txt deprecated by Google?

  • Google announced back in 2019 that the robots.txt to block indexing would no longer be honored.

    Google recommends the META TAG per page avenue of <meta name=”robots” content=”noindex,nofollow” />

    These tags appear correctly on my pages.

    My WordPress site is still creating the robots.txt file

    Google says – https://developers.google.com/search/docs/advanced/crawling/block-indexing

    Important: For the noindex directive to be effective, the page must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file or it can’t access the page, the crawler will never see the noindex directive, and the page can still appear in search results, for example if other pages link to it.

    Whats a feller to do?

    What does WordPress recommend?

    Should I delete the robots.txt file??

    Any thoughts or comments appreciated!

Viewing 1 replies (of 1 total)
  • Moderator Yui

    (@fierevere)

    永子

    You can delete robots.txt file.

    However….

    * There are other search engines but Google and they still honor the presence of robots.txt and its directives

    * WordPress, together with installed SEO plugin, will generate meta robots tags (index,noindex…) and X-robots HTTP headers according to your settings.

    * robots.txt will still be generated (virtual file) according to your settings.
    (thats why you can delete this file from disk)

Viewing 1 replies (of 1 total)
  • The topic ‘robots.txt deprecated by Google?’ is closed to new replies.