Exclude search in robts.txt bug

  • Resolved tokyojoe99

    (@tokyojoe99)


    1 year, 8 months ago

    Hi,

    I want to exclude /search and /?s= from robots.txt but AIOSEO appends a ‘/’ trailing slash that stops this targetting properly.

    How do I achieve this?

Viewing 2 replies - 1 through 2 (of 2 total)
  • Plugin Support Steve M

    (@wpsmort)

    Hi @tokyojoe99,

    WordPress automatically set the search and search results page to NOINDEX to stop search engines indexing these.

    Google strongly recommends you don’t block them from crawling these pages using the robots.txt because then they won’t see the NOINDEX meta tag.

    Search engines need to be able to crawl those URLs in order to see that meta tag and not index them.

    Thread Starter tokyojoe99

    (@tokyojoe99)

    Hi Steve,

    That is a great answer, thank you.

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Exclude search in robts.txt bug’ is closed to new replies.