Robots.txt unreachable

  • aitta10

    (@aitta10)


    4 weeks ago

    Google and Bing cannot fetch my robots.txt file. It was being fetched normally for long time and I didnt make any changes.

    The page I need help with: [log in to see the link]

Viewing 2 replies - 1 through 2 (of 2 total)
  • MarkRH

    (@markrh)

    Well, I was able to view your robots.txt file in Firefox so it seems to be there. I changed my user agent to a Google bot variant and I got a 500 Internal Server error response instead. Not real sure why changing the user agent would cause that but would explain the bots are not getting the file.

    threadi

    (@threadi)

    If an error 500 really occurs for the user agent, you should check the error log to see what is causing it. It could be a security tool on your hosting or a security plugin in your WordPress.

Viewing 2 replies - 1 through 2 (of 2 total)
  • You must be logged in to reply to this topic.