robots.txt not showing blocked crawlers

  • Resolved tombie

    (@tombie)


    2 weeks, 3 days ago

    Hi!

    I activated the plugin and see no change when browsing my robots.txt file. I have no manually created robots.txt so I’m wondering what could be the problem. And to my knowledge, none of my other installed plugins should be do anything with the robots.txt. I’m not very techsavy, so thanks for any help!

    The page I need help with: [log in to see the link]

Viewing 2 replies - 1 through 2 (of 2 total)
  • Plugin Author lastsplash (a11n)

    (@lastsplash)

    Hi @tombie –

    Sorry for the wait here.

    I haven’t had any reports of the robots.txt not displaying, but most often, issues like this are caused by a plugin or theme conflict.

    Could you temporarily deactivate all plugins other than Block AI Crawlers and then access your robots.txt to see if it includes the AI blocks?

    Thread Starter tombie

    (@tombie)

    Hi! I found the plugin responsible. It was XML Sitemap Generator for Google. There is a setting in the plugin that I had to uncheck for the blocker to add the stuff:

    ?[ ] Add sitemap URL to the virtual robots.txt file.
    <small>The virtual robots.txt generated by WordPress is used. A real robots.txt file must NOT exist in the site directory!</small>

    Sadly this also makes the two sitemap links that the addon makes not show up in the robots.txt. But at least my content is protected from AI scrappers now!

Viewing 2 replies - 1 through 2 (of 2 total)
  • You must be logged in to reply to this topic.