Robots.txt Tool Conflict with WP Rocket and Cloudflare?

  • Resolved bella2011

    (@bella2011)


    3 years, 3 months ago

    We recently downloaded All in One SEO and we are considering upgrading, however, there seems to be a conflict with our caching plugin (WP Rocket) and CDN (Cloudlfare).

    When the Robots.txt Tool is updated and saved. The file is fine. However, when we clear the site cache and the Cloudlfare cache, it deletes all of our updates and returns to the standard robots.txt file.

    Can you share a fix for this?

Viewing 6 replies - 1 through 6 (of 6 total)
  • Thread Starter bella2011

    (@bella2011)

    Update: I’ve noticed that when we clear the cache (WP Rocket), we lose all of our robots.txt customizations.

    Thread Starter bella2011

    (@bella2011)

    Second issue:

    We migrated from Yoast, and never had an issue with Google fetching the robots.txt file. However, when using Google Webmaster testing tool (https://www.google.com/webmasters/tools/robots-testing-tool), to update the file in Google, we receive the following error:

    robots.txt fetch failed
    You have a robots.txt file that we are currently unable to fetch. In such cases we stop crawling your site until we get hold of a robots.txt, or fall back to the last known good robots.txt file. Learn more.

    The file is accessible at its origin point (domain.com/robots.txt), and we’ve entered the URL into the Google Robots Testing Tool to see if GoobleBot is blocked, and it is Allowed.

    Can you shed light on this issue and a fix?

    Plugin Author arnaudbroes

    (@arnaudbroes)

    @bella2011 I do know of some hosting providers that generate their own robots.txt file, even if you delete it in the site root directory. May I ask which hosting provider you’re using for this website?

    Also, can you share the URL to your robots.txt. Unfortunately we’re not able to provide much help without access to the page.

    Thread Starter bella2011

    (@bella2011)

    Hi there,

    So, we talked with our hosting provider – they do not generate the robots.txt file, they must be manually added in the root directory.

    I’ve recreated the issue again, after confirming the removal of the robots.txt file from the root.

    When changes are made inside the AIOSEO dashboard to robots.txt file and saved. I then hit Clear Cache and all of the updates are erased.

    I’ve read your post on Cloudflare Firewall rules (https://aioseo.com/docs/aioseo-uses-rest-api/) and updated Cloudflare.

    So the conflict is with our Caching plugin (WP-Rocket).

    Have you had prior issues with caching plugins and is there a fix?

    Thanks.

    Thread Starter bella2011

    (@bella2011)

    Update:

    I have fixed the Error:robots.txt fetch failed in Google Webmaster testing tool.

    The only error that remains with the WP Rocket Caching plugin, which erases any changes made to the Robots.txt file in the AIOSEO dashboard, after updates have been saved.

    Plugin Author arnaudbroes

    (@arnaudbroes)

    @bella2011 if you disable WP Rocket and save, the changes do not go away, correct?

Viewing 6 replies - 1 through 6 (of 6 total)
  • The topic ‘Robots.txt Tool Conflict with WP Rocket and Cloudflare?’ is closed to new replies.