W3TC conflict with AIOSEO?

  • Resolved yknb

    (@yknb)


    3 years, 6 months ago

    Hello everybody.
    After installing W3TC to my site I noticed that the attendance of my site dramatically reduced.
    I noticed that AIOSEO began complaining like this: “AIOSEO has detected a physical robots.txt file in the root folder of your WordPress installation. We recommend removing this file as it could cause conflicts with WordPress’ dynamically generated one. AIOSEO can import this file and delete it, or you can simply delete it.”
    Deleting robots.txt did not help much. It was recreated by something.
    When I had a look at it contents it was clear what has been recreating it:

    # BEGIN W3TC ROBOTS
    User-agent: *
    Disallow: /wp-content/cache/
    # END W3TC ROBOTS

    So, the scenario seems to be as follows: AIOSEO creates its own dynamical robots.txt with all SEO bells and whistles. However W3TC automatically creates (and re-creates) physical robots.txt in the root directory of the site.
    Physical files take precedence over dynamically generated ones (if the web server finds a physical file, it will serve that and immediately return before checking the rewrite rules), so AIOSEO won’t be able to serve its robots.txt

    Thus W3TC makes useless all AIOSEO efforts.

    Please, correct me if I’m wrong. And advise how to resolve this.

    Thank you.

Viewing 8 replies - 1 through 8 (of 8 total)

  • AIOSEO dev here.

    I’ve reproduced this and confirmed that this conflict is caused by W3TC. The changelog says they have “disallowed crawling of cache directory” in version 2.7.1. They are forcefully generating a physical robots.txt file, overriding rules from any other plugin that touches the robots.txt file.

    I’ve opened an issue on their end so they can address this – https://github.com/W3EDGE/w3-total-cache/issues/420

    Plugin Contributor Marko Vasiljevic

    (@vmarko)

    Hello @arnaudbroes

    I am sorry about the issue you are experiencing.
    @arnaudbroes I’ve seen that you’ve opened an issue and we are working on a fix for this.
    Thank you for understanding and your patience.

    Hi Marko

    Can we safely revert to v2.1.6 just by uploading the files, overwriting 2.1.7?

    The robots.txt issue is causing Google to flag our site for mobile usability issues, because Googlebot cannot access the cached css and js, therefore it renders the page incorrectly.

    Thanks.

    Plugin Contributor Marko Vasiljevic

    (@vmarko)

    Hello @swiftyfrisko

    The best thing to do is to delete the robots.txt file, create your own and set the permissions to 444.
    Alternatively, you can install the WP rollback plugin and revert to the previous version of the plugin.
    The files that were changed are:
    Generic_Environment.php
    Util_Rule.php
    w3-total-cache-api.php
    So you can download the previous version from here and upload the files that were changed.
    Once again thank you for your patience and we are working on a fix for this.

    Ok, thanks for the quick and comprehensive reply Marko!

    Plugin Author Joe Cartonia

    (@joemoto)

    @yknb @arnaudbroes @swiftyfrisko We have released a patch in version 2.1.8. We do apologize for any inconvenience. Please update and do let us know if there are any issues. We will be happy to assist you. Thank you.

    Thanks guys. Robots.txt is back to normal in 2.1.8.

    Plugin Contributor Marko Vasiljevic

    (@vmarko)

    Hello @swiftyfrisko

    Thank you for the information.
    Glad to know the issue is now resolved and once again we are sorry for the inconvenience and thank you for your patience.

Viewing 8 replies - 1 through 8 (of 8 total)
  • The topic ‘W3TC conflict with AIOSEO?’ is closed to new replies.