Something is updating my Robots.txt

  • Resolved Rob

    (@gravelrash)


    3 years, 2 months ago

    Hi,

    I am looking for a bit of help in troubleshooting an odd issue. There seems to be a process that is running every 30 minutes or so on the site that is changing the ROBOTS.TXT file and adding this text

    User-Agent: Scrapy
    Allow: /

    Ok, this wouldn’t be huge issue because I don’t actively block any bots but when the file is edited it messes up the formatting so the Robots.txt file is unreadable which has meant the site has dropped out of Bing’s search indexer.

    The site is running on a Windows server and I’ve used Process Monitor to watch for changes and I can see the file get updated by the PHP-CGI process but I can’t tell what is triggering the action. Is there any logging I can switch on or is there anything I can query in the database to hopefully provide a pointer to the issue (I suspect its a Plugin doing the changes).

    Any help or thoughts gratefully received

    Rob

Viewing 2 replies - 1 through 2 (of 2 total)

  • The first thing I would do is search the code base.

    Either copy all the code locally and load it into an IDE and search.

    Or just at command line

    grep -iRn 'Scrapy' /path/to/your/install

    Thread Starter Rob

    (@gravelrash)

    Hi Alan

    Thanks for pointing me in the right direction, it looks like its the Termly GDPR/CCPA Cookie Consent Banner plugin that is making the changes so I’ll be disabling that and finding a plugin that doesn’t modify important files badly.

    Cheers
    Rob

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Something is updating my Robots.txt’ is closed to new replies.