WARNING: This plugin amends robots.txt to allow total access for Scrapy

  • $ grep -iRn ‘Scrapy’ wp-content/plugins
    wp-content/plugins/uk-cookie-consent/includes/controllers/class-robots-txt.php:35: return ‘User-agent: Scrapy
    wp-content/plugins/uk-cookie-consent/includes/controllers/class-robots-txt.php:89: $robots_rule = ‘User-agent: Scrapy

    This results in

    User-agent: Scrapy
    Allow: /

Viewing 1 replies (of 1 total)
Viewing 1 replies (of 1 total)
  • The topic ‘WARNING: This plugin amends robots.txt to allow total access for Scrapy’ is closed to new replies.