Indexed, though blocked by robots.txt

Viewing 1 replies (of 1 total)
  • Ray Abella

    (@supportray)

    Hi @uraniumhazee,

    Thanks for contacting us!

    If you block Googlebot from crawling your links/pages, these can still get indexed if there are other links that are pointing to them. Now, it sends out an error because Googlebot doesn’t see the content since they are blocked from crawling that link.

    To fix this issue, you’ll need to add a rel=”noindex” tag to your affiliate links, found in Link Option settings (https://snag.gy/ZclTu0.jpg) or apply this in your ThirstyAffiliates > Settings > Link Appearance > Additional rel attribute tags. Then allow Googlebot to crawl the link.

    Please disable the Enable Bot Crawl Blocker Script on ThirstyAffiliates > Settings > General and check your .htaccess and robots.txt file to make sure that Googlebot is allowed to crawl the link.

    Since you need to allow Googlebot to crawl your links, I suggest setting the No follow option to Yes, which adds the rel=”nofollow” tag on your affiliate links.

    I hope this helps! Let me know if you have any other questions!

Viewing 1 replies (of 1 total)
  • The topic ‘Indexed, though blocked by robots.txt’ is closed to new replies.