Robots.txt file Made Easy & Awesome

Robots.txt is an important file that tells search engines what to crawl and what to ignore. I tried a number of formats for the robots.txt for our website and was lost in the world of ‘no follow’, ‘no index’ etc. when I was miraculously saved by Better Robots.txt plugin. It has automated the entire process that’s extremely easy to set up.
Better Robots.txt plugin creates a virtual robots.tx file and guides you through the entire process of selecting the options and creating a file that works. It also works seamlessly with the Yoast plugin. This means if you are using the free version that doesn’t allow you to add a sitemap to the file, your Yoast sitemap will do the trick. The Pro version has many tempting features like Backlinks requests, crawl relevant pages, bad bot blocker etc. And yes, their support is awesome. Highly recommended.

• This topic was modified 5 years, 10 months ago by naureen.