Remove unwanted URL’s
-
My website has more than 3000 unwanted urls including tags, RSS feeds url’s, cat URLs, pagination urls, and some other types of URLs’s here you can see some screenshots.
I want to block all these url’s to not be crawled by Google bots. i have applied all the options from Yoast > setting > Crawl optimization here you can see https://tinyurl.com/ypon8p5j
But I can’t see any text in the robots.txt file here you can see https://tinyurl.com/yve54tvf
After applying all these options I sent a request to Google Crawl from GSC but it failed.
So can you please me out from all these?
Thanks
-
Hello @akhtarjan,
Thanks for using the Yoast SEO plugin.
When you create a robots.txt file with the Yoast SEO plugin, it should have default directives as shown in this screenshot. It’s strange that yours is empty unless the contents were previously removed.
I checked your virtual robots.txt file, created by WordPress, and it has the directives. You can copy those directives to the one created by Yoast SEO and save changes.
Furthermore, it’s not every setting in the crawl optimization feature that’s meant to add a directive to your robots.txt file. For instance, the first three buttons you toggled ON are to remove unwanted URLs. To confirm they worked, view the page source and search for shortlink and it should not be there.
If you toggle the buttons for the Internal site search cleanup, extra directives such as
Disallow: /?s= Disallow: /page/*/?s= Disallow: /search/will be added to the virtual file.One of your screenshots also shows /feed/ URLs. You should check out the Disable unwanted content formats feature. I highly recommend reading what each setting does before actually toggling the buttons – Yoast SEO settings: Crawl Optimization.
Do let me know how it goes.
- The topic ‘Remove unwanted URL’s’ is closed to new replies.
