Prominent words directory being crawled

@joshuagellock We’re sorry to hear you are experiencing this issue on your site.

If you want to prevent a directory from being crawled you can use the Disallow directive in your robots.txt file. You can find more information on how to implement this here: https://yoast.com/ultimate-guide-robots-txt/

Hi, @devnihil –

Thanks for the response. I’m familiar with using robots.txt to prevent the crawling of directories and files, but I guess I was trying to figure out if this is expected behavior from the plugin. Is it? It seems odd that I’ve used Yoast on countless sites but have never experienced this issue before.

Thanks!

Thanks for following up with us. We can confirm that it’s not expected at all to see the Yoast SEO-specific prominent words URLs are being indexed by Google as prominent words are internal and Yoast SEO doesn’t output anything for those.

So, if those URLs are being crawled by Google, this means somehow Google crawler is finding during the site crawling process. You’ll need to make sure that nothing is linking such an URL on your website, nor those URLs appear on the frontend part of your website where Google crawler could reach and crawl them. That’s how you could prevent them from being crawled by search engines.

We are going ahead and marking this issue as resolved due to inactivity. If you require any further assistance please create a new issue.