How do I stop Google from indexing URLs that contain the product attributes?

Hello @newuseruk,

Thank you for contacting Rank Math support.

The attribute URL has the canonical tag of the correct URL so Google shouldn’t index it. However, we can check that Google is indexing those URLs for your site. In this case, please add the following rule in your site’s robots.txt file to prevent Google from crawling such URLs:

Disallow: *?attribute*

Here is how to edit your site’s robots.txt file: https://rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/

Hope that helps. Let us know if you need any other assistance.

Thank you for your reply.

The attribute URL has the canonical tag of the correct URL so Google shouldn’t index it. However, we can check that Google is indexing those URLs for your site.

How do I check this please? I saw the 5xx error on Google Search console, but only for one of the products out of the three that have the same attributes. See image here: https://snipboard.io/iBqb3r.jpg

All of them have canonical links by default and I haven’t overridden it on the product page. Should I do this?

And with regards to robots.txt, I have to edit the file directly on the file system as RankMath won’t let me edit it because the file is on the root folder.

User-agent: *
Crawl-delay: 30
Disallow: *?attribute*


#Begin Attracta SEO Tools Sitemap. Do not remove
sitemap: https://cdn.attracta.com/sitemap/55555.xml.gz
#End Attracta SEO Tools Sitemap. Do not remove

Is this correct?

Thank you

Hello @newuseruk,

It seems that you are using a different SEO software that manages your robots.txt. Can you please confirm? It appears this SEO tool is server based. Please make sure to disable that so you can only allow our plugin to generate the robots.txt.

As for the 5xx error, this is a server error so there must be some error logs written in your host which we can check to determine what’s causing that.

Looking forward to helping you.

I used to use Yoast and then moved to RankMath. I also changed hosting providers from inmotion to Greengeeks. Not sure if any of this has changed the control of robots.txt

How do I change the control back to RankMath please? I will enquire with my provider too.

And with regards to the 5xx error – please see my original post and the reply to it – its because the attributes for the product are being added into the URL and as the date has passsed, the attribute has been removed which causes this error.

Hello @newuseruk,

You will have to delete the physical robots.txt file to allow our plugin to generate its dynamic robots.txt.

The URL is working fine when you visit it directly and Google should be able to crawl it without any issues. We will need the exact error that is being logged in the server error log to help you resolve the issue. You can get in touch with your host and ask them to share the error log for the date Google crawled the URL.

Looking forward to helping you.

I have renamed the robots.txt file to robots.old.txt in the root folder and RankMath still doesn’t recognise that the file isn’t there anymore. Why?

Hello @newuseruk,

We can check that your robots.txt file is now showing the default rules of our plugin: https://theisifiso.com/robots.txt

If you are still not able to edit it, please share a screenshot of the message you are seeing in WP Dashboard > Rank Math SEO > General Settings > Edit Robots.txt so we can check this further.

You can use a tool like https://imgur.com/ to share the screenshots.

Looking forward to helping you.

I still can’t edit the file. Here is the snapshot: https://snipboard.io/LbjvcX.jpg

Hello @newuseruk,

This error is due to the file edits being disabled through your wp-config.php file.

In this case, please follow our guide here to edit your robots.txt with Rank Math:
https://rankmath.com/kb/cant-edit-robots-txt/#not-writeable

You may also get in touch with your web host for assistance in modifying the file.

Hope that helps.