Problems in Google

Ah, and just as proof – as I know that you guys like a bit of proof, here’s the screenshot

Think I worked out why rankmath is not indexing.
Seeing as its a new website and there was no robots.txt file in my host (even though there was an uneditable one in rankmath’s settings), I reckon that all this depends on users migrating from the likes of Yoast, which actually adds a robots.txt file straight into your hosting panel.
So now then it’s just the ‘noindex’ problem for pages, posts, categories, tags and whatever.
At least when Yoast had a bug in their indexing (for attachments), they eventually had to own up to it and pushed out a new update.
So what’s the problem with rankmath noindexing everything?

Decided to throw rankmath away and go back to Yoast.
Within the aforementioned half hour, heres the SCREENSHOT of how the website looked in Google Search Console.
Nuff said, eh?

• This reply was modified 4 years, 8 months ago by classies.

Hello @classies

Thank you for contacting the support and sorry for any inconvenience that might have been caused due to that.

As far as I know, no plugin handles the robots.txt file, WordPress does. The file created by the plugins is virtual, which means it doesn’t exist on the disk/server, just like any of the posts and pages.

With that said, we can understand your frustration and are sorry about that.

If in the future, you decide to try Rank Math, we recommend following this guide while migrating:
https://www.matthewwoodward.co.uk/seo/reviews/rank-math/

If there’s anything, please let us know. We would be more than happy to assist.

As far as I know, no plugin handles the robots.txt file, WordPress does.

No, not really.
In fact it’s highly publicised that to create a .htaccess file you just need to trash the present one and refresh the permalinks.
However a robots.txt file is NEVER created by WordPress.
Did you not know that?
A robots.txt file either has to be placed manually, or one is created for you by Yoast.
And if so, why was there an un-editable robots.txt file in rankmath (due to there not being one in there) and when I placed a robots.txt file due to noindexing problems, why did rankmath lock it down instead of allowing to edit it?
In fact, why is the robots.txt file even in there at all if it cannot be edited?

Hello @classies

A robots.txt file is never created by WordPress (core), that is correct. WordPress has a rewrite rule to “catch” requests made to /robots.txt and if there’s no actual robots.txt file on the server then WP serves the file contents dynamically, and the client will see it as if there was a robots.txt.

Yoast in their plugin opted for using/editing a physical robots.txt file, so the plugin will create one if you haven’t already created it. Rank Math on the other hand (just like other SEO plugins) changes the dynamic output generated by WordPress. If a real robots.txt file exists on the server, then the field in Rank Math will be uneditable and you can only edit the file manually. Rank Math never creates or edits an actual file.

The way this works is described in the plugin right above the field where you can edit the robots.txt, along with a link to our Knowledge Base where we explain it in more details.

I’d like to add that the robots.txt file is probably not the cause of your indexation issues, as Google recently dropped support for noindex and some other directives in the robots.txt: https://searchengineland.com/google-to-stop-supporting-noindex-directive-in-robots-txt-319003
It might cause issues if parts of your site are disallowed with the robots.txt but even that only stops crawling but not indexing.

Hope that helps.

Dear @rankmath,
Please check the screenshots available in the second and the fourth posts in this thread to check the issues received on noindex.

Hello @classies

We are extremely sorry for the trouble once more.

If you check the screenshot you shared, you will clearly notice that Google says the page wasn’t indexed due to a robots meta tag:
https://i.rankmath.com/IxIq0W

If it were caused by the robots.txt file (or lack there of), it would mention that clearly.

There are a couple of places where you add the noindex robots meta tag. The first one is at WordPress Dashboard > Settings > Reading

The other one is offered by your SEO plugin. Rank Math offers that at WordPress Dashboard > Rank Math > Titles & Meta > Homepage (or other tabs for other pages)
And on your individual post/page/CPT using the advanced tab: https://i.rankmath.com/UGciOq

Once you change the robots meta tag, you must allow Google some time to recrawl your website, see the updated robots tag, and act accordingly. If it was noindex before, it will take 48-72 hours for Google to see that the tag has changed to index.

Also, the new Google Search Console sometimes throws errors that do not exist on a website.

With that said, if you ever decide to try the Rank Math plugin again, please let us know and we can help you migrate.

Good day.

Well this is going nowhere, as even though the first screenshot (using rank math) clearly shows noindexing errors, and the second screenshot (using Yoast) clearly shows that the errors have gone within less than thirty minutes (rather than 48hrs) (which is why I published the screenshots in the first place as i knew that you wouldn’t accept it), it seems that the support team are acting rather strangely.
I won’t be using rankmath again, so you can now mark this as resolved.

Just one more thing:
You say that ‘If you check the screenshot you shared, you will clearly notice that Google says the page wasn’t indexed due to a robots meta tag:‘
However, this screenshot was taken when using rankmath, with no robots.txt added or taken away, and set up using rankmaths directives.
So going on the above logic, how on earth can you say that rankmaths way of working is better than Yoast’s way of working?
As far as I can see, rankmath works ok for websites that have been transferred over from the likes of Yoast, and so use Yoast’s, or a manual sitemap.
For new websites however, rankmath leaves a lot to be desired.

troll