Failed: Robots.txt unreachable

Hello @wandex,

Thank you for contacting Rank Math and bringing your concern to our attention.

This situation usually happens when there’s a problem with the file’s permissions or location on the server.

Please contact your hosting provider for assistance. They should be able to help you resolve the issue and grant the necessary permissions to the robots.txt file so that search engines can crawl your website.

You can also check the file path to ensure that it’s in the correct location and that the file name is written correctly as “robots.txt”.

If an actual robots.txt file is present in the root folder of your site, the option to edit the file from Rank Math > General Settings > Edit robots.txt won’t be available.

Please delete it from the root folder and edit it from the Rank Math Plugin if required.

Learn more about robots.txt here:
https://rankmath.com/kb/general-settings/#edit-robotstxt

We hope that helps, and please do not hesitate to let us know if you need our assistance with anything else.

Hi,

thanks for quick response, file doesn’t exist in root folder.
I edit robots.txt file in rankmaths.
also i checked with hosting company, they set the permissions. no restrictions.

still the issue remains the same, if you could check in more, that will be helpful.

Hi,
I am also facing the same issue with AIOSEO plugin after update to 6.2

Pages are not get index. robots.txt file is blocked for google so google ad-words also not working due to this issue.

I did contact to hosting company, they checked there is no restriction to googlebot.

kindly check and let us know.

https://sparklelaser.com/robots.txt

Hello @wandex,

Your robots.txt includes a rule to disallow Googlebot on your entire website. Please make sure that you remove that and add the suggested rules here: https://rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/#default-rules

Otherwise, you can add a manual Robots.txt file like the following:
https://rankmath.com/robots.txt

Don’t hesitate to get in touch if you have any other questions.

P.S. @keyurshah1141,

Since your issue is with a different plugin, we would ask that you contact their support about this situation. Thank you.

Hi,

It is like you mentioned, kindly check

https://rightmotions.com/robots.txt

if you require wp logins, let me know.

Hello @wandex,

Your robots.txt seems to be loading fine in the browser. However, Google is still not able to reach it and it must be related to a server issue. Please check with your host if they or any firewall on the server is blocking Googlebot.

And currently, your robots.txt file doesn’t have the default rules recommended by our plugin. Please make sure to add these rules: https://rankmath.com/kb/how-to-edit-robots-txt-with-rank-math/#default-rules

Let us know how that goes. Looking forward to helping you.

Hello @wandex I have the same problem. Have you solved it please?

@wandex Have you solved this issue ?

Hello @lodenn and @tomickck,

Please follow the steps mentioned in our previous reply to resolve the issue on your sites:
https://www.ads-software.com/support/topic/failed-robots-txt-unreachable/#post-16635120

Let us know how that goes. Looking forward to helping you.

Same problem here. Have checked the mentioned steps and the settings are extactly the same.

Hello @lwup,

Please open a new support topic so we can assist you better.

For now, if your robots.txt file is also unreachable for Google, then please check your site’s firewall or get in touch with your host to make sure that Googlebot is not being blocked.

If the issue persists, please share the URL of your site so we can check on our end.

Looking forward to helping you.

Hi Support,
Same problem here :(?

Error (translated): Robots.txt file is unreachable in writing (cannot be edited…)

CONTEXT:?

WordPress 6.3.1

Clean domain name ( already indexed from nov. 22 to may 2023)

Theme: Blocksy (basic)/Child Theme, Flatsome (licensed)

I installed RankMath but I noticed that my page are not indexed. ?

TROUBLESHOOTING:?

Typing this URL to check the accessibility of ‘Robots.txt’? https://*mydomain/robots.txt > OK (content is displayed)

Typing this URL to check the sitemap :? https://*mydomain/sitemap_index.xml > OK ?

Informational banner (translated): “this plan of site XML is managed by the SEO plugin Rank Math for WordPress. […]

NOTE: 3 plans are displayed but more recent is stamped on 2023-09-15 > OK
> https://*mydomain/page-sitemap.xml

Checking from Google Search Tester > OK ?

But timestamp of the latest version seen is prior I installed RankMath plugin:? seen on 9/8/23?

https://www.google.com/webmasters/tools/robots-testing-tool?siteUrl=https://*mydomain


Browsing /public_html > I can’t see Robots.txt file

Searching in Child Theme or Flatsome folders > not found.?

Seems like the ‘Robots.txt’ has been moved, then it’s only reachable via a redirection…(?!?)


Thank you in advance for your help. :)?

PS: known issue but in my case the file is not listed in /public_html
https://azdigi.com/blog/en/website-tutorials/wordpress-en/fixing-the-robots-txt-file-is-not-writable-error-when-using-rank-math/

• This reply was modified 1 year, 2 months ago by charlyb11.

Hello @charlyb11,

Please note that our plugin doesn’t have the ability to block incoming crawlers in your robots.txt as the robots.txt will only instruct crawlers which route is blocked or not so if you receive any reports that your robots.txt is not reachable to search engines, you would need to check with your hosting if they are blocking incoming bots.

You may share your website URL here so we can check the issue further.

Looking forward to helping you.

Hi Support,
Thank you for your reply.
This is the URL + sitemap:
https://bebe-waw.com/sitemap_index.xml

NOTE: The timestamp is up to Sep 11.

Today, I noticed that 32 pages are now displayed in Google Search Console which is good since I added robots.txt file manually in the /Public_html folder on Sep, 16.
However, the sitemap link were still not displayed in the ‘Sitemap’ in the Search Console as a ‘Submitted sitemaps’ and as I mentioned above in the ‘NOTE’ the timestamp is Sep 16.
I’ll keep monitoring this then update you asap.
Thanks and regards,
—
CharlyB

Hello @wandex,

Your sitemap seems to be working and validating fine. We can also check that it is updated: https://i.rankmath.com/i/HeKy2T

Please submit it manually to the sitemap section of your GSC account and Google should be able to read it without any issues.

Let us know how that goes. Looking forward to helping you.