robots.txt file issues

Hi Lucy,

It’s very likely that your WordPress website was set up to discourage search engines.
You can make it public again by unchecking the “Search Engine Visibility” option.

It’s found here, at the bottom: /wp-admin/options-reading.php

Once that’s unchecked, The SEO Framework can take over the robots txt and meta settings.

I also had this problem. It is not working for me. I need to manually do it. I also never ever check the Search Engine Visibility because i definitely welcome google search.

Hi @infinitymusics,

I’m glad you’ve found a manual workaround.

In your case, I’d suggest to access the website via FTP, and delete the robots.txt file in the root folder. Then, WordPress–and in extent, The SEO Framework–can handle a virtual version for you.

If, however, you’re still experiencing this issue, then another plugin might be fiddling with things it shouldn’t. The SEO Framework will never block crawlers via the robots.txt file, as it’s anti-SEO.

I’m not sure if your inquiry is completely related to the question Lucy has. So, if you’d wish to follow up with more details, then I suggest opening a new topic.

Thanks! ??

Hi @cybr,

Thank you for the reply.

I have made those changes as they were indeed checked!

The robots page now says :

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Does this look accurate now?
Google is still showing as no information available. – is this just going to take some time to update or is there still an issue?

Thanks for your help.

Lucy

Hi Lucy,

That looks as it should be ??

It’s going to take some time before search engines notice the change.

Speeding up the process
These steps are completely optional. Without your input, search engines will figure this out automatically.

You can speed up this process by signing up for Google Search Console.
You’ll be asked to verify your account. In The SEO Framework’s SEO settings, you can paste the verification HTML meta tag into respective the Webmaster Meta Settings field.

When verified, follow the link below, select your property, and hit “Submit” at the right when the data looks alright:
https://www.google.com/webmasters/tools/robots-testing-tool

In the new dashboard, you can submit a sitemap, too. It will help search engines quickly route through your website:
https://search.google.com/search-console/sitemaps

With The SEO Framework, the sitemap URL should be like so:
example.com/sitemap.xml

I hope this all helps!