Site blocked by robots.txt but I cannot find where

  • Resolved designerleelee

    (@designerleelee)


    1 year, 1 month ago

    Google search console says this URL is blocked by robots.txt but I cannot see where or how. In the WP settings > reading — search engine visibility is on (box is unchecked). In Yoast I have meta data entered with pages set to allow. What am I missing here? Is there another setting in Yoast to allow search indexing?

    The page I need help with: [log in to see the link]

Viewing 6 replies - 1 through 6 (of 6 total)
  • Plugin Support Maybellyne

    (@maybellyne)

    Hello @designerleelee,

    Thanks for reaching out regarding your site indexing. I see a discrepancy between the robots.txt file by Yoast SEO and the virtual robots.txt file. There’s a directive in the former that truly blocks indexing; the contents need to be the same.

    Please copy the contents of the virtual robots.txt file and paste in WordPress > Yoast SEO > Tools > File Editor > robots.txt.

    Let me know how it goes.

    Thread Starter designerleelee

    (@designerleelee)

    Thank you for your response. I have done that and tried the indexing tool in Google Search Console and continue to receive the following error: URL is not available to Google. I also see in the search console that the sitemap could not be read and to verify it is in the correct location. I’ve tested the sitemap link myself and it works. Any ideas? None of the metadata is showing up for this site in Google search.

    Plugin Support Maybellyne

    (@maybellyne)

    I have done that and tried the indexing tool in Google Search Console and continue to receive the following error: URL is not available to Google.

    Please use the robots.txt tester to validate the file

    I also see in the search console that the sitemap could not be read and to verify it is in the correct location.

    Your sitemap is fine so it’s unclear why Google Search Console couldn’t fetch it. Can you remove any sitemaps you have and resubmit just the sitemap_index?

    Let me know how it goes.

    Thread Starter designerleelee

    (@designerleelee)

    Thank you for your help!

    I have used the robots.txt tester you provided the link for and the robots.txt file was successfully fetched.

    I don’t have any other sitemaps that I can see…Only this one from Yoast. I have resubmitted it and Google Search Console continues to say that it’s blocked. I have tried Bing search as well and Bing is also saying the site is blocked.

    Not sure what to do next. Appreciate any suggestions.

    Thread Starter designerleelee

    (@designerleelee)

    Update: There was a misunderstanding with the directions : “Please copy the contents of the?virtual robots.txt?file and paste in?WordPress > Yoast SEO > Tools > File Editor > robots.txt.”

    I did do this but this was the txt file that contained all the directives blocking the search engines:

    # START YOAST BLOCK # —————————
    User-agent: *
    Disallow: /?s=
    Disallow: /page/*/?s=
    Disallow: /search/
    Disallow: /wp-json/
    Disallow: /?rest_route= User-agent:
    AdsBot Disallow: /
    Sitemap: https://sarasmeaton.com/sitemap_index.xml
    # ————————— # END YOAST BLOCK

    I’ve removed all of this and the site is accessible again.

    Thread Starter designerleelee

    (@designerleelee)

    Marking as resolved.

Viewing 6 replies - 1 through 6 (of 6 total)
  • The topic ‘Site blocked by robots.txt but I cannot find where’ is closed to new replies.