Problem Stopping Search Engines From Crawling Site

  • Resolved robertlf

    (@robertlf)


    6 years, 6 months ago

    I had a consultant install and configure a WordPress site on one of my servers for me. I now want to begin adding blog posts but I don’t the search engines to crawl it quite yet. I logged into the WP Dashboard and went to Settings > Reading and checked the Search Engine Visibility checkbox but I noticed that even after I did this, the pages still seem allow links to be followed even though indexing appears to be disabled:

    <meta name="robots" content="noindex,follow"/>

    I was reading a Yoast article Preventing your site from being indexed the right way that said to add this line to my .htaccess file:

    Header set X-Robots-Tag "noindex, nofollow"

    I did that but the robots meta tag in each page still says “follow” instead of “nofollow” and I don’t understand why. I do have the Yoast SEO plugin installed so I’m wondering if that could be the problem. If so, how do I temporarily disable it? Currently that plugin is showing a warning notification in the dashboard that says, “Huge SEO Issue: You’re blocking access to robots.”

    It might also be due to the fact that I added a block of code to the very end of my .htaccess file in the document root directory to password-protect the blog site https://example.com/blog just as an extra precaution. The end of that file now looks like this:

    
...
# END WordPress
Header set X-Robots-Tag "noindex, nofollow"
AuthType Basic
AuthName "Restricted Area"
AuthUserFile /var/www/html/.htpasswd
require valid-user

    What do I need to do so that the robots meta tag will say “noindex, nofollow”?

    Thanks.

Viewing 2 replies - 1 through 2 (of 2 total)

  • I think your main issue is with the definition of “Follow” vs “No Follow”.

    First off, by checking the Search Engine Visibility box you’ve already accomplished what you wanted. Be aware that search engines may or may not honor your choice, but it will dramatically limit them indexing your site prematurely.

    “Follow/Do Follow” means when search engine crawlers (bots) are indexing a site and come across a link, they will allow that link to boost the “Authority” of the page it goes to.

    “No Follow” means the page that link goes to will not get any “Authority” from the page or site the link is on.

    So basically this setting is you deciding if you want to share your website’s “Authority” with the pages you’re linking to. Usually it’s fine to let your page’s authority pass along to sites you’re linking to, but the No Follow option exists for times when you may not want to.

    Now I’ve mentioned “Authority” a few times here and here’s what it means.

    Google measures how much Authority a site has based on how many other websites are linking to it, and how “authoritative” those websites are. Google measures 2 types of authority. 1 for just the individual page in question, and 1 for the domain as a whole. Google calls these “Domain Rank (DR)” and “Page Rank (PR)”.

    Moz.com also has a similar system they use to accurately approximate the authority ratings Google assigns to a website called Domain Authority (DA) and Page Authority (PA).

    I hope this background information helps you better understand what’s going on with your website and Yoast SEO settings.

    Thread Starter robertlf

    (@robertlf)

    Thanks for that clarification!

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Problem Stopping Search Engines From Crawling Site’ is closed to new replies.

Tags