[Plugin: WordPress SEO by Yoast] How to keep certain pages from Search Engines

I went through the configuration pages for WordPress SEO and there doesn’t appear to be any provisions within the plugin itself to exclude specific URLs.

In my case, I used the robots.txt file to prevent search engines from crawling specific URLs.

Excluding URLs using robots.txt is probably the preferred method as it is independent of any plugin and it’s old web standard that compliant search bots follow.

In addition to excluding URLs using robots.txt, I also logged into Google Webmaster Tools and requested a removal of all pages that I didn’t want listed in the Google SERP.

Thanks for the info…I’m brand to this, so not sure, where to find this robot.txt file…I will try and see if I can find out today, and will post again…I may need some additional guidance, lol ??

As a follow-up, did some searching on the SEO, but couldn’t find anything worthwhile. Finally went into Pages, and below that the SEO section where you can modify SEO settings per page.

Found a section titled:

Meta Robots Index-which I changed from Follow to No Follow
Meta Robots Follow-Changed from Follow to No Follow.

I changed all my pages to No Follow on the two areas mentioned above, but left my About page the same. Couldn’t find anything specific on how to do this, or what methods are normal, but will keep the post alive, in case someone else has any info, or needs help.

About the only setting I could see for this problem, so will see what happens, next time they crawl the site.

Follow/NoFollow is actually for specifying which links should, or should not influence the page ranking that the link points to.

From what I understand, Follow/NoFollow is a way for site owners to control the flow of “link juice” (to use the parlance) and was introduced by Google as a way to mitigate the attempts by spammers to manipulate page ranking via link juice harvesting and manipulation.

Follow/NoFollow is not really used for controlling content access by search bots. That’s still robots.txt’s job.

Google Webmaster Tools has an easy-to-use robots.txt generator which you can find under Site Configuration -> Crawler Access -> Generate Robots.txt tab

Once generated, you simply upload it into your site’s root directory.

Note: If you have your blog installed into a sub-directory AND you have a subdomain for your blog (.e.g https://myblog.mydomain.com) then you have to upload your robots.txt into your blog’s sub-directory and not the site’s root directory.

Really appreciate your help in this matter. I’ll do as you suggest and check this out. In the meantime, I guess I should go back and change the No Follow on these pages back to Follow, is that correct?

Yes, that’s right. You should switch those pages back to Follow.

Here’s a video where Google Software Engineer Matt Cutts answers a user question regarding NoFollow and internal links.

In a nutshell, Matt advises NOT to use NoFollow on internal links. Aside from very specific cases, you SHOULD allow page rank to flow freely throughout your site via its internal link structure.

Oh, and having pages indexed that have some technical relevancy to your site (like your log-in page, or your site policy pages etc.) but have no “search engine value” is not considered a negative thing by Google.

It doesn’t do your site any harm. But I understand why people prefer that they not show up in the SERP.

That’s my preference too. I use robots.txt to stop Google from indexing any page/link on my blogs that serve a purely technical purpose and don’t have any search engine value with regards to relevant content.

I went back reinstalled the follow links for all pages. I didn’t know and wasn’t concerned abut messing up the page ranking…The thing is, that it shows the site url on top, and then lists every technical page below that. I guess as you say, it doesn’t hurt, but I don’t see the need for having these pages show up.

I haven’t got around to your suggestion on the robots.txt., but I think I iwll give it a go. I just can’t see the need for my tech pages being listed.

Thanks again for all the advice ??

I looked over the video link for Matt Cuts…Thanks so much for the info. Lots of good video here, that I need to look over. ??