solrax

I seemed to have figured it out so I will be posting my solution for others who may encounter a similar situation.

1) Sometimes we overlook the smallest things. With that being said, even though I had not created it, there was a robots.txt in my root directory that was overriding wordpress and my functions.php. A simple delete took care of that.

2)If you use Google XML Sitemap plugin, make sure that the option that states it will automatically index your sitemap to the virtual robots.txt is turned off.

I appreciate the help that everyone gave me. Here is spending several hours trying to figure out something I could have avoided by first looking at the simple things.

Aitor, I have that setting unchecked. By that setting, it should have my robots.txt updated to the proper setting but it doesn’t. I just got home. I’m gonna do a bit a digging and see if I can figure it out.

I went to functions.php and I found what I believe to be robots.txt. I could be wrong, but does anyone know if this looks right?

function do_robots() {
header( ‘Content-Type: text/plain; charset=utf-8’ );

/**
* Fires when displaying the robots.txt file.
*
* @since 2.1.0
*/
do_action( ‘do_robotstxt’ );

$output = “User-agent: *\n”;
$public = get_option( ‘blog_public’ );
if ( ‘0’ == $public ) {
$output .= “Disallow: /\n”;
} else {
$site_url = parse_url( site_url() );
$path = ( !empty( $site_url[‘path’] ) ) ? $site_url[‘path’] : ”;
$output .= “Disallow: $path/wp-admin/\n”;

I will definitely check for another plugin. I am about to go have dinner with the family, but will return and try to find where the robots.txt is. Is there any suggestions on where I would be able to find it?