Google will index everything on my blog EXCEPT my posts. Help?

My site is hooked up to Google Webmaster Tools, but unfortunately there’s nothing helpful there.

I have no messages of any kind in the “Messages” section, there are zero errors, and the only “warning” of any kind that I have on the whole site is 8 warnings attached to my post-sitemap (visible at https://www.toyreviewdaily.com/post-sitemap.xml).

The warnings don’t make any sense to me at all, though, as the only description I get is the line “Sitemap contains [8] urls which are blocked by robots.txt. Mar 19, 2012
Value: https://www.reviewstarwars.com/forums/images/smilies/icon_lol.gif”

My robots.txt is set to allow EVERYTHING to be found in searches, that URL isn’t even part of my site, and I can’t find it anywhere within the sitemap. I don’t understand what it is or why it’s there.

Even so, these are only 8 warnings, and 150+ articles are messed up and not being retrieved by Google. I don’t even know if that’s the issue.

Any expert advice, anyone?

I do not see anything blatant on the site at first glance which could be causing it.

By any chance have you obtained any unnatural links (paid, spammy, etc.)?

One thing to to take a look at is many WordPress sites display new blog posts on the front page and then have them fall off as newer ones are published. This sometimes has the effect of having the new articles rank well while they are linked from the homepage, but when they are no longer linked from the homepage they do not rank well any longer.

This can be due to the fact that there is more link juice flowing from the homepage because most links point to a site’s homepage and are not deep links to an article. To counter this you want to get deep links going directly to the article and make sure you are interlinking your newer articles with your old ones.

I have not looked at your link profile or interlinking structure, so this may not be the case, but it is something I have seen happen with WordPress sites in the past.

Thanks for the continuing assistance, revium. I’m not good with these things so I’m sort of flying blind here.

I don’t think it can be an issue of the site not being deep-linked or interlinking. The articles getting the most retrievals from Google had always been the older ones, and Google search had always retrieved the actual articles up until last week. Now it still gets you to the articles, if you just through hoops and click pages that have links to the articles.

However–what constitutes an “unnatural link”? I have affiliate links on my site, as it’s a review website and includes a “Where to Buy” section with links to things like Amazon and Ebay (example page: https://www.toyreviewdaily.com/2012/03/20/daily-toy-review-80-thunderball-of-the-wrecking-crew-marvel-legends-series-2-variant-2012-action-figure-hasbro/). It’s never caused a problem before, but that’s the ONLY thing I can think of. All of my links are clearly marked and I absolutely never try to mask or mislead anyone. How could I tell if Google was marking a page as “Spammy”?

I would doubt the affiliate links would be the cause of the drop in rankings/traffic because you have lots of unique content on the page besides the links. If the affiliate links were the only thing on the page then it would be more likely they could be the cause.

If you look under the crawl stats in Webmaster Central do you see a drop in the crawl activity to your site recently from Google?

Under Crawl Stats for “pages crawled per day”, it shows that earlier in March Google was crawling over 600 pages per day, and it looks like it’s crawling over right around 400 at the present. I’m not sure what that means exactly, though, since the pages were already indexed?

“Kilobytes downloaded per day” was over 17,000 earlier this month and is now around 9,000. I’m not familiar enough with that to know what it means, though.

“Time spent downloading a page” has been constant for the last few months with no fluctuations.

The only other potential issue I can think of is that I do post the full-text of my reviews on various message boards with links back to my site so that no one is forced to click through for information (example: https://thundercatslair.org/forums/showthread.php?t=8061), and Google search is sometimes retrieving those forum threads now instead of the actual articles from my site, but that’s never been a problem before. Everything on my site should be the canonical version of the content, so I don’t know if posting the information elsewhere could be an issue.

Nothing seems out of the ordinary with your crawl stats. One thing I look at with them is if the amount of pages drops per day while the time spent downloading a page increases then it can be a sign of the site running slow which could cause the search engines to not crawl as often or as many pages, but that does not seem to be an issue with your site.

I think posting the full text of your reviews on the various message boards could definitely be an issue though. Even though you have the canonical tag on your site, there is no canonical tag on the forums pointing to your site to give you credit as the original source so it is very possible that the search engines are crediting them with the content which is why you are seeing your rankings drop.

The search engines are constantly working to improve their logic to identify which sites are the original source of content and unfortunately they do not always get it correct which sounds like it is the case with your content if those forums are out ranking your site.

Personally, I would just post a snippet of the review on the forums and then link back to your article page for the rest of the text. It may seem less useful to a reader of the forums, but it should protect your content better and help the search engines to identify your site as the original source.

Alright, I will definitely try that, revium!

Thank you so much for the help!

I’m still new to how search rankings work and such–if I go back and delete/cut back the articles I’ve posted to other sites and message boards, will Google eventually pick up on that and re-credit the content to my site once it’s not elsewhere? Or once they’ve assigned it to someone else, am I pretty much screwed?

Once Google recrawls a page and sees that the content has changed they should adjust their index accordingly. It is hard to say how it will change things, but if the content only appears on your site then you should have a good chance of ranking above the others.

Hi Dabid,
I had completely same problem on my rybioko.cz . Everything was visible except the posts, although I set everything very cerafully.
E.g. the tag ref rybioko.cz/tag/hosting
was indexed, ranked and diplayed, but post https://rybioko.cz/jine/nejlevnejsi-webhosting was indexed, but not ranked nor displayed by google.
I think I solved it today.
I inserted this code to my wp-config.php:


remove_action( 'wp_head', 'feed_links_extra', 3 ); // Display the links to the extra feeds such as category feeds
remove_action( 'wp_head', 'feed_links', 2 ); // Display the links to the general feeds: Post and Comment Feed
remove_action( 'wp_head', 'rsd_link' ); // Display the link to the Really Simple Discovery service endpoint, EditURI link
remove_action( 'wp_head', 'wlwmanifest_link' ); // Display the link to the Windows Live Writer manifest file.
remove_action( 'wp_head', 'index_rel_link' ); // index link
remove_action( 'wp_head', 'parent_post_rel_link', 10, 0 ); // prev link
remove_action( 'wp_head', 'start_post_rel_link', 10, 0 ); // start link
remove_action( 'wp_head', 'adjacent_posts_rel_link_wp_head' );
remove_action( 'wp_head', 'wp_generator' ); // Display the XHTML generator that is generated on the wp_head hook, WP version
remove_action( 'wp_head', 'adjacent_posts_rel_link', 10, 0 ); // Display relational links for the posts adjacent to the current post.


(Found on https://wpengineer.com/1438/wordpress-header/)

After that, google displayed my posts immediately.
Hope it helps to you too.
Jeseter.

Hey Jeseter–

Thanks so much for the advice! Unfortunately, my coding skills are pretty poor. Is there a specific place I need to insert that code in the wp-config.php file? As soon as I downloaded the file and inserted the code and re-uploaded it, it crashed my blog completely. I was smart enough to save a copy of the original beforehand to reupload and get everything running, but I don’t know how to add those functions without killing my blog?

Thanks so much for the help! I’m getting so, so frustrated as time goes on and things don’t get better…

Sorry, my coding ability is probably not better. I just put it at the end of the file and my web works. Try to ask at the mentioned wpengineer blog.