Google not capturing my homepage description, robots.txt problem

  • northstatehillbilly

    (@northstatehillbilly)


    9 years ago

    I’ve spent a few hours learning about this and don’t know how to go any farther. Maybe you can make some suggestions.

    I launched my site a few days ago and this morning I checked for it in Google. The URL shows up at the top of the Google search results with this message: “A description for this result is not available because of this site’s robots.txt – learn more.”

    When I clicked robots.txt, it showed the file with a few lines of code. But I couldn’t find the file anywhere when I went into the file manager. My hosting company also said they couldn’t see the file and created one for me.

    I used the ‘Fetch and render’ tool in Google Webmaster Tools, and asked Google to index the page. It said partially verified. I’m not seeing anything that tells me why I can’t pull up my site in Google with the site description.

    If you do a Google search for my site’s about page, you can see it in the search results with the first sentence or two as an excerpt. But the homepage in the search results only shows the error message.

    I unchecked ‘Block search engines’ in the WordPress settings a few days ago. Someone said that may have blocked them from finding her homepage for a few days, and then finally she could see it.

    But I’m reading other sources that say it should cause any problems.

    Can you please help me with this?

    I’m able to do a search for my post title and pull it up in the Google results. But the homepage is a mystery.

    Thank you.

Viewing 3 replies - 1 through 3 (of 3 total)
  • Digico Paris

    (@digico-paris)

    Hello,

    Normally, WordPress creates the robots.txt itself, if your host allows it.
    But sometimes you have wrongings like that, because your webhost doesn’t allow to rewrite the file after it was done the 1st time …

    Best way to check that, is to register your site on Google Search console:

    https://www.google.com/webmasters/tools/home

    After registration – it will take a few days to work – , you’ll be able to check if Google found problems with any robots.txt file. If you still have problems, you could show that to your hosting company, and kindly ask them to modify, this time with a screenshot to proof it.

    But, in general, it takes some time for Google index to reflect changes. You can also try this command in the meantime, to check if your webpages are – or not – in Google index (type this in Google search):

    site:www.mysite.com

    That way, Google will show you the number of pages it grabbed (or not).

    Hope it helps,

    Thread Starter northstatehillbilly

    (@northstatehillbilly)

    Thank you!

    I did the search for pages and it only found five. The homepage is one of the pages missing.

    I didn’t realize it can take a few days for Google to index my site. Indexing means crawling through each page and registering it so that each one can be found in search results, correct?

    You’d think if most of the others are showing up, the homepage would be the first one it indexes. But that’s the one it’s not pulling in a description for. It shows up first when you type the homepage URL into the search box, but no description.

    I”ll wait a few days and see what happens, though.

    Digico Paris

    (@digico-paris)

    You’re welcome,

    What about robots.txt? (it’s also in Console – it tells if errors found or no robots.txt found)

    Indexing is the big word for “getting my pages in Google”, to put it funny but it’s an art when you have more than 500, 2000, or 200 000 pages like many ecommerce folks.

    Yes, like you say, you’d think homepage is the first to be in google index, but as google crawlers are not totally sync over the planet, it can take some time so that you even use same site:mysite.com and see exact same result accross the globe.

    After, I’m far from a master on that, just check what happens on Google Search Console, try to make sure your posts/pages uses “workable” titles and descriptions in the meantime – it always takes more time than expected. If you’re interested on topic, you can read more on SEO and things that makes Google algorithm ranking system.

    Good luck,

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘Google not capturing my homepage description, robots.txt problem’ is closed to new replies.