Discourage Search Engines help

  • Resolved marcd77

    (@marcd77)


    10 years, 10 months ago

    Hi,

    I’ve recently taken an old HTML website that was ranking high in google for a specific search and created a copy of the website (with a few updates) in a wordpress sub folder. By default the Discourage search engines …. box was ticked during development. To make the site live I deleted the original html files and change the setting in General -> Site address url, then moved my index.php file to the main root of the website. This seemed to work great.
    THE PROBLEM
    2 days later my client contacted me to say that the google ranking had dropped and now instead of a meta description I get

    A description for this result is not available because of this site’s robots.txt

    Line. The first thing I did was go to the settings Reading and untick the box and save the changes. However 1 day later this hasn’t resolved the problem.

    When I click to the robots.txt file, which I understand WordPress makes virtually, I see the following

    User-agent: *
    Disallow: /wordpress/wp-admin/
    Disallow: /wordpress/wp-includes/

    My questions are … should I create my own and what should the content be? From looking around on the internet I think it should be

    User-agent: *
    Allow: /wordpress/
    Disallow: /wordpress/wp-admin/
    Disallow: /wordpress/wp-includes/

    but I’m not sure

    When should I expect google to re-index my website?

    Is there a way to speed this up?

    and the big question …
    The site used to rank high on searches, because of my error will it regain it’s high ranking immediately ? or if not how long to get that ranking back?

    I’d really appreciate any advice,

    thanks
    Marc

Viewing 5 replies - 1 through 5 (of 5 total)

  • You have to first fix the error and check that search engines can crawl the webpage . Goto webmaster tools >> crawl >> blocked urls . Put your urls and robots txt there to check .

    Secondly go to webmaster tools >> crawl >> fetch as google . Put your address and fetch it and submit for index and linked pages .

    It would take 2 to 3 days to return to normal .

    Thread Starter marcd77

    (@marcd77)

    Thanks for the response. I’ve done some more google searching and created the following robots.txt file that i’ve uploaded to the root directory.

    User-agent: *
    Allow: /
    Disallow: /wordpress/wp-admin
    Disallow: /wordpress/wp-includes
    Allow: /wordpress

    User-agent: Mediapartners-Google
    Allow: /
    Allow: /wordpress

    User-agent: Adsbot-Google
    Allow: /
    Allow: /wordpress

    User-agent: Googlebot-Image
    Allow: /
    Allow: /wordpress

    User-agent: Googlebot-Mobile
    Allow: /
    Allow: /wordpress

    Sitemap: https://www.domain.com/sitemap

    Will this help in with my problem? At present I don’t have control over the analytics for the website, will this stop me from using webmaster tools with this site?

    I’m not too concerned about past analytic data so should i create a new account with new tags, enabling me to use the webmaster tools, invoking a site crawl?

    For robots.txt use this one .

    User-Agent: *
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes/
    Disallow: /readme.html
    Disallow: /drafts/
    Disallow: /refer/
    Disallow: */?replytocom=
    Sitemap: https://www.domain.com/sitemap

    Secondly you can use webmaster tools instantly . No need for past data or analytics . Just verify the site and start using it in real time

    Do as above, but,
    If it was me, I would also put all the .html files back where they were and continue to build the WP site.

    The google ranking was built on those .html files over time, delete them and what have you got to keep your rank with?

    Just my 2 cents…

    Thread Starter marcd77

    (@marcd77)

    Hi all,

    Thanks for your advice it has helped greatly. My website resumed it’s healthy position in the google rankings yesterday, which meant a total down time of 1.5 days.

    I have taken the following steps, I am not sure if 1 in particular or all these actions helped contribute.

    1. unticked the wordpress setting regards discourage search engine.
    2. 1 day later created the robots.txt file. and uploaded original HTML files even though I have 301 redirects so these pages would never show up. created a sitemap page with simple sitemap in wordpress.
    2. today, set up webmaster xml sitemap and submitted.

    thanks again

Viewing 5 replies - 1 through 5 (of 5 total)
  • The topic ‘Discourage Search Engines help’ is closed to new replies.