Bots keep parsing old languages URLs

  • Hi,
    over a year ago I used to include chinese to my language list. Since then I removed it.
    However, googlebot and bingbot still try to crawl non existent URLs ( foo.com/blog/zh/bar ).
    I noticed that if a Transposh language is non existent on one’s blog, the user gets redirected to the default language of the blog.
    Still, I find such crawling attempts useless, and they waste precious server resources.
    Can I safely add
    Disallow: /blog/zh/
    to robots.txt without risking that the original language ( /blog ) and the existing Transposh translations will be affected?

    Thanks in advance

    https://www.ads-software.com/plugins/transposh-translation-filter-for-wordpress/

Viewing 4 replies - 1 through 4 (of 4 total)
  • Plugin Author oferwald

    (@oferwald)

    Hi,

    When the language does not exist the user gets redirected by the wordpress core itself to a page that seems appropriate, this is not a redirect so the search engines keep on hitting those pages

    Solutions,
    1. What you suggested seems ok ??
    2. Make sure you have a rel=canonical
    3. Ignore them

    Good luck,

    Thread Starter flapane

    (@flapane)

    Hopefully, at some point in the time the bots will stop looking for these pages.
    I’ve been using http 410 responses for some pages on my website, and 4 months haven’t been enough for bots to stop requesting them. ??
    Thanks.

    Ofer: Ignore them isn’t a solution. It is a hope we will not be penalized when Google is showing us exactly what they do not like. I’m struggling with a couple of issues I’ve mentioned elsewhere and just need a little help understanding what exactly goes on when. If we purge the translations using the utilities page then where does google get the notion that translations exist if the plug in is not integrated into the sitemaps (I’m using WP SEO by Yoast) and the pages are not in my sitemap. If I disable a translation I end up with a bunch of 404 errors on google webmaster notices. So knowing how to connect to WP SEO – other than the statement about finding the instructions in the code someplace – removing translation engines I do not want and cleaning up google

    I think this is all one task interconnected and needing to be understood a little bit better

    Plugin Author oferwald

    (@oferwald)

    @iaddic

    First – Please start your own topic next time,

    Second – What? your message was not coherent to me, if translations were removed, why would google visit those pages? Google will visit pages it thinks exist (it doesn’t wait for your sitemap), and it only has the 41x results to know it should stop

    Last – feel free to add disallow to your robots.txt, as you wish. I don’t intend to answer Google queries in the near future, I don’t work for them, never have, and am far from in charge of what they are doing.

    YMMV

    Good luck,

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘Bots keep parsing old languages URLs’ is closed to new replies.