Is my Robots.txt file correct?

  • TheGift73

    (@thegift73)


    12 years ago

    Hi,

    I have recently updated my Robots.txt file and was wondering if I am blocking items that I should. Also, the way I currently have it, is it doing any harm to my site ranking/indexing wise?

    Regards,

    Richard

    My site is: https://techfleece.com/

    My robots.txt file reads as:

    User-agent: *

Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content

User-agent: Mediapartners-Google
Allow: /

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Image
Allow: /

User-agent: Googlebot-Mobile
Allow: /

User-Agent: bingbot
Allow: /
Viewing 4 replies - 1 through 4 (of 4 total)
  • Gwythan

    (@kevin-ashbridge)

    That looks like a good WP robots.txt file to me.

    Google will rank you on the quality, freshness and relevancy of your content, plus the social proof around it. That’s the important stuff to concentrate on.

    — Kevin.

    Thread Starter TheGift73

    (@thegift73)

    Hi Kevin,

    That’s good to know. It was mainly as I seem to rank well for Google queries (average 1,400+ a day) but poorly on Bing (6 or 7 hits a day) I was just worried that Bing was being blocked from certain things that it needs in order to index correctly.

    Glad it looks OK to you tho.

    Thanks Kevin.

    –Richard

    Gwythan

    (@kevin-ashbridge)

    Probably fewer of your users use Bing (!)

    The thing to check is if you both rank reasonably equally within both engines for the same keywords – such as ‘both on page 2’. Then at least you know your website is returned consistently in the search results of either engine for your search terms.

    Thread Starter TheGift73

    (@thegift73)

    Hi Kevin,

    I just did a quick check on both search engines for the exact same terms for my most viewed articles. The results are below. The only other thing that I can think of is that I don’t use Tags or Categories (very rarely) on any of my posts. Maybe Bing’s algorithm requires these to be present in order to rank/index correctly?

    Search Term: “Not enough server storage is available to process this command”

    Google – Page 1 at the top
    Bing – Checked up to page 10 and still not showing

    Search Term: “Take Ownership hard drive”

    Google – Page 1 at the top
    Bing – Page 2

    Search Term: “Split MKV files”

    Google – Page 1 at the top
    Bing – Page 1 in the middle

    Search Term: “Convert RAW Files To JPEG”

    Google – Page 1 at the top (no.3)
    Bing – Checked up to page 10 and still not showing

    It’s interesting to see how the various engines check though. There are many articles that rank for a specific term that actually lack the particular relevancy.

    Richard.

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘Is my Robots.txt file correct?’ is closed to new replies.