Viewing 6 replies - 16 through 21 (of 21 total)
  • Thread Starter Nikonn

    (@nikonn)

    Okay!

    Thread Starter Nikonn

    (@nikonn)

    Jeff, there’s a question. I don’t know about you, but in my case, I have to create a robot.txt and prescribe rules for different bots in it, including separately for Yandex, because he considers himself particularly gifted. The bottom line is that I have set the rules that are prescribed in your plugin settings for only one bot,

    User-agent: *
    Disallow: /*blackhole
    Disallow: /?blackhole

    and I didn’t specify these rules for Yandex. Can this lead to the consequences that my sites are currently experiencing? Do I need to specify the rules for configuring your plugin for Yandex bots as well? Maybe this will fix the problem!?

    Now my robots txt file looks like this:

    User-agent: *
    Disallow: /*blackhole
    Disallow: /?blackhole
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /trackback
    Disallow: */trackback
    Disallow: */*/trackback
    Disallow: */*/feed/*/
    Disallow: */feed
    Disallow: /tag
    Disallow: /ads
    
    User-agent: Yandex
    Disallow:
    Clean-param: blackhole /
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /trackback
    Disallow: */trackback
    Disallow: */*/trackback
    Disallow: */*/feed/*/
    Disallow: */feed
    Disallow: /tag
    Disallow: /xmlrpc.php
    Plugin Author Jeff Starr

    (@specialk)

    Make sure this is added to both sets of rules (Yandex and wildcard *):

    Disallow: /*blackhole
    Disallow: /?blackhole

    Otherwise Yandex will not see it.

    Thread Starter Nikonn

    (@nikonn)

    and what you advised yesterday, to leave?

    Disallow:
    Clean-param: blackhole /
    Plugin Author Jeff Starr

    (@specialk)

    Good question. Looks like a grey area. I am new to the Yandex Clean-param rule, so not sure how/if/when it should be used with Disallow, if it’s required or not, etc. Maybe reach out to Yandex (there is a link to contact on the resource you provided previously). Explain the situation and ask for their recommendations. That way you will know for sure what is the proper syntax, etc.

    Thread Starter Nikonn

    (@nikonn)

    All right, Jeff! In principle, I planned to contact them already, since the common sense of a normal person is not able to understand the dark thoughts of Yandex. Thank you for your help!

Viewing 6 replies - 16 through 21 (of 21 total)
  • The topic ‘There is a problem with Yandex’ is closed to new replies.