Google-Bot and other search engine ID's for whitelisting

Viewing 3 replies - 1 through 3 (of 3 total)

  • +1

    Googlebot should never, ever, ever, ever, ever be locked out for any reason.

    One of the configurations tends to block the AdWords bot.

    Security > Tweaks > Filter Suspicious Query Strings

    When enabled, a rule gets added to the htaccess file with a plethora of conditions. One in particular is problematic…

    RewriteCond %{QUERY_STRING} ^.*(\[|\]|\(|\)|<|>|ê|”|;|\?|\*|=$).* [NC,OR]

    It’s common practice with AdWords to dynamically add the search term keywords as a parameter to the URL. The problem is that the Bot doesn’t use a keyword when checking to make sure that the URL is valid, so it ends up with something like this…

    https://domain/path/to/page/?kw=

    Because the kw parameter is left empty by the bot, the last part of the expression is met and the rule is fired….

    RewriteRule ^(.*)$ – [F,L]

    As a result, Google receives a 403 forbidden, and the campaign is turned off.

    Also, it’s worth noting that the behavior described above won’t be apparent while you’re logged into wordpress, due to another condition for the same rule…

    RewriteCond %{HTTP_COOKIE} !^.*wordpress_logged_in_.*$

    That little nugget had me scratching my head when Google was receiving 403’s and I wasn’t.

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘Google-Bot and other search engine ID's for whitelisting’ is closed to new replies.