Should googleusercontent be whitelisted by default?

  • Resolved kcwebguy

    (@kcwebguy)


    3 years, 4 months ago

    See below…

    googleusercontent is a legitimate bot that is being blocked. So I whitelisted them by adding “, googleusercontent” to my whitelist. However, they are still being blocked.

    1) Should they be added as a built-in white-listed crawler in the next version of the plugin?

    2) Why are they still being blocked even though they have been whitelisted?

    Note that all caching has been cleared and the cache is not the issue.

    *****************

    2021/11/07 @ 12:10:10 pm

    Request URI: /?blackhole=26d12725e1
    IP Address: 34.140.154.199
    Host Name: 199.154.140.34.bc.googleusercontent.com
    User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:52.0) Gecko/20100101 Firefox/52.0

    Whois Lookup:

    ARIN WHOIS data and services are subject to the Terms of Use
    available at: https://www.arin.net/resources/registry/whois/tou/
    If you see inaccuracies in the results, please report at
    https://www.arin.net/resources/registry/whois/inaccuracy_reporting/
    Copyright 1997-2021, American Registry for Internet Numbers, Ltd.

    NetRange: 34.128.0.0 – 34.191.255.255
    CIDR: 34.128.0.0/10
    NetName: GOOGL-2
    NetHandle: NET-34-128-0-0-1
    Parent: NET34 (NET-34-0-0-0-0)
    NetType: Direct Allocation
    OriginAS:
    Organization: Google LLC (GOOGL-2)
    RegDate: 2021-01-08
    Updated: 2021-01-08
    Ref: https://rdap.arin.net/registry/ip/34.128.0.0
    OrgName: Google LLC
    OrgId: GOOGL-2
    Address: 1600 Amphitheatre Parkway
    City: Mountain View
    StateProv: CA
    PostalCode: 94043
    Country: US
    RegDate: 2006-09-29
    Updated: 2019-11-01
    Comment: *** The IP addresses under this Org-ID are in use by Google Cloud customers ***

Viewing 2 replies - 1 through 2 (of 2 total)
  • Plugin Author Jeff Starr

    (@specialk)

    Thanks for reporting this.

    For that request, “googleusercontent” is the host name, not the user agent. The user agent as shown in your report is “Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:52.0) Gecko/20100101 Firefox/52.0”. You can verify that “googleusercontent” is not one of Google’s user agents here. So with that in mind..

    1) “Should they be added as a built-in white-listed crawler in the next version of the plugin?”

    No because see above. The plugin whitelists based on user agent or IP address. There is no whitelist option for host name.

    2) “Why are they still being blocked even though they have been whitelisted?”

    Because see above. The requests do not include “googleusercontent” in the user agent, so nothing is matched.

    To whitelist the reported entity, you can use the user agent or IP address(es).

    Thread Starter kcwebguy

    (@kcwebguy)

    Thanks for the follow-up AND the education. Much appreciated!

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Should googleusercontent be whitelisted by default?’ is closed to new replies.