403 on content pages when crawling site

  • Although I can view all of my content pages (both logged out of admin or logged in), whenever I went to check what the crawlers would see on my site, all of my content pages came back with a 403 response.

    I know that it has something to do with iThemes Security which I have been enjoying using, because when I deactivate the plugin then generate my crawler report again, then the responses are all 200’s.

    Can you explain what I might be doing wrong? I love this tool, but it’s not going to do me any good if the crawlers cannot index my website.

    Thanks!

    https://www.ads-software.com/plugins/better-wp-security/

Viewing 3 replies - 1 through 3 (of 3 total)
  • Thread Starter jlgill

    (@jlgill)

    I’ve done a bit of experimenting and narrowed it down to the setting: Enable HackRepair.com’s blacklist feature. When I turn it off, the Google analysis is able to crawl my entire site properly.

    Haven’t looked at the entire list of hackrepair.com’s blacklist, but apparently it banks good bots. Which isn’t good.

    @the Hack Repair Guy

    Any idea which User Agent entry in the blacklist could cause this issue ?

    dwinden

    Thread Starter jlgill

    (@jlgill)

    I have not been able to narrow it down.

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘403 on content pages when crawling site’ is closed to new replies.

Tags