How to whitelist an internal web crawler

  • Resolved fakename

    (@fakename)


    6 years, 11 months ago

    Hi there,

    I was wondering if i could whitelist an internal web crawler so that it can index our WP site. Currently our users are authenticating via shibboleth but when the crawler goes to index the site, it cant authenticate then it can’t index our site. Any thoughts or ideas that I’m not thinking of here?

    Thanks!

Viewing 1 replies (of 1 total)
  • Plugin Author michaelryanmcneill

    (@michaelryanmcneill)

    This isn’t really a plugin question. If you’ve got your site entirely protected by Shibboleth, you can modify your Apache configuration to allow the crawler to access the site. If you’re protecting the entire site using WordPress, then you’d need to either configure a service account for the crawler to use, or you’d need to hook into Core and make changes for that specific IP (I’d highly advise against that). Either way, this is a question that is outside of the scope of the plugin.

Viewing 1 replies (of 1 total)
  • The topic ‘How to whitelist an internal web crawler’ is closed to new replies.