Do we even need robots.txt nowadays?
-
So after reading this post it made me wonder whether I even should use a robots.txt.
It seems Mr. de Valk is suggesting there isn’t anything on a wordpress install that needs to be disallowed and he doesn’t recommend adding the sitemap either.
Does anyone here have anymore insights (know reasons why you should have a robots.txt) or do you agree that the robots.txt is very much optional at this point?
- The topic ‘Do we even need robots.txt nowadays?’ is closed to new replies.
