How to let google crawl website

That content only blocks Google from /wp-admin/ and /wp-includes/ not your entire website.

What exactly is Google telling you?

WordPress dashboard > Settings > Reading – check that the box to discourage search engines isn’t ticked.

Install a plugin to automatically create an XML sitemap – there are several good ones to choose from.

Get a free Google Webmaster Tools account, submit your sitemap, and use it to figure out how well Google is indexing your site, and whether it encounters any problems crawling your pages.

THanks guys!

Whenever I use the fetch as google tool in the Google Search Console it says that my main webpage is blocked. I tried to submit a sitemap and got an error message saying that the URL is blocked by robots.txt.

Ah, it’s a bit of a misnomer. ??

You haven’t blocked Google from accessing your site, but you have blocked it from being able to render a few things, which is kind of bad.

See https://yoast.com/dont-block-your-css-and-js-files/ for lots more words on the subject. ??

I tried doing what everything in that link said but it still didn’t work.

my robots.txt file reads

User-agent: *
Disallow: /wp-includes/

and google is saying

# Added by Google
# Modify or delete lines below to allow robots to crawl your site.
# Block all robots
User-agent: *
Disallow: /

The disallow section of what google is saying is in my robots.txt file is highlighted red with a little red circle with a white line through it next to it. BUT, it is also saying that there are no errors or warnings and it is still blocked.

You have to tell Google to fetch it again.

Also, try this:

User-agent: *
Disallow: /wp-login.php

(wp-includes still has things necessary for properly rendering the site)

OK. I feel like I have literally tried everything now! LOL.

below is how my robots.txt file reads

# Added by Google
# Modify or delete lines below to allow robots to crawl your site.
# Block all robots
User-agent: *
Disallow: /

The robots.txt tester in google is showing the same text, still with the disallow section highlighted in red with a red circle with a dash trough it next to it. Still says there are no errors and warnings.

I tried changing it to Disallow: /wp-login.php and still got the same result.

My robots.txt file can be found here

I am using the KB robots.Txt editor plugin.

For some reason whenever I make changes to the text that is in the file it doesn’t actually change it?

Did you tell Google to fetch it again, as I mentioned earlier?

Right next to the error, you should see something like “Fetch as Google” which will direct Google to re-fetch it. If you haven’t done that, that explains why Google is still displaying the old one.

I’ve done this plenty of times, but don’t have a warnings on any of my sites to walk you through a specific example. So, if the above doesn’t work, I recommend reaching out at https://productforums.google.com/forum/#!forum/webmasters next.

Yes, I do that each time I make changes it comes up with the same message.

Ok, I recommend asking at https://productforums.google.com/forum/#!forum/webmasters so Google’s developers and support community can help you with this.