[NSFW] Advice Please?

Hello @deeveearr

Thanks for reaching out about your Google Search Console report. Can you provide some URLs reported as indexed but are not part of the 350-ish pages you have on your site?

We look forward to hearing from you.

Hi @maybellyne

I’ve just taken a screenshot of my Google Search Console (the PAGES section) which can be viewed HERE.

The Various ‘pages’ which are not getting indexed are:

Page with redirect

Alternate page with proper canonical tag

Excluded by ‘no-index’ tag

Soft 404

Not found (404)

…etc etc and the list goes on.

This only started happening a few months ago – did Google downgrade their search console software?

We don’t have evidence to confirm if Google changed its crawling algorithm in recent times, however, those errors are very common depending upon the site structure and the moment Google crawled your entire website.

To see if this helps to some extend, we suggest the following actions:

1. Clear all your caching from your theme, plugin, server, CDN like CloudFlare or browser. If you are not sure how to clear caching from theme/plugin please speak to those authors. If you want to clear cache from the server, please speak to your host provider. To clear the cache from a browser use this guide: https://yoast.com/help/submit-sitemap-search-engines/

2. Delete your sitemap in Google. Doing so will not hurt the SEO of the site. This guide explains more (scroll to the bottom): https://support.google.com/webmasters/answer/183669?hl=en.

3. Submit the sitemap again. Be sure to submit only sitemap_index.xml and nothing else. This guide explains more: 

4. Monitor the results to see if the issue is resolved. It can take some time before Google can report on the crawled URLs after you re-submit your sitemap, especially for?large sites?like yours. Google Search Console reports?are not real-time.

Hi @suascat_wp

Ok, so I did all of the above steps EXCEPT for submitting the sitemap again.

I’ll submit the sitemap again later tonight or tomorrow morning.

Hey @deeveearr,

Okay, do let us know how it goes after resubmitting later.

Right then guys, site-map re-submitted, but it looks the same as before actually.

I did notice a drop-down menu at the top of the ‘PAGES’ section of the search report though, and changed it from ‘all known pages’ to ‘all submitted pages’, which makes it look a little more realistic.

On this new setting, there are 298 pages indexed, with 293 pages NOT indexed.

In this new section, there are just TWO-page types that seem to have problems instead of the 10+ page types mentioned above –

1. Crawled currently not indexed
2. Dicovered currently not indexed

both of which are claiming that ‘Google Systems’ is the ‘source’.

So I’ll just re-verify these two errors.

It was a lot easier using the old Webmaster Tools console!

@jeroenrotty

Depending upon how large your website is and how often your website gets crawled by Google, the Google Search Console reports would be updated with new information.

In the meantime, if you could provide some example URLs under a couple of those reports, we might help you verify if there are any reasons Google cannot index them or simply if those reports are false-positive and still wait for some more time to see the stable statistics.

Hi @suascat_wp

Ok, we’ll go for a couple of url’s listed under the ‘Google Systems’ reports.

The first one HERE is an example of ‘Crawled – Currently not indexed’.

The second one HERE is an example of ‘Discovered – currently not indexed’.

The first one?HERE?is an example of ‘Crawled – Currently not indexed’.

This notification suggests that there isn’t a problem on your site but that Google just took some time to fully crawl and index your site, which is perfectly normal.

I inspected the given page and didn’t notice any blocking issue with indexing.

Please remember that in all cases besides time, Google decides how and when the results will be displayed.

The second one?HERE?is an example of ‘Discovered – currently not indexed’.

Likewise in above case, the given page here is discovered but yet not put in index list.

This means that Google knows about these URLs, but they haven’t crawled (and therefore indexed) them yet. If you’re running a small website with good-quality content, this URL state will automatically resolve after Google crawls the URLs.

You can force Google to crawl a post/page using the “Request Indexing” feature. You can use this feature to force Google to index the post or page a bit faster than the usual crawling of Google.?Please follow the steps here.

Additional Tips:

There are a couple of possible reasons why Google is slow when spidering your site. The first might seem obvious: if Google doesn’t find enough (quality) links pointing to your site, it doesn’t think your site is very important. The?other reasons are technical: Google has too much to crawl on your site, your site is too slow, or it’s encountering too many?errors.

Please read this article.

If this remains the same after submitting the URLs via the Request Indexing after a week, we’d recommend you to start a topic on the Google webmaster forums to see if they can find anything that would make this indexing take so long. You can find these Google forums here:?https://support.google.com/webmasters/community?hl=en.

This thread was marked resolved due to a lack of activity, but you’re always welcome to re-open the topic. Please read this post before opening a new request.