Follow Up to Disappearing Images Thread

[steveraven]

Hi guys,

Dunno if this means anything, but I just checked my robots.text file in Search Console and the following codes had been added – probably by a plugin – but I have no clue which one:

Normal Yoast Sitemaps, but then –
# BEGIN WBCPBlocker

SetEnvIfNoCase User-Agent “AhrefsBot” block_bot

SetEnvIfNoCase User-Agent “MJ12bot” block_bot

SetEnvIfNoCase User-Agent “Rogerbot” block_bot

SetEnvIfNoCase User-Agent “SemrushBot” block_bot

SetEnvIfNoCase User-Agent “ia_archiver” block_bot

SetEnvIfNoCase User-Agent “ScoutJet” block_bot

SetEnvIfNoCase User-Agent “sistrix” block_bot

SetEnvIfNoCase User-Agent “SearchmetricsBot” block_bot

SetEnvIfNoCase User-Agent “SEOkicks-Robot” block_bot

SetEnvIfNoCase User-Agent “Lipperhey Spider” block_bot

SetEnvIfNoCase User-Agent “Exabot” block_bot

SetEnvIfNoCase User-Agent “NCBot” block_bot

SetEnvIfNoCase User-Agent “BacklinkCrawler” block_bot

SetEnvIfNoCase User-Agent “meanpathbot” block_bot

SetEnvIfNoCase User-Agent “PagesInventory” block_bot

SetEnvIfNoCase User-Agent “Aboundexbot” block_bot

SetEnvIfNoCase User-Agent “spbot” block_bot

SetEnvIfNoCase User-Agent “linkdexbot” block_bot

SetEnvIfNoCase User-Agent “Gigabot” block_bot

SetEnvIfNoCase User-Agent “dotbot” block_bot

SetEnvIfNoCase User-Agent “Nutch” block_bot

SetEnvIfNoCase User-Agent “BLEXBot” block_bot

SetEnvIfNoCase User-Agent “Ezooms” block_bot

SetEnvIfNoCase User-Agent “Majestic-12” block_bot

SetEnvIfNoCase User-Agent “Majestic-SEO” block_bot

SetEnvIfNoCase User-Agent “DSearch” block_bot

SetEnvIfNoCase User-Agent “BlekkoBot” block_bot

SetEnvIfNoCase User-Agent “NerdyBot” block_bot

SetEnvIfNoCase User-Agent “JamesBOT” block_bot

SetEnvIfNoCase User-Agent “monitorbacklinks” block_bot

<Limit GET POST HEAD>

Order Allow,Deny

Allow from all

Deny from env=block_bot

</Limit>

# END WBCPBlocker

I know for a fact that this stuff was NOT there whilst I was playing around with resubmitting sitemaps to get my images working during the last week.

So where has this extra code come from?

And what exactly is ‘WBCPBlocker’?

Because it looks like that might be the source of the problem.

Thanks for any info!

[Michael Ti?a]

Can you determine if this was being caused by the Yoast SEO for WordPress plugin?

We do know that the robots.txt file can be edited with the Yoast SEO plugin, but it shouldn’t add unexpected code to the file itself. You can refer to these articles for more information on the robots.txt file and the default that is set with the Yoast plugin:

https://kb.yoast.com/kb/how-to-edit-robots-txt-through-yoast-seo/
https://yoast.com/ultimate-guide-robots-txt/
https://yoast.com/wordpress-robots-txt-example/

It might be a plugin or theme conflict. Can you try and gather as much information for us as possible? Please perform the following:

1. Make sure the issue doesn’t persist when Yoast SEO is disabled.
2. Check for conflicts.
3. Check for JavaScript errors with your console.

If you find any JavaScript errors related to Yoast SEO or if there is a conflict with a plugin or a theme, you can create a new GitHub issue for our developers. Please report the issue to a third party developer as well.

If you didn’t find any conflicts or errors, we think the issue is specific to your site. We’d need to investigate further but are unable to do so on these forums. You can purchase Yoast SEO Premium and receive our Premium email support and we can help you further.

[steveraven]

Yes, I reckon that the rogue code was added by ‘Broken Link Checker’ which I only enabled for a couple of days to check my links.

It’s been removed now.

Back to the follow-up thing – I removed all of the sitemaps from my robots.txt and made it so that it just read:

User-agent: *
Disallow:

and this morning when I did a ‘site:mysite.com’ google search in images, I found that there were thousands of images, all linking back to my website – and yet they’re still not showing up in the ‘images indexed’ column.

I suppose this is Googles way of saying that they were wrong all along, but we’ll still drag it out as long as we can.

[Michael Ti?a]

If you’ve set up the correct crawl directives for your robots.txt file, then this would mean that your website can be crawled as expected.

Can you confirm you are getting a big disparity between the submitted and actual indexed images in Google Search Console and did this just recently occur?

[steveraven]

Hi Michael,

As you can see, I put exactly what I’m using in robots.txt above, and it’s still the same.

Doing a site:mysite.com images search shows loads of images from my website, yet in Google Search Console, the images indexed are 4 Indexed, from 292 images submitted.

I’ve tried disabling and re-enabling my CDN which had no effect, and even trying out an image sitemap, but no joy again, and I’ve run out of ideas what to do now.

The images stopped being indexed last weekend.

[Md Mazedul Islam Khan]

Can you please share your sitemap URL here with us so that we can take a look at it before we can confirm anything?

[steveraven]

Sure, its:

https forward slashes midlandsmaidens-escorts (dot) com/post-sitemap.xml
https forward slashes midlandsmaidens-escorts (dot) com/page-sitemap.xml
https forward slashes midlandsmaidens-escorts (dot) com/portfolio-sitemap.xml
https forward slashes midlandsmaidens-escorts (dot) com/staff-sitemap.xml
https forward slashes midlandsmaidens-escorts (dot) com/testimonials-sitemap.xml
https forward slashes midlandsmaidens-escorts (dot) com/wlshowcase-sitemap.xml
https forward slashes midlandsmaidens-escorts (dot) com/category-sitemap.xml
https forward slashes midlandsmaidens-escorts (dot) com/post_tag-sitemap.xml
https forward slashes midlandsmaidens-escorts (dot) com/portfolio_tag-sitemap.xml
https forward slashes midlandsmaidens-escorts (dot) com/portfolio_category-sitemap.xml
https forward slashes midlandsmaidens-escorts (dot) com/staff_tag-sitemap.xml
https forward slashes midlandsmaidens-escorts (dot) com/staff_category-sitemap.xml
https forward slashes midlandsmaidens-escorts (dot) com/testimonials_category-sitemap.xml
https forward slashes midlandsmaidens-escorts (dot) com/post_series-sitemap.xml

…as set up in search console.

Excuse all the ‘forward slashes’ and ( dot) com bits, as it wouldn’t let me post a load of links.

My robots.txt is

User-agent: *
Disallow:

I’ve tried it both WITH the full sitemap in robots.txt, and without it (like above) but it doesn’t make any difference really – images won’t index.

• This reply was modified 6 years, 1 month ago by steveraven.
• This reply was modified 6 years, 1 month ago by steveraven.

[steveraven]

Oh, and the actual sitemap index is at https://www.midlandsmaidens-escorts.com/sitemap_index.xml

Might have been easier if I’d thought of doing that before ??

• This reply was modified 6 years, 1 month ago by steveraven.

[Michael Ti?a]

We checked your /sitemap_index.xml and robots.txt file and we see nothing that should be the cause that the images would stop indexing. If you are able to find your images in a SITE: search but are getting an inaccurate report from Google Search Console, you may want to reach out to Google directly — https://productforums.google.com/forum/#!forum/webmasters

[Jerlyn]

Closed. No further questions.