marcusdeman

Thank you for your help Maybellyne

We did not add noindex directives in the robots.txt. Yoast adds Disallow directives in the robots.txt but will Google still be able to see that these search results pages are set to noindex?

The removal tool only removes these indexed entries temporarily. We are looking for a permanent solution.

I can visit the indexed spam URL and they are displaying an actual page on the website. So no 404 is shown.

If the disallow is put in the robots.txt how does Google determine that the noindex tag is set for these pages as Google is disallowed for crawling these pages.

Google pages state that it is best to remove these pages from not being indexed by using the noindex but not by disallowing them in robots.txt.

https://support.google.com/webmasters/thread/197853493/indexed-though-blocked-by-robots-txt?hl=en
https://support.google.com/webmasters/thread/197435259/how-do-i-get-rid-of-bogus-urls-search-console-found-that-don-t-even-exist-on-my-site?hl=en

In this article Google also states that using robots.txt is not the way to go for removing these spam searches:
https://support.google.com/webmasters/answer/9689846

• This reply was modified 7 months, 2 weeks ago by marcusdeman.

Hi Mushrit,

Thank you for your response. It’s only because of the robots.txt setting that Google did not crawl these links and did not see the “noindex” being applied to. I have removed the disallow statement from the robots.txt to see if Google picks up the “noindex” tag.

In GSC I do see these spam results now as “Indexed, though blocked by robots.txt”

How can this situation be resolved?

Hi Chandelierrr,

Sorry for the late response. I was a bit busy with things and did not have time yet to try your suggestions.

But I managed to do it as of now and this is the response to the curl command:

{“code”:”rest_cannot_access”,”message”:”DRA: Only authenticated users can access the REST API.”,”data”:{“status”:401}}

What would be the next step?

Hi Chandelierrr,

Thanks for your response.

How can I check whether the abovementioned REST HTTP methods are allowed?

id => 35457
module => site-scanner
type => warning
code => scan-failure-client-error
timestamp => 2024-01-29 15:32:30
init_timestamp => 2024-01-29 15:32:28
remote_ip => 10.128.63.162
user_id => [empty string]
url => wp-cron
memory_current => 15618616
memory_peak => 17432288
data => Array
results => Object WP_Error
errors => Array
site_verification_failed.connection_error => Array
0 => Unable to determine if the scan target is allowed: Target site returned invalid response. Do you have a plugin that disables access to the REST API? For instance, Coming Soon Page, W3 Total Cache, Password Protected, Members, Profile Builder, Disable Login, Clearfy, Disable REST API, REST API Toolbox, Defender Security, Shield Security, or Force Login? Try turning off that feature and using "Restricted REST API Access" in iThemes Security instead.
error_data => Array
site_verification_failed.connection_error => Array
status => [integer] 401
url => https://website.com
cached => [boolean] false

I adjusted the url to not show our website domain.

• This reply was modified 9 months, 3 weeks ago by marcusdeman.

The URL of the page is:
https://website.com/de/plattform/ueberblick/

But the url with UTM parameters in my chrome browser could be:

https://website.com/de/plattform/ueberblick/?utm_campaign=wiki&utm_source=website

And these are the html codes I see on the page in regards to hreflang:

<link rel="alternate" hreflang="en"  />
<link rel="alternate" hreflang="de"  />
<link rel="alternate" hreflang="x-default"  />

I am also using the Chrome hreflang tag checker widget and when I test it out on the page it will also state that the hreflang url’s found in the source does not match up with the URL address that I am visiting.

• This reply was modified 10 months ago by marcusdeman.
• This reply was modified 10 months ago by marcusdeman.
• This reply was modified 10 months ago by marcusdeman.

Hi Maybellyne,

Here is my response to your questions.

I did another SEMRush audit and now the issue of hreflangs and cannonicals not matching is gone. Pretty peculiar.

However, SEMRush still states that there is no self-referencing hreflang found in these pages with UTM parameters. How/where is this self-referencing hreflang tags set?

Not sure if it is still relevant to the new case but we haven’t configured / are using the Advanced: URL cleanup options in Yoast.

Set as resolved.

Hi Cornel,

So it seems that the screen options were indeed hard set in the theme so one of the dev’s had to set for my user the screen options to some designated number. Unfortunately, when I change the screen options I will have to bug him again lol.

This error has nothing to do with search analytics and all with the WP theme as used.

This topic can be set as resolved.

Thanks for all your help and considering what possibilities would there be that hinders me from setting the screen options myself.

Hi Ben,

So I have checked the log file again of iThemes / SolidWP.

It tries to reach remote ip “10.128.63.162”

When pinging the host from my own computer there is no response. Is the IP address still correct?

Hi Cornel,

Thanks for checking up!

We are using some proprietary plugins and a self-developed theme so I don’t think it will help you in your research. However, we are now checking up with the developer of the theme why this “mwtsa_entries_per_page” key is missing.

Will report back when I have more info on the matter.

• This reply was modified 10 months ago by marcusdeman.

Hi Maybellyne,

The pages are not category pages.

What I noticed now based on the Semrush audit is that it views the page with UTM parameters (because we have several of these URL links with them throughout the website).

The Semrush bot crawls the page and sees that the URL is different than the canonical URL. Is that possible?

Hi Cornel,

Yes after changing the screen options panel I still see the default 20 value.