URL not able to be indexed

Hi @robertkun

the last version has no changes that could change the behavior regarding SEO.

Here you can see the Change-Log: https://www.ads-software.com/plugins/specific-content-for-mobile/#developers

The critical changes that could have impacted the SEO were done on version 0.1.4 that was published 7 months ago.

If a page is private, the only way Google can be aware of that page is that the page is included in a site map, or linked somewhere in the website.
If no sitemap has https://shroomstocks.nl/waarom_investeren_in_psychedelica-mobile/ and nowhere you have a link that points to https://shroomstocks.nl/waarom_investeren_in_psychedelica-mobile/ Google will not know that page exists.

Now it looks the plugin is disabled, so I can’t check.

When it’s enabled, you should:
– Check that the sitemap https://shroomstocks.nl/page-sitemap.xml doesn’t include
https://shroomstocks.nl/waarom_investeren_in_psychedelica-mobile/
– Go to https://shroomstocks.nl/waarom_investeren_in_psychedelica/ using Google Chromme
– Inspect elements (right-click => Inpsect Elements)
– Search for “nofollow” and “noindex”.
– Click on the Mobile simulation, and refresh the page.
– Search again for “nofollow” and “noindex”.

If you don’t find “nofollow” and “noindex” inspecting elements, and no stiemap includes the page https://shroomstocks.nl/waarom_investeren_in_psychedelica-mobile/, you are not asking Google to don’t index the page, and the problem is something else.
In that case, have a look at the Google Seach Console if there are any errors.
You should first understand if Google doesn’t index the page because an error, or because you are asking Google to don’t index it (not because you want it, but because I the HTML the page has a “noindex”).

If inspecting elements you see a “noindex” in the mobile or desktop version, then we can investigate deeper on that direction.

• This reply was modified 3 years, 10 months ago by Jose.

Hello Jose, thanks for your extensive answer. I downloaded an earlier version first to see if it solves the problem. It doesnt. Google search console still shows the not able to index errors.

I went ahead and inspected both desktop and mobile page sources. The mobile page source shows a noindex whenever its set to status private.

The problem here is that adding a canonical URL pointing to the desktop variant is not possible whenever that page is set to private. Because google does not spot a canonical page on the mobile variant, it does not know which page to index. It tries to index the mobile page, despite it being set to private, and spots the robot noindex tag.

I checked the sitemap, and made sure no mobile URL’s were to be found in here.

—————-

I have one question for you. If we publish a mobile private page you say this is bad for SEO because google will see this as a duplicate. But a canonical URL is used to avoid duplication issues. Why cant we just use a canonical URL, instead of a private page to solve any SEO issues?

• This reply was modified 3 years, 10 months ago by robertkun.

Hi @robertkun

I see you have another problem. The cache of WPRocket is not generating a different cache version for mobile.

Imagine this scenario:
– The cache is cleared or still not generated
– A user visits the page with a mobile device
– The cache is created according to the content served to mobile devices
– All other users will see the mobile content until the cache is expired or deleted, no matter if they have a desktop or a mobile device

So, first, solve that issue. You should have the option to say to WPRocket to generate a separate cache for mobile. As I know WPRocket has that option.

Then, in the backend, check if on the mobile page you have set noindex with Yoast SEO. On the desktop version, it’s ok, but because the cache generated on mobile, you have noindex also on desktop. The problem of noindex is only on the mobile version. I think because the settings of Yoast SEO are noindex on the mobile page, but you can set it to index.

Hello Jose,

1. I made sure WP rocket is catching mobile devices separately, this is the case. No issue found from my side here.

2. I also made sure that on backend i havent checked the Yoast no-index checkbox.

From what i saw setting a page to private (which your plugin forces to do so), wordpress will automatically put a no-index tag on such page. I understand why it does this, but the problem arrises when google suddenly picks up private pages (which i still dont know why or how as they are nowhere to be found in a sitemap).

—-

My possible solution

I know i keep mentioning the canonical and alternate tags, but i do think this is the way to counter above issue. What i did:

1. I disabled your plugin, and made all private pages public.

2. I then added a canonical URL reference on the mobile version of the page, pointing to the desktop page. Google knows which URL is prefered to show on google search, and this will solve content/SEO duplication issues.

3. I also added a alternate URL reference on the desktop version of the page, pointing to the mobile version. Google can now make an easy distinction between desktop, mobile, and serves the pages according to user screensize.

Page example:

https://shroomstocks.nl/hoe-koop-ik-psychedelica-aandelen/

Alternate tag pointing to mobile:

<link rel=”alternate” media=”only screen and (max-width: 640px)” href=”https://shroomstocks.nl/hoe-koop-ik-psychedelica-aandelen-mobiel/”&gt;

Page example:

https://shroomstocks.nl/hoe-koop-ik-psychedelica-aandelen-mobiel/

Canonical tag pointing to desktop:

<link rel=”canonical” href=”https://shroomstocks.nl/hoe-koop-ik-psychedelica-aandelen/”&gt;

—————————-

I checked all pages on google search console. No issues found.

Hi @robertkun

you have surely a problem with the cache, no doubts about it.

I’ve right now inspected elements and seen the same cached version on mobile and desktop. On desktop, I see the comment <!– Specific Content For Mobile replaced the desktop content with the mobile content on this page (Desktop ID: 2; Mobile ID: 905 –>
Specific Content For Mobile does absolutely nothing on desktop, and it adds that comment only on a mobile device.

Be sure to check the page when you are logged out, or with an incognito window, if you are logged in, you probably see two different versions for mobile and desktop, but. if you are logged out you should see what I saw.

First, you should solve the issue with the cache.
If you are sure the options of WP Rocket are already set to create a separate cache for mobile, check if the server is also adding a layer of cache.

Hey Jose, thanks once again for your answer. I cleared and pre-loaded the cache with WP-rocket, then went on incognito chrome, to the above mentioned page.

I inspected the page and searched for terms like “specific” and “mobile”. I cannot find anything related to this when viewing page source, or inspecting elements. Maybe this is solved because i pre-loaded the cache? Do you still see your plugin info on the desktop variant?

Hi @robertkun

I don’t see anymore the comment of Specific Content For Mobile, but now also the mobile version doesn’t have it, and it should. So it looks the same version of cache is served both for desktop and mobile, but the cache was generated visiting the page with a desktop.
If you have an old version WPRocket visits the page simulating a desktop.
Or maybe you haven’t checked the checkbox “Separate cache files for mobile devices” in the settings Cache => Mobile Cache.

I suggest you read here: https://docs.wp-rocket.me/article/708-mobile-caching
And here about mobile preloading: https://wp-rocket.me/blog/wp-rocket-3-5/

Hi @robertkun

do you have news? Do you still need help on this topic?