Robots.text by AIOSEO

I suspect you posted 2 different robots.txt contents in your post. I’m just not sure where one starts and the other ends.

Correct would be:

User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/

Sitemap: https://www.mysite/sitemap.xml

The following excludes your page from indexing:

User-agent: *
disallow:

User-agent: *
Disallow: /search
Disallow: /category/
Disallow: /tag/

Thanks, I have deleted the robots.text file in the server and I am using the robots geberate by AIOSEO whicjh is stored in the databse, I have modified it and it looks like this now. Anyway google is bloking all my site, Post are not indexed and if I render them manually to google in the search console either, videos are hosted on my site but not indexed by google either, so I got a big problem to solve, I am on it but I need some help. Google says that they could not determine the prominet video on the page. They are actually videos that I publish as posts in my site but without text content.

`User-agent: *

Allow: /wp-admin/admin-ajax.php

Allow: /allow: /wp-content/uploads/

Disallow: /wp-admin/

Disallow: /disallow: /wp-content/plugins/

Disallow: /disallow: /readme.html

Disallow: /disallow: /refer/

Disallow: /disallow: /tag/ /

Disallow: /disallow: /author/

Disallow: /disallow: /404-error/

Sitemap: https://www.mysite/sitemap.xml

• This reply was modified 2 years, 2 months ago by jmfvecu.

Show a link to the website in question, then it can certainly be evaluated here. Or you can simply ask directly at https://www.ads-software.com/support/plugin/all-in-one-seo-pack/ – there are certainly more people there who can assess this correctly.

Thanks, in the link is the url to my site, i post here some screen shots of the error message in Google search console.

Mysite

search console 1

search console 2

• This reply was modified 2 years, 2 months ago by jmfvecu.

Your robots.txt is formatted a bit awkwardly:

User-agent: *
Allow: /wp-admin/admin-ajax.php
Allow: /allow: /wp-content/uploads/
Disallow: /wp-admin/
Disallow: /disallow: /wp-content/plugins/
Disallow: /disallow: /readme.html
Disallow: /disallow: /refer/
Disallow: /disallow: /tag/ /
Disallow: /disallow: /author/
Disallow: /disallow: /404-error/

Sitemap: https://www.julioflores.site/sitemap.xml

This would be the right way:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-admin/
Disallow: /wp-content/plugins/
Disallow: /readme.html
Disallow: /refer/
Disallow: /tag/ /
Disallow: /author/
Disallow: /404-error/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.julioflores.site/sitemap.xml

If you cannot change this yourself, then I would again recommend that you contact the support of the plugin you are using.

The GSC forum might also be able to help, since the actual problem is a conspicuousness on Google: https://support.google.com/webmasters/community?hl=de

Wordpress support here cannot help you in this regard.

I can change it myself but why you took off the next line in your example?, and why disallow wp admin is two times in the code?

allow: /wp-content/uploads/

• This reply was modified 2 years, 2 months ago by jmfvecu.
• This reply was modified 2 years, 2 months ago by jmfvecu.

I overlooked it, but it doesn’t matter and doesn’t create a syntactical error. The upload directory was already removed in your code. I don’t see any reason for that either. You can also omit the line if your media files are all allowed to be indexed (some exclude this in their project because of PDF files).

Thanks for your help, I have disabled the AIOSEO robots and i have uploaded mine to the root. I want to improve my robots.text cause google is not indexing my posts nor my videos which are hosted in my own site but posted as posts with no text content, the reason in google search console is that

“Google could not determine the prominent video on the page”

and

“No thumbnail URL provided”.

I am not sure how to solve this.

MY ROBOTS LOOKS LIKE THIS NOW

`User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /readme.html
Disallow: /refer/
Disallow: /tag/ /
Disallow: /author/
Disallow: /404-error/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.mysite/sitemap.xml

• This reply was modified 2 years, 2 months ago by jmfvecu.
• This reply was modified 2 years, 2 months ago by jmfvecu.
• This reply was modified 2 years, 2 months ago by jmfvecu.
• This reply was modified 2 years, 2 months ago by jmfvecu.

This has nothing to do with the robots.txt but with the way videos are embedded in your page. In order not to blow up the topic here, I would recommend you (also this time) to ask a question about this in the AIOSEO support forum. They are more familiar with these topics than the general WordPress forum.

I know, I have published in the AIOSEO forum too but nobody answer so far.

AIOSEO Forum