@alivanrental
We usually recommend the file to look like this:
User-Agent: *
Disallow: /wp_admin/
Allow: /wp-admin/admin-ajax.php
This would allow GoogleBot to crawl all the links on your site and better understand how the pages are viewed and displayed. This guide explains more:?https://yoast.com/ultimate-guide-robots-txt/. We have additional recommendations on robots.txt here:?https://yoast.com/wordpress-robots-txt-example/.
In addition, we suggest the following actions:
1. Clear all your caching from your theme, plugin, server, CDN like CloudFlare or browser. If you are not sure how to clear caching from theme/plugin please speak to those authors. If you want to clear cache from the server, please speak to your host provider. To clear the cache from a browser use this guide:?https://kb.yoast.com/kb/how-to-clear-my-browsers-cache/.
2. Delete your sitemap in Google. Doing so will not hurt the SEO of the site. This guide explains more (scroll to the bottom):?https://support.google.com/webmasters/answer/183669?hl=en.
3. Submit the sitemap again. Be sure to submit only sitemap_index.xml and nothing else. This guide explains more:?https://kb.yoast.com/kb/submit-sitemap-search-engines/.
4. Monitor the results to see if the issue is resolved. It can take some time before Google can report on the crawled URLs after you re-submit your sitemap, especially for?large sites?it takes some time. Google Search Console reports?are not real-time.
We hope this helps!
-
This reply was modified 4 years ago by
Suwash.