Google Keeps Crawling "/feed" URLs That Don't Exist

  • Resolved Nick M.

    (@nick-m)


    9 years, 2 months ago

    As per the Google Webmaster error report, over 1000 URLs with extension “feed” have been created from my gallery pages (every individual image) that lead to non-existent pages (not found errors 404) ever since. Incidentally, that started happening after I activated “XML Sitemap” module from the “All In One SEO Pack” plugin (I recently changed from “Google XML Sitemaps” plugin to AIOSEOP).

    Should I disable “Galleries” in “Post Types” and left enabled only “Posts” and “Pages” in “XML Sitemap” settings?

    https://www.ads-software.com/plugins/all-in-one-seo-pack/

Viewing 8 replies - 1 through 8 (of 8 total)
  • Plugin Support Steve M

    (@wpsmort)

    Check your XML sitemap and see if any of those URLs appear in the sitemap, if they don’t then this is not caused by your XML sitemap. If Google has access to those URLs then it will crawl them, you can block access to these URLs in the robots.txt file.

    Thread Starter Nick M.

    (@nick-m)

    Thank you!

    Thread Starter Nick M.

    (@nick-m)

    Unfortunately, the number of URLs that lead to non-existent pages only slowdown in trend growth. I am not sure how to block access to more than 1000 non-existent pages. (I am not sure if this is right solution.)

    I have disabled “Galleries” in “Post Types” and left enabled only “Posts” and “Pages” in “XML Sitemap” settings and the number of generated errors decreased from 197 (01/08 – 01/10) to 12 (01/10 – 01/12).

    As can be seen from the graph HERE, the number of errors increased drastically when I changed the sitemap generator (01/04). Moreover, a large number of those URLs are linked from the sitemap.

    I am hoping you can further advise me.

    Plugin Support Steve M

    (@wpsmort)

    Please provide the URL for your XML Sitemap.

    Thread Starter Nick M.

    (@nick-m)

    Here you are. Thank you.

    Plugin Support Steve M

    (@wpsmort)

    I checked your XML sitemap and there are no URLs present with the word feed in them so you should be fine as far as our XML sitemap goes. Again, to repeat my earlier response, if Google can crawl those URLs and it wants to index them then it will. The only way to stop it crawling those URLs is via the robots.txt file or your .htaccess file.

    Thread Starter Nick M.

    (@nick-m)

    Okay. Maybe at some point Google will stop indexing those false pages. Am I wrong?

    Plugin Support Steve M

    (@wpsmort)

    That’s entirely up to Google.

Viewing 8 replies - 1 through 8 (of 8 total)
  • The topic ‘Google Keeps Crawling "/feed" URLs That Don't Exist’ is closed to new replies.