Crawl Errors

  • We’re having a BIG issue with crawl errors on a recently published site: https://moasphalt.org

    Google Web Master Tools has recorded 817 404 errors to date (They doubled in the past 2 weeks.)

    There are a few that we are working to correct because they look legit. but the other 600+ seem to be generated by something.

    I am questioning if it is one of the plugins we’re running because they all are formatted the same with slight numerical variations of this:
    2013/05/23/page/2/?thumb_date=2024-05-01
    And they’re linked from this same url (minus the page/2/)

    Has anyone experienced this? I’m running the following plugins:
    (We run all with * on other sites without this issue)
    Better WP Security
    Calendar Category
    *Collapse-O-Matic
    *Easy Columns
    Embed RSS
    *Featured Articles Lite
    *Formidable
    *FullThrottle Calendar
    *Google Analytics for WordPress
    *Google Doc Embedder
    Google Sitemap Plugin
    Link Manager
    Map List Pro
    *MapPress Easy Google Maps
    *NextGEN Gallery
    *Velvet Blues Update URLs

    I suspect FullThrottle Calendar just because the urls with errors have links that look like dates, but I don’t know.
    Any ideas or help is much appreciated!

Viewing 1 replies (of 1 total)

  • You could edit the robots.txt to disallow any directories or pages with excessive errors, You could submit a sitemap of the pages and posts you want google to crawl, as long as none of those error out you should be good.

    I notice you already have a sitemap so including it in your robots.txt could also help with other search engines. Check out, Yoast WordPress SEO. It allows you to automatically create sitemaps of your site, including pages, posts, authors, categories and tags. It also allows you to edit robots.txt and .htaccess from the backend.

    WordPress creates robots.txt dynamically but you can always manually create it in your root directory. I’d recommend including the location of your sitemap in robots.txt as well.

    robots.txt sample:

    # DEFAULT WORDPRESS ARGS
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

#DISALLOW ADDITION
Disallow: /2013/05/23/

# SITEMAP ADDITION
Sitemap: https://moasphalt.org/sitemap_index.xml

Viewing 1 replies (of 1 total)
  • The topic ‘Crawl Errors’ is closed to new replies.