crawlers crawling many thousands of duplicate pages

  • On a few sites, I’ve seen Googlebot and other crawlers presumably trying to index thousands of copies of a page where I have a calendar widget. The queries look something like this:

    /calendar-page/?id=1113957581&ajaxCalendar=1&mo=6&yr=2024

    The issue is not that it’s crawling through all the possible months and years (although that could be an issue too, if there’s no cap on how far it could get into the past or future); rather, the problem is the id= value which appears to be random. The crawlers end up with many copies of the same page, indexed with different values of id.

    I noticed this simply because it was putting an unusual amount of load on my servers, but it’s also not ideal behavior for the widget. Why is there an id value specified at all? Getting rid of that, if possible, would help a lot. Maybe these links should have rel=nofollow too, so the crawlers won’t bother with them in the first place. Of course, one wouldn’t want to block a crawler’s ability to find a legitimate calendar entry, but surely there’s a better way than to have it scan through every month/year view, regardless. In my case, Googlebot can find all my calendar entries via my sitemap.xml, which is much more efficient.

    In the meantime, I added something like Disallow: /calendar-page/?*ajaxCalendar* to my robots.txt which should help. But I’m curious to see if anyone else has a better solution.

Viewing 2 replies - 1 through 2 (of 2 total)

  • You could add the following filter to fix this problem:

    add_filter('em_widget_calendar_get_args', function( $instance ) {
        $instance['id'] = 1;
        return $instance;
    });

    You could add the above code snipet with the Code Snippets plugin.

    Plugin Author Marcus

    (@msykes)

    We need to evaluate this a little more closely, but essentially I agree with both your points.

    We’ve removed the id= querystring because and we can add it on via JS, that’ll help immediately and will be out in today’s update.

    Additionally, we’re adding a calendar_nav_nofollow parameter you can add to your shortcode. We’ll elaborate further on this in a following release, but you can also make that a default value set to try by hooking into the em_calendar_get_default_search filter, the array key is in there. We’ll look int making that some sort of option in the settings page.

Viewing 2 replies - 1 through 2 (of 2 total)
  • You must be logged in to reply to this topic.