Prevent indexing of internal files

  • Resolved CGItaly s.r.l.

    (@consultingroupitaly)


    6 years, 4 months ago

    These URL appears in webmaster tools:

    /xmlrpc.php?rsd
/wp-includes/wlwmanifest.xml
/wp-admin/admin-ajax.php

    Could you please add an options in the plug-in to automatically add this HTTP header to prevent indexing of these 3 files? X-Robots-Tag: noindex

    There are some external sites linking these internal files directly and I cannot remove the links.

Viewing 3 replies - 1 through 3 (of 3 total)
  • Sa?a

    (@stodorovic)

    WP core already sends this header for admin-ajax.php. You can look this line in admin-ajax.php#L38.

    Also, admin-ajax.php and xmlrpc.php return 40x HTTP status codes if request isn’t valid. In this case, sending X-Robots-Tag doesn’t make sense.
    File /wp-includes/wlwmanifest.xml is static and it should be handled by server rules (eg. Apache Module mod_headers ).

    So, it’s something which isn’t related to Yoast SEO plugin. You could remove links from your pages with following code (to reduce number of links to xmlrpc.php and wlwmanifest.xml):

    remove_action( 'wp_head', 'wp_shortlink_wp_head' );
remove_action( 'wp_head', 'rsd_link' );
remove_action( 'wp_head', 'wlwmanifest_link' );

    More details about these functions: wp_shortlink_wp_head, rsd_link and wlwmanifest_link.

    Thread Starter CGItaly s.r.l.

    (@consultingroupitaly)

    1) Yoy can see this: if ( empty( $_REQUEST['action'] ) ) wp_die( '0', 400 );
    so it will never reach: @header( 'X-Robots-Tag: noindex' );

    2) Having pages with 40x HTTP status decrease SEO so it is important to avoid it.
    Here it is stated that noindex is fine for 404 page (not sure for 400): https://forum.webflow.com/t/best-practice-for-disallow-or-noindex-404-page/24135

    But probably it is better to use a 200 HTTP status with X-Robots-Tag: noindex

    3) You are right for wlwmanifest.xml, I hadn’t noticed it was static.

    PS: I don’t want to remove internal links, also it would be useless since there are external domain (not controlled by us) that link them.
    PS2: Yoast SEO should improve SEO in general, so I think it is related.

    Sa?a

    (@stodorovic)

    It’s AJAX request (not HTML page) and it should return different HTTP status codes (depends on action, is user logged in, etc). Same for xmlrpc.php.
    Anyway, it’s something which could be fixed only in WP core (and it shouldn’t be part on any plugin) and you can create new ticket on https://make.www.ads-software.com/core/reports/ if you see issues related to admin-ajax.php and xmlrpc.php.
    If you want to always set X-Robots then you could try to do it in .htaccess via server rules.

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘Prevent indexing of internal files’ is closed to new replies.