Disable robots.txt changing by YOAST SEO

Hello Max,

Thanks for reaching out regarding your robots.txt file. If I understand correctly, you use another plugin to manage your robots.txt file and will like to disable to directives added by our plugin.

Since you’re in a multisite environment, that block of text in the file can’t be removed. But there’s a filter to disable the subsites sitemap link from robots.txt. Further information is available here: https://github.com/Yoast/wordpress-seo/pull/18909.

You may also consider customizing the robots.txt crawl rules using Yoast SEO’s hook instead of writing yours. But again, there’s no way to remove the existing crawl rules. Here’s the relevant hook to work with https://github.com/Yoast/wordpress-seo/blob/f43026341e91e40bae9950ff9f6f4dae616bf172/src/integrations/front-end/robots-txt-integration.php#L97

Hello Maybellyne,

Thanks for reaching out regarding your robots.txt file. If I understand correctly, you use another plugin to manage your robots.txt file and will like to disable to directives added by our plugin.

Yes, this is correct.

Yoast\WP\SEO\register_robots_rules hook returns robots_txt_user_agents and robots_txt_sitemaps
But I need to remove whole section

# START YOAST BLOCK
...
# END YOAST BLOCK

Because it contains directive User-agent: * witch doublicate my own ‘User-agent: *’ directive. And this causes robots.txt validation error.

So, I don’t need to exclude multisite xml sitemaps, I need to turn Yoast robots.txt support off at all and managed them by another plugin which works with multisite network more correctly.
But also, I need to use Yoast XML sitemaps and don’t want to change them, because they work perfectly.

Thank you in advance for your help.

Hello @makspostal

I’m sorry that what you currently want is not possible. Please submit a feature request for our product team on GitHub at https://github.com/Yoast/wordpress-seo/issues

This thread was marked resolved due to a lack of activity, but you’re always welcome to re-open the topic. Please read this post before opening a new request.

Hello @maybellyne

Thank you for your help.
Could you tell me what version of the plugin doesn’t contain this feature?
I mean add Yoast SEO block to robots.txt?

# START YOAST BLOCK
…
# END YOAST BLOCK

I would like to downgrade the plugin.

Thanks.

Hi @makspostal,

Just a quick fix from my side to disable the corresponding filter to attach to the WordPress hook robots_txt.

@maybellyne: Thanks for your effort to making the Yoast plugin such as great tool. However, it appears to be a very invasive way to attach to robots_txt at priority “99999” without letting people knowing that the plugin starts manipulating robots.txt. Please consider to add a custom preference setting or a filter, which can be overwritten in function.php of a theme, to control the call of the function filter_robots.

--- src/integrations/front-end/robots-txt-integration.org.php      2022-11-08 15:04:33.425277229 +0100
+++ src/integrations/front-end/robots-txt-integration.php  2022-11-08 15:04:42.677050142 +0100
@@ -64,7 +64,7 @@
         * @return void
         */
        public function register_hooks() {
-               \add_filter( 'robots_txt', [ $this, 'filter_robots' ], 99999 );
+       //      \add_filter( 'robots_txt', [ $this, 'filter_robots' ], 99999 );
        }

        /**

Cheers,

Matthieu

• This reply was modified 1 year, 11 months ago by Matthieu-P. Schapranow.
• This reply was modified 1 year, 11 months ago by Matthieu-P. Schapranow.
• This reply was modified 1 year, 11 months ago by Matthieu-P. Schapranow.

Hi @theschappy

I totaly agree with you. Yoast SEO is the best SEO sulution for WordPress, but this featue is unacceptable in some cases. In my opinion, it has to be optional.
Thanks for your solution.

https://github.com/Yoast/wordpress-seo/issues/19161

• This reply was modified 1 year, 11 months ago by Max. Reason: Add Github link

I think they turned it off with the latest update, right?

No, the plugin developers have not fixed this problem yet. It’s a pity, it’s a big inconvenience because of this.

Hello all, I did this solution, hope it help for someone.

add_action('wp_loaded', 'show_all_filters');
/**
 * List of all filters with their function names.
 */
function show_all_filters()
{
    remove_filter_by_method_name('filter_robots');
}

/**
 * Remove needed filter from list
 *
 * @param string $method_name Name of the method called to the filter.
 * @param int $priority Set the priorite from that filter if we need to remove just from it.
 *
 * @return void
 */
function remove_filter_by_method_name(string $method_name, int $priority = 0)
{
    global $wp_filter;

    if (isset($wp_filter['robots_txt']->callbacks) && is_array($wp_filter['robots_txt']->callbacks)) {
        foreach ($wp_filter['robots_txt']->callbacks as $callback_priority => $callback) {
            foreach ($callback as $function_key => $function) {
                if ('filter_robots' === $function['function'][1]) {
                    $priority = 0 === $priority ? $callback_priority : $priority;
                    unset($wp_filter['robots_txt']->callbacks[$priority][$function_key]);
                }
            }
        }
    }
}

Hi @nsukonny

Thank you for the solution!

I just simplified your snippet a bit for this specific case:

/**
 * Fix Yoast SEO robots.txt changes.
 * https://www.ads-software.com/support/topic/disable-robots-txt-changing-by-yoast-seo/#post-16648736
 */
function wpwc_fix_yoast_seo_robots_txt() {

	global $wp_filter;

	if ( isset( $wp_filter['robots_txt']->callbacks ) && is_array( $wp_filter['robots_txt']->callbacks ) ) {

		foreach ( $wp_filter['robots_txt']->callbacks as $callback_priority => $callback ) {
			foreach ( $callback as $function_key => $function ) {

				if ( 'filter_robots' === $function['function'][1] ) {
					unset( $wp_filter['robots_txt']->callbacks[ $callback_priority ][ $function_key ] );
				}
			}
		}
	}
}

add_action( 'wp_loaded', 'wpwc_fix_yoast_seo_robots_txt' );

@nsukonny and @makspostal

Where this piece of code must be placed?

/**
 * Fix Yoast SEO robots.txt changes.
 * https://www.ads-software.com/support/topic/disable-robots-txt-changing-by-yoast-seo/#post-16648736
 */
function wpwc_fix_yoast_seo_robots_txt() {

	global $wp_filter;

	if ( isset( $wp_filter['robots_txt']->callbacks ) && is_array( $wp_filter['robots_txt']->callbacks ) ) {

		foreach ( $wp_filter['robots_txt']->callbacks as $callback_priority => $callback ) {
			foreach ( $callback as $function_key => $function ) {

				if ( 'filter_robots' === $function['function'][1] ) {
					unset( $wp_filter['robots_txt']->callbacks[ $callback_priority ][ $function_key ] );
				}
			}
		}
	}
}

add_action( 'wp_loaded', 'wpwc_fix_yoast_seo_robots_txt' );

• This reply was modified 1 year, 6 months ago by Sergio Urra.

@sergiourra Just put it to functions.php in your theme.

It works. Thank you @nsukonny.
I hope Yoast fixes this issue.

Testing wordpress 6.0 yoast. Thanks for the info!