Problem with robots.txt

  • Hello, I have the following problem:

    I am an administrator of a wordpress multisite with more than 1000 sites. The Wordfence plugin has been installed as a security plugin for less than a month.

    Since then I have a problem with Google indexing, it has stopped indexing my pages and when I inspect the URLs through Google Search Console, they tell me that it is not available. It tells me that there is a problem with the robots.txt.

    The file can be accessed without problem from the browser and its content has continued to work well for years.

    The plugin has been installed and little by little the content indexed on Google decreases. Reviewing the plugin configuration, it does not seem to block any Google Bots and the Rating Limitation option is also disabled.

    I’ve been looking for a solution for several days but I can’t find anything.

    I appreciate all the help possible.

Viewing 4 replies - 1 through 4 (of 4 total)
  • Plugin Support wfpeter

    (@wfpeter)

    Hi @mjsdsoyyo, sorry to see you’re having problems with Google Search Console picking up pages on your sites.

    I have seen some similar robots.txt warnings for users of other plugins around SEO or other security products when looking into this. I should be able to assist if I see a copy of a diagnostic report and current contents of robots.txt from one of the affected sites.

    You can simply paste the latter into a code block here in the forum editor, but you can send a diagnostic from the Wordfence > Tools > Diagnostics menu to us at wftest @ wordfence . com. Click on “Send Report by Email”. Please add your forum username where indicated and respond here after you have sent it.

    NOTE: It should look as follows – Screenshot of Tools > Diagnostic > Send by Email

    Thanks,
    Peter.

    Thread Starter mjsdsoyyo

    (@mjsdsoyyo)


    Hello, in the diagnostic that you request, there is sensitive information that for security reasons we cannot provide. Could you tell me exactly what information you need to assess whether it can be provided?
    The robots.txt includes the following information

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /pre/
    Allow: /wp-admin/admin-ajax.php

    Sitemap: https://dominio.es/sitemap.xml

    Saludos

    Hello Peter,

    i have the same problem. I sent the wordfence diagnostics file to wftest @ wordfence . com.

    By the way, google says, that following sites could be indexed, because of a “noindex” tag
    https://fv-schweizer-viertel-bremen.de/comments/feed/
    https://fv-schweizer-viertel-bremen.de/datenschutz/feed/

    I disallowed comments in the blog and what feed?

    It would be great, if there can be found a solution.

    The robots.txt file looks like this, as in all my WP installations:

    START YOAST BLOCK —————————

    User-agent: *
    Disallow:

    Sitemap: https://fv-schweizer-viertel-bremen.de/sitemap_index.xml ————————— END YOAST BLOCK

    Many thanks in advance
    Michael

    crazyhamster

    (@crazyhamster)

    I have the same problem on two my sites. What should I do? Everything worked fine. But as soon as I installed this plugin, the problems started. I send diagnostics

Viewing 4 replies - 1 through 4 (of 4 total)
  • You must be logged in to reply to this topic.

Tags