I have a setup where the WordPress code is located in a subdirectory even though it appears to be in the top-level (root) directory (see https://developer.www.ads-software.com/advanced-administration/server/wordpress-in-directory/). I couldn’t get this plugin to work with that setup. The robots.txt file didn’t change. And the “click here to see your robots.txt” link pointed to the subdirectory instead of the root directory. I ended up using a different plugin but thought you might like to know.
]]>Hi,
On a multisite installation with subfolders
(ex. https://www.mymultiste/site1, https://www.mymultiste/site2) the robots.txt url return a 404 error for every site.
Is the plugin compatible with this type of installation?
]]>The issue brought up in “Bugs found: missing CR and wrong replacement of &” still has not been fixed: WP Robots.txt still breaks other plugins adding to the virtual robots.txt, as any trailing newline is being stripped, resulting in an unwanted joining of lines, as shown in the original post.
A fix would be to add a trailing newline in inc/core-functionality.php :
function robtxt_filter_robots( $output, $public ) {
$content = get_option( 'robtxt_content' );
if ( $content ) {
$output = esc_attr( wp_strip_all_tags( $content ) ) . PHP_EOL;
}
return $output;
}It would be most helpful to get this integrated. Thanks for considering.
]]>Does this plugin work for a specific subsite in a multisite enviroment?
]]>A recent Wordfence scan flagged this plugin as having a ‘critical’ security vulnerability on 2022-10-21 (details below). Just wanted to mention to see if this is something you’re aware of and working on. I noticed that the plugin was last updated 4 days ago, so perhaps this update contained the vulnerability. Thank you for your help.
]]>Plugin Name: WP Robots Txt
Current Plugin Version: 1.2
Details: To protect your site from this vulnerability, the safest option is to deactivate and completely remove “WP Robots Txt” until a patched version is available. Get more information.
Hello Christopher,
It seems that your WP Robots Txt plugin has been abandoned or you don’t have time to maintain it, since it was last updated 8 years ago. There are deprecated functions that need updating and also Notices/Warnings that should be resolved to make it compatible with the latest WordPress and PHP versions.
Instead of letting this plugin stagnate, I would be interested in adopting it and continue development where you left off, especially since there is still a number of people using it.
Would you be willing to add me as a committer as explained here? I will then be responsible for future development, maintenance, updates and to ensure the plugin complies with all WP plugin guidelines.
Thanks,
George
Hey, I have a form page, which I need to add to robots.txt. Should I just write Disallow: /contact-form/, or I need to specify the full URL?
Hello,
Very useful plugin for WordPress multisite.
However, I have found some little bugs.
1. When other plugins add some lines in robots.txt, they appear on the same line. Ex: Disallow: /wp-admin/Disallow xxx
The solution is to add a trailing CR in the option just before saving it.
2. the & character is encoded to & That is not correct.
A solution is to not use esc_html() and esc_attr().
Here are the patches to fix these bugs:
]]>Index: core.php
===================================================================
— core.php (revision 148089)
+++ core.php (revision 148090)
@@ -29,14 +29,13 @@
*
* @since 1.0
* @uses get_option
– * @uses esc_attr
* @return string
*/
function cd_rdte_filter_robots($rv, $public)
{
$content = get_option(‘cd_rdte_content’);
if ($content) {
– $rv = esc_attr(strip_tags($content));
+ $rv = strip_tags($content);
}return $rv;
Index: options-page.php
===================================================================
— options-page.php (revision 148089)
+++ options-page.php (revision 148090)
@@ -123,7 +123,6 @@
* robots.txt field
*
* @since 1.0
– * @uses esc_html
* @uses add_settings_error
*/
public function cleanSetting($in)
@@ -138,7 +137,7 @@
);
}– return esc_html(strip_tags($in));
+ return strip_tags($in) . “\n”;
}/**
Hi,
I have two websites.
The one is WP ver. 4.2.2., the plugin was installed and the robots.txt was modified super fast. ref.: https://www.bulgarianplates.com
I got another one ver. 4.0.1., and there is ssl installed. I tried several times but the robots.txt cannot be changed. ref.: https://www.immigration2bulgaria.com
Thanks
]]>Good Afternoon,
When I load the robots.txt file for my site the contents doesn’t match the content in the WordPress settings.
How do I fix this?
Thank you for your assistance.
Grace
]]>I would like to add this plugin, but my current XML-Sitemap Generator plugin has the following option, which I have selected. Will this cause an issue? meaning will this create a robots.txt file in my directory?
SELECTED – Add sitemap URL to the virtual robots.txt file.
The virtual robots.txt generated by WordPress is used. A real robots.txt file must NOT exist in the blog directory!
Is it possible to use your plugin to allow access to most of a wp site, but discourage them from visiting one particular page? It’s not super-critical info that warrants a password, just a page I don’t want indexed.
I have not linked the page via navigation/menu or any other embedded link, so in theory, I should be safe from search engines indexing it.
But I was hoping I could use this plugin as an extra precaution. I can’t just use the permalink to the page, right? If yes, can you give me an example of a URL path to a single page to add to the Disallow area?
e.g. Disallow: /wp-content/???
or does this require some poking around in the MySQL area of my server?
]]>The default robots.txt still seems to be appearing. What I filled in on Settings -> Reading is missing.
]]>is this 3.7.1 compatible?
]]>I installed the plugin, but there is nothing in the Settings or Settings/Privacy. So nothing happened. Im hosted on Godaddy, I checked the FTP – your robots plugin is listed but no file. What am I doing wrong?
]]>hello,
this plugin has been working great for me up until a week or so ago. if you go and look at my robots file https://www.cheesycorporatelingo.com/robots.txt, you will see that it is duplicating the entry.
any idea what might be causing this?
]]>After installing WP Robots Txt, I noticed that once I pressed “Update” after any changes on the 2 websites I installed it in, I would get a blank page and I’d have to return to those pages manually through the URL.
Now it got worse, I can’t access the backend of any of the sites I installed it in. If I type example.com/wp-admin, I get a blank page.
I believe I left the User-agent:* Disallow:/wp-admin/ and Disallow /wp-includes/ on the file.
Could that have caused it? Regardless… How do I go back to what I had before? I already ftp’d into the server and deleted the plugin files manually but I still get a blank page.
How can I fix this?
Thanks!
]]>Please I want to know if there’s anyway to increase my google search page rank to No.1. When searching google, Live365 always appear first even when i search it with my site’s name (https://365praise.net) and mine appears second. HELP!
]]>Can we expect an update for 3.5 up to 3.6?
Or is this plugin dead?
]]>Hi,
Just installed and activated the plugin. Unfortunately the ‘Privacy’ area in settings doesn’t seem to appear.
Cheers,
Ed.
]]>WP Robots Txt is not compatible with 3.5 ?
When is the new update coming ?