robots.txt file hidden

The robots.txt file is either in your public domain folder. Another place you might find it, could be through your Host’s CPanel.

Thanks Duncan,
I looked in the cpanel and only saw the empty file. I’ll try the public domain folder (not clear if you’re suggesting my dashboard for the public domain folder but I’ll try dashboard first).

On my cPanel I checked all through the public_ftp & public_htm files and I’m still looking. I don’t see any public domain folder or I guess I can check domain………
Thanks

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

That’s typically the contents of the default WordPress virtual robots.txt file.You can view it here: https://www.lingolconstruction.com/robots.txt It isn’t a physical file.

I put in a robots.txt file which is empty

A) Make sure the file made it into the root directory (and that there are no typo’s in the file name)

B) Try adding something to it and see if it begins to override the virtual file (normally, the virtual file will be overridden when an actual robots.txt file has been added)

Try adding this to the blank robots.txt file you created…

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cgi-bin/

..and then visit https://www.lingolconstruction.com/robots.txt to see if the above shows in the file.

Thank you Clayton. I’ll try that now.

No change that I can tell so the robots.txt file I have in the root directory is not being detected.

I checked all through the public_ftp & public_htm files and I’m still looking. I don’t see any public domain folder

It won’t be named public domain. /public_html is probably what you want. If that’s where WordPress is installed, that’s where the robots.txt file needs to go.

If www.lingolconstruction.com actually points to a different directory where the WordPress files are installed, then that’s the WordPress root directory, and that’s where the file needs to go.

O.K.,
I found the wp-admin/admin-ajax.php file in my cpanel on host site. I’m trying search (F3 key) to find pertinent text to focus on. I know just the basic info about PHP from a php online class.

Maybe this text is pertinent:

‘
/** Load Ajax Handlers for WordPress Core */
require_once( ABSPATH . ‘wp-admin/includes/ajax-actions.php’ );

@header( ‘Content-Type: text/html; charset=’ . get_option( ‘blog_charset’ ) );
@header( ‘X-Robots-Tag: noindex’ );

send_nosniff_header();
nocache_headers();

/** This action is documented in wp-admin/admin.php */
do_action( ‘admin_init’ );

‘

Then from the admin.php file in last line above:

/**
 * Fires as an admin screen or script is being initialized.
 *
 * Note, this does not just run on user-facing admin screens.
 * It runs on admin-ajax.php and admin-post.php as well.
 *
 * This is roughly analogous to the more general 'init' hook, which fires earlier.
 *
 * @since 2.5.0
 */
do_action( 'admin_init' );

if ( isset($plugin_page) ) {
	if ( !empty($typenow) )
		$the_parent = $pagenow . '?post_type=' . $typenow;
	else
		$the_parent = $pagenow;
	if ( ! $page_hook = get_plugin_page_hook($plugin_page, $the_parent) ) {
		$page_hook = get_plugin_page_hook($plugin_page, $plugin_page);

		// Backwards compatibility for plugins using add_management_page().
		if ( empty( $page_hook ) && 'edit.php' == $pagenow && '' != get_plugin_page_hook($plugin_page, 'tools.php') ) {
			// There could be plugin specific params on the URL, so we need the whole query string
			if ( !empty($_SERVER[ 'QUERY_STRING' ]) )
				$query_string = $_SERVER[ 'QUERY_STRING' ];
			else
				$query_string = 'page=' . $plugin_page;
			wp_redirect( admin_url('tools.php?' . $query_string) );
			exit;
		}
	}
	unset($the_parent);
}

Whoa.. hang on, hold up..

..don’t do anything with any php code in a core file. It has nothing to do with your robots.txt file issue. You don’t need to be there. Big red flag. ??

Let’s start over.

A) Can you confirm by looking in the /public_html directory using an FTP client, that there is a physical file named robots.txt?

B) If yes, are you using any caching plugins or services (cloundflare for example)

C) Are you using any SEO plugins that also include features that might allow you to alter the WordPress virtual robots.txt file?

D) what is the exact name of your robots file, including any upper/lower case and its file extension? It should be robots.txt – no upper case letters or spaces, and .txt is the correct extension.

No, I never found a robs.txt
Like you said it seem virtual. I get the feeling it’s generated by the PHP and,or ajax.

I’ve been trying to find that type of information you mentioned in plugins.

I have the spelling correct in my robots.txt file (which now is no longer empty)

It has the text you mentioned inside:

Here’s copied and pasted from it.

Right out of dreamweaver where I edited before uploading with filezilla

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cgi-bin/

Only plugin I have activated now is:

NextGEN Gallery

I deactivated the rest for process of elimination, etc.

No, I never found a robs.txt

Then that’s the problem.

Right out of dreamweaver where I edited before uploading with filezilla

Where is that file right now – to what directory was it uploaded?

[edit]

Like you said it seem virtual.

It is virtual, in the absence of a real one. Then the real one takes precedence.