Robotstxt

  • Hi there,
    I am trying to set up my robotstxt file, but for some reason doesn’t work out.

    I did verify the ownership of my domain on google.
    I did write the robotstxt file on wp.org and on google too.
    But every time I go and check what’s on google console, my robotstxt file is not there.

    Any help, please?!

    Second issue

    The logo on my head bar doesn’t fit.

    Not sure why.
    I did try all the resolution possible, I did change the with and high under the customize section, but no, doesn’t wanna fit.

    I did save it for the web, or as a normal .jpg, no, doesn’t work.
    ??
    [ redundant link removed ]

    The page I need help with: [log in to see the link]

Viewing 2 replies - 1 through 2 (of 2 total)
  • Anonymous User 14808221

    (@anonymized-14808221)

    Please open another topic with the second issue, it will be easier to help each other out like that.

    Related to the robots file, you don’t need google to get a robots file to work
    You just need to add the robots.txt file to the root of your install.

    Like, if your WordPress website is my-site.com then place your robots txt file at my-site.com/robots.txt

    This is done by placing it in the FTP on the same level as your wp-config.php file is, for example (or .htaccess file, or wp-admin folder and such “root” components)

    In the robots file itself you can add something like this:

    
# All Bots
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

# SemRush
User-agent: SemrushBot
Disallow: /

Sitemap: https://www.my-site.com/sitemap.xml

    This will for example block semrush bots, direct the bots towards your sitemap if you have one, and set some allow/disallow rules for general access to folders/files on your install.

    I myself do not use Google console as it will not really increase any form of traffic and usually just signifies more work for not much profit.
    However, if you need it, you might want to start here to validate your robots file:
    https://support.google.com/webmasters/answer/6062598

    Hope this helps!

    Thread Starter giolrcc86

    (@giolrcc86)

    Thanks Beda,
    I did sort out the second issue.

    In regard the robots.txt file, I did write it in the Seo plugin.

    This is what I wrote:

    “User-agent: *

    Disallow: /wp-admin/

    Allow: /wp-admin/admin-ajax.php

    Allow: /wp-content/uploads/

    Sitemap: https://example.com/post-sitemap.xml

    Sitemap: https://example.com/page-sitemap.xml”

    Why I was concern about Google, as because, when I Google my website I get below the finding:
    “No info for this website” learn why.

    If then I click on learn why, it make me go through the domain verification (done it), and the robots.txt configuration.

    Anyway, I also have been told to wait, as this may happens because Google didn’t index my website yet.

    Thanks again.

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Robotstxt’ is closed to new replies.