robots.txt is not valid (Lighthouse was unable to download a robots.txt file)

  • Resolved Mostafiz Shamim

    (@itzmostafiz)


    3 years, 5 months ago

    Hi,

    I created the robots.txt file from Yoast and did not make any changes manually. But Lighthouse is showing that there is no valid robots.txt file on my website.

    The robots text inside the file is,

    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php

    Would you please tell me How I can solve the problem?

    Regards.

    The page I need help with: [log in to see the link]

Viewing 3 replies - 1 through 3 (of 3 total)
  • Plugin Support Michael Ti?a

    (@mikes41720)

    Hi @itzmostafiz

    Upon checking, we are able to access and see a valid robots.txt file being generated by the Yoast SEO plugin, so we’re not exactly sure why you are receiving that assessment from Lighthouse.

    Could you run your file through Google’s robots.txt file tester and see if it is able to detect it properly? You can learn more about it here – https://support.google.com/webmasters/answer/6062598/

    Thread Starter Mostafiz Shamim

    (@itzmostafiz)

    Would you please check it through Lighthouse? After checking through the lighthouse, would you tell me the positive solution?

    Thank you.

    Thread Starter Mostafiz Shamim

    (@itzmostafiz)

    Hey, I solved the problem myself. Just added the following code in the robots file and everything is alright now.

    Disallow: /cdn-cgi/

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘robots.txt is not valid (Lighthouse was unable to download a robots.txt file)’ is closed to new replies.