Crawl Optimazation Setting and robots.txt

  • Hello, how are you all?

    I noticed something very weird.

    I adjusted the settings in crawl optimization before couple of days, but now I see that the robots.txt file (in the Yoast plugin interface) did not change after.

    Today, I opened a staging website on my computer to check something, and I needed to generate the robots.txt file through the Yoast plugin again.

    When I generated it, I got a different robots.txt file compared to the original website.

    It’s very important to note that: this is the same website; I just created a copy on my computer to test something. The plugins and the Yoast settings were the same as the original, but when I regenerated the robots.txt file, it was different.

    I noticed that the robots.txt files are not the same.

    Screenshot:

    https://i.ibb.co/Dg6bQ64/image.png

    OG = Original website

    Copy = Staging website

    ——-

    If you notice the copy have:

    Disallow: /?s=
    Disallow: /page/*/?s=
    Disallow: /search/

    This because i adjust this “search” to no index in the “Crawl Optimization” setting of Yoast plugin.

    But as i said before, to see this, i was need to re-generate the robots.txt again.

    on the original website (on the right inside my screenshot), its still the default robots.txt

    Hope you can tell me if its a bug or maybe i wrong and you can light me.

    Regards,

    Nadav

    The page I need help with: [log in to see the link]

Viewing 2 replies - 1 through 2 (of 2 total)
  • Plugin Support Maybellyne

    (@maybellyne)

    Hello @123nadav

    Thanks for using Yoast SEO.

    When you install WordPress without having Yoast SEO, WordPress generates a virtual?robots.txt?file at /?robots=1. Once you install Yoast SEO, we hook into the dynamic file, and you can use our file editor tool to create a static robots.txt file at /robots.txt.

    However, some directives from our crawl optimization settings are added to the virtual file instead of the static one.?It’s currently on our product roadmap to revisit our?robots.txt?implementation.?I apologize for the confusion.

    In the meantime, please copy the contents of the virtual file to the static file so you have the same directives in both since it’s the static file you can validate in the robots.txt tester in Google Search Console.

    Let me know if you have follow-up questions.

    Thread Starter Nadav Levi

    (@123nadav)

    Hey how are you? thank you for your respond.

    from what i see the static robots.txt not actully working:

    https://ibb.co/0yJWFsG

    As you see in the screenshot, google search console robots.txt dont see the static file (?robots=1)

    Its mean i must copy from here:

    https://locksmithunit.com/?robots=1

    to the original robots.txt….

    Very hard to understand this error.

Viewing 2 replies - 1 through 2 (of 2 total)
  • You must be logged in to reply to this topic.