Indexing issues

  • Resolved inceptiontech

    (@inceptiontech)


    8 months, 1 week ago

    Hello,

    It seems that initial indexing works. However, when trying to URL inspect afterwards, it tells us “Failed: Blocked due to access forbidden (403)” but the crawl is allowed.

    We use Yoast SEO (free) and iThemes Security
    -I’ve tried disabling iThemes and the issue still remained.

    Google bot is allowed in robots.txt
    Sitemap added to robots.txt

    Nothing on our firewall suggests that it blocks Google Bots – Otherwise the original indexing wouldn’t have worked at the first time around.

    Thoughts?

    The page I need help with: [log in to see the link]

Viewing 4 replies - 1 through 4 (of 4 total)
  • Plugin Support Mushrit Shabnam

    (@611shabnam)

    Hi @inceptiontech,

    Can you kindly remove the User-agent: directives for Googlebot and other search engine bots that you have added manually in your robots.txt file ?

    To correct this, kindly follow the suggested directives.

    Thread Starter inceptiontech

    (@inceptiontech)

    Thank you for the reply,
    I’ve removed those directives from the robots.txt

    Let me know if you come across anything

    Plugin Support Maybellyne

    (@maybellyne)

    Please navigate to WordPress > Yoast SEO > Tools > File Editor > Robots.txt and ensure you have these directives:

    # START YOAST BLOCK
    # ---------------------------
    User-agent: *
    Disallow:
    Sitemap: https://www.dynamicspine.com.au/sitemap_index.xml
    # ---------------------------
    # END YOAST BLOCK

    • This reply was modified 8 months ago by Maybellyne.
    Plugin Support Mushrit Shabnam

    (@611shabnam)

    This thread was marked resolved due to a lack of activity, but you’re always welcome to re-open the topic. Please read this post before opening a new request.

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘Indexing issues’ is closed to new replies.

Tags