When visiting my site via Firefox the sites return rndmly: User-agent: * Disallo

  • Hello!

    I am experiencing a weird issue when using Firefox (Mac as well as Windows
    machines) and visiting a WordPress site that is built using custom post
    types.

    The issue occurs randomly yet consistently and is usually solved when
    clearing firefox’s browser cache – at least until for a few reloads:

    1. Fire up firefox
    2. Visit for example https://saskialund.de
    3. Navigate to “Referenzen”
    4. Hope to see the Referenzen page… if not it will show
    User-agent: *
    Disallow: /wp-admin/
    instead
    5. keep browsing the page. The above text will show eventually, once
    firefox starts to build the cache.

    Questions:
    1. Why is the robots.txt being displayed instead of the page content?
    2. Did I do something wrong when setting up the installation?

    On all pages that have this issue the following plugins are installed:
    Jetpack
    Yoast Seo

    I am experiencing this issue with all of my wordpress sites that have
    custom post types registered and in use.
    Even in a page that is using a twentyfifteen child (see here:
    https://luciaschwalenberg.de ).

    I am aware of this issue since WP 4.1 however I am not sure if it was
    there before 4.1 or not and as said: it only occurs when using firefox.

    What could be the issue? Is this a bug?

    Thanks for looking into this!

    https://www.ads-software.com/plugins/jetpack/

Viewing 3 replies - 1 through 3 (of 3 total)
  • Plugin Contributor James Huff

    (@macmanx)

    That sounds like your robots.txt file is being displayed, which is not a part of Jetpack.

    Try deactivating all plugins. If that resolves the issue, reactivate each one individually until you find the cause.

    If that does not resolve the issue, try switching to the Twenty Fifteen theme to rule-out a theme-specific issue (theme functions can interfere like plugins).

    If you still need help, I recommend re-asking over at https://www.ads-software.com/support/forum/how-to-and-troubleshooting#postform as we cannot move threads outside of the plugin-specific support forums.

    Thread Starter Saskia Teichmann

    (@jyria)

    I found the culprit!

    On my server was an nginx directive snippet in place which caused firefox to stumble over its feet while interpreting Cache-Control which was sent by my nginx.
    The problem was that wrong quote marks were used to wrap “public” in the following code snippet: ( these were the wrong ones: ″ )

    location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
    expires 30d;
    add_header Pragma public;
    add_header Cache-Control "public";
    try_files $uri @fallback;
}

    Firefox showed broken Cache-Control definitions. Instead of these correct ones:
    Cache-Control: “max-age=2592000, public”
    something like this was shown (unfortunately I didnt save the exact interpretation):
    Cache-Control: “max-age=2592000, ?public?”

    Chrome didn’t have a problem with the other font quotes and therefore never showed that behaviour.

    Also did I find that mod_pagespeed which was activated on that server was causing caching problems.

    Funny thing is, I installed a site on a client’s webhosting which is just a shared webhosting with german STRATO. Same caching problem occurs over there.
    They might have this very nginx directive in place also with those wrong quotes that can’t be interpreted correctly by firefox. MAybe even also installed mod_pagespeed which is not fully configured.

    Plugin Contributor James Huff

    (@macmanx)

    Ouch, that sounds nasty. I’m rather rusty in my Nginx knowledge, I’d recommend checking with your hosting provider’s support community, if they have one, or your hosting provider’s direct support.

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘When visiting my site via Firefox the sites return rndmly: User-agent: * Disallo’ is closed to new replies.