gavindarkvisitors

Glad I could help! And thank you for the updated rating. Let me know if you run into anything else I can help with.

Gavin

@cotros I just checked the logs for recently created projects, and I think I found yours based on what you described. It looks like your website might be hacked, possibly by another WordPress plugin. Visits from Googlebot and Bingbot are getting re-routed to a completely different website, which is why they aren’t showing up in Dark Visitors. You can test this yourself by spoofing the Googlebot user agent in Chrome, like this:

In Chrome with your website open,

• Open the View menu > Developer > Developer Tools
• Then click the vertical “…” in the top right corner > More tools > Network conditions
• Then go to the “User agent” section, uncheck “Use browser default”, and set it to “Googlebot Desktop” in the dropdown
• With the Developer Tools panel still open, refresh the page. You should now see your website rendering a completely different website.

Please let me know if you’re seeing the same thing. If so, I recommend disabling the other plugins you have installed 1 by 1 until you don’t see this happening. With that resolved, Dark Visitors will work as expected. Again, please reach out to me over email if you’d like to discuss this privately in more detail.

Gavin

• This reply was modified 4 months, 2 weeks ago by gavindarkvisitors.

Hi @cotros, agent analytics should see visits from Googlebot and Bingbot (as well as all the others).

2 things to check:

• Make sure the correct project’s access token is correctly pasted into the text box, without any spaces on the sides
• Make sure the “Enable Agent Analytics” box is checked

If you’re still having trouble, please reach out to me at [email protected]. If you share more detail with me I can help debug the issue. I’m more than happy to help.

Gavin

• This reply was modified 4 months, 2 weeks ago by gavindarkvisitors.

Hi @dbareis, checking back in. Let me know if things haven’t fixed themselves as expected. In the meantime, I’ll mark this question as resolved.

• This reply was modified 7 months ago by gavindarkvisitors.

Nope, ~50% is actually pretty common: https://www.google.com/search?q=what+percentage+of+internet+traffic+is+bots

The point of the Agent Analytics feature is actually to analyze this type of traffic.

@dragon013 this should be fixed now, but let me know if not.

Did you add AhrefsBot to your robots.txt within the past day?

Dark Visitors takes some time to re-read your robots.txt, which it currently does every 24 hours. It does this in order to not spam your website by reading it constantly. Rule-breaking visits will start to show up as expected after it re-reads the updated version.

Hope that helps!

Gavin

Hi Matt, they should work together. You should be able to see content from both, combined.

I just tested it myself by going to All in One SEO > Tools > Robots.txt Editor, and adding some custom rules. I see them correctly appear above the Dark Visitors rules in my robots.txt.

Do you have a physical robots.txt file? That will override the dynamic one created by the plugins. Or are you doing something different than what I tested?

Hi again Neal. You have the analytics feature enabled (the “Enable Agent Analytics” checkbox in the plugin settings page).

It literally means that more than half of your website’s visitors are bots rather than humans. You can see which bots they are if you log into the Dark Visitors website and go to the “Insights” page for your project.

Analytics is a paid feature because it uses a lot of compute power. If you’re only interested in the robots.txt feature we talked about before (which is free), then feel free to uncheck the “Enable Agent Analytics” checkbox in the plugin settings page, and ignore that email.

Hope that helps.

They will complement each other. Dark Visitors will be a first line of defense by setting robots.txt rules. CloudFlare will stop any agents that try to ignore the robots.txt rules.

• This reply was modified 7 months, 3 weeks ago by gavindarkvisitors.

Ticking the “AI Data Scrapers” box will do what you want. You already did that, so you are all set.

I mean on each individual agent detail page, not the page I linked to. You need to click on an agent in that list to see the agent detail page. On that page, you will see that header. Let me know if that makes sense. Sorry for the confusion.

• This reply was modified 7 months, 3 weeks ago by gavindarkvisitors.

For model training data, would be “AI Data Scrapers”.

On each agent detail page (https://darkvisitors.com/agents), there’s a “Should I Block … ?” section that goes into more detail about each type.

Hope that helps.

Super helpful, thank you. I think you’re right, I’ll add this fix in the next version.

All the best,

Gavin

Thanks Jim! Feel free to reach out to “contact at darkvisitors.com” if you run into any issues or have any feedback. Happy to help.

Gavin