Robots…

besides that they are still coming… the common ones still won’t stop and they are still trying to got o pages and files that just aren’t there and creating hits that way… they’ve never done that before….just started as soon as i upgraded

As I have already stated:

“Well, due to the change in your blog’s generator, the Googlebot and other major search bots may be re-indexing your site and comparing their existing index with your current site to remove any “dead” pages from their indexes.”

Bad Behavior will not stop legitimate bots. If you want to disappear from the listings and results of all major search engines, feel free to use a .htaccess file to block those bots by their user-agents. Otherwise, just get some sleep and let them run their course.

oh…sorry… i just get worried when i look and see them trying to hit files and pages that aren’t there and sometimes the link they are looking for is just a bunch of random letters and numbers…

Well, like I said, “If you want to disappear from the listings and results of all major search engines, feel free to use a .htaccess file to block those bots by their user-agents.” If it really is a major concern to you, then I suggest that you contact the administrators of the “offending” search engines.

Additionally, this article may interest you:

https://ajaydsouza.wordpress.com/2005/10/02/protecting-your-blog-from-spam/

Here is the list of the “bots” that are doing this… I’ve had maybe 10 legitimate hits since the counter reset at midnight and its 8 AM here now and it says that I am pushing nearly 200 hits. I am wondering if others are having this happen as well. One imaginary page they keep trying to go to is robots.txt, which simply isn’t there and never has been. Also, it these “legitimate” bots are just updating their listings, why do they keep coming back and just repeating the hits over and over… I mean, I can understand what macmanx is saying that there are times when all the bots do things like this, but its been almost non stop for two days, ever since the minute I upgraded to wordpress 2.0…

Mozilla/5.0 (compatible; Yahoo! Slurp; https://help.yahoo.com/help/us/ysearch/slurp)

Mozilla/5.0 (compatible; Googlebot/2.1; +https://www.google.com/bot.html)

PHP/4.2.2

msnbot/1.0 (+https://search.msn.com/msnbot.htm)

Googlebot/2.1 (+https://www.google.com/bot.html)

GivingYouFreeLinks/0.1 (+https://referer.org/)

Baiduspider+(+https://www.baidu.com/search/spider.htm)

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Larbin/2.6.3 ([email protected])

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)

Gigabot/2.0

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312465)

Mozilla/4.0 (compatible;)

PEAR HTTP_Request class ( https://pear.php.net/ )

Java/1.4.1_04

WordPress/2.0

I’ve been having the same problems for the whole mobth of December and I am still on 1.5.2. I’ve had to increase my bandwidth several times throughout the month. I’m now up to 1GB. It seems that half are the “search bots”. I am concerned, too.

exactly jwurster… my bandwidth is going way up!!! and after looking over the log, you’re right, it has been december, prior to my upgrade to 2.0…

i know that most people wouldn’t complain about their hit counter running up cause it makes it look as though you site is getting all these visitors, but when they run up your bandwidth, it becomes a problem…

and like i said, i’ve gotten these bots in the past, no big deal, but its just been nonstop lately to the same pages over and over and to pages/files that don’t exist…

i’m glad its not just me…

OK… Now its a problem cause my site is down…and i get this messege…

Precondition Failed

We’re sorry, but we could not fulfill your request for /wp-admin/link-manager.php on this server.

We have established rules for access to this server, and any person or robot that violates these rules will be unable to access this site.

To resolve this problem, please try the following steps:

* Ensure that your computer is free of viruses, Trojan horses, spyware or any other sort of malicious software.
* If you are using any sort of personal firewall or browser privacy software, check to ensure that its settings do not cause your web browser to inadvertently violate any of the rules listed below.
* If you are behind a Web proxy or corporate firewall, the proxy must conform to the HTTP specification with respect to proxy servers. Contact your network administrator if the trouble persists, or bypass the proxy and connect directly if possible.
* Disable any download accelerators you may be using. They don’t speed up your downloads anyway; in most cases, they actually run slower!
* If all else fails, try using a different Web browser, such as Firefox.

If you still need assistance, please contact admin at mykahil.com.
More Information

For your reference, the conditions for access to this server are:
Robots:

* MUST read and obey robots.txt.
* MUST identify themselves properly; for example MUST NOT identify as Mozilla.
* MUST NOT pretend to be a human.

Humans:

* MUST NOT pretend to be a robot.
* MUST NOT use a computer infected with viruses, Trojan horses or other malicious software.

Both:

* MUST NOT harvest email addresses.
* MUST NOT attempt to send spam.
* MUST NOT attempt to compromise server security.
* MUST NOT use excessive amounts of bandwidth or other server resources.

The precondition on the request for the URL /wp-admin/link-manager.php evaluated to false.

ahhh… figured it out…

it was the plugin “bad behavior” that someone suggested i use for the above bot problem… bad idea i guess…

Kahil, I’m sorry, I guess that I’ve given you all the wrong advice, or that you just don’t want to listen to it. I will re-post the important points, incase you are in a mood to listen.

* Bad Behavior stops hits from known “bad bots” including those which falsely pretend to be major search bots, attempt to steal content, post spam, harvest email addresses, or hunt for security exploits. If you are suspicious of these bots, activate the plugin, and let it decide on whether these bots are legitimate or malicious.

* If you get blocked by Bad Behavior, read this: https://error.wordpress.com/2005/09/30/what-to-do-when-bad-behavior-blocks-you-or-your-friends/

* Due to the change in your blog’s generator, the Googlebot and other major search bots may be re-indexing your site and comparing their existing index with your current site to remove any “dead” pages from their indexes.

* If Bad Behavior is not blocking it, then that means that it is the one and only genuine Googlebot. The same can be said for the Yahoo and MSN bots. And, you certainly don’t want to block those. They’re probably just re-indexing your site. That does happen from time to time.

* If you want to disappear from the listings and results of all major search engines, feel free to use a .htaccess file to block those bots by their user-agents. Otherwise, just get some sleep and let them run their course.

* If it really is a major concern to you, then I suggest that you contact the administrators of the “offending” search engines.

easy macmanx…

summerdonna…

i’m just saying that there was no change in bot activity and the plugin blocked me all of a sudden… i was in my admin section one moment, then all of a sudden i get blocked…

and since someone else has come forward with the same problem, there has to be something going on. it has become a problem cause its eating up all the bandwidth.

i’m not saying you aren’t right with your explanation, just that there may be another and we shouldn’t just toss it aside.

Have you contacted the administrators of the “offending” search engines yet?

no… cause other than google, yahoo and msn, there are still several others doing this… like just a few mins ago, java something put almost 50 hits on my site in a matter of minutes…

i have a large bandwidth, and haven’t had the issue this has caused to jwurster above, so i’m not gonna go that far yet. but since others have that problem, i think we should find out why. its not gonna do us no good if we ask them to stop and not know why all the bots are doing this…

Any user-agent containing “java” should have been blocked by Bad Behavior. Are you sure that it’s active, or have you de-activated it?