BOTS
Bots are programs that extract information from your website. Certain bots are good and are used by Search Engines, however there are many bots that are not upto any good, or do not following the "Robots exclusion" rules. We recommend blocking these bots, by "User Agent" or by IP.
We categorize the bad or unneeded bots in the following manner :
1. Block on Sight - bots that dont conform.
2. Annoyances, that serve no purpose, just waste your bandwidth - Block
3. Invasive, scamware/spambots bots looking for email adresses - Block
4. Monitoring Bots - unrequested - BlockHere is a few links to sites that have more info on bots and user agents.
Robots .txt sites : Robots exclusion Bot knowledge Info on Bad Bots : Botspot It is good to ensure that you have a robots.txt file with all websites, however those bots that do not honor this file should be blocked by IP.
Honoring robots.txt files - to exclude the NPbot crawler, use "NPbot" as the user-agent name in your robots.txt
Bad Sources of Spam and complete network IP classes will be listed here. Total block on sight is recommended.
Block Bot infected Machines, an never ending source of spam, unauthorized login attempts and Port scans from IP's - We recommend blocking the full class "C" otherwise the providers do nothing :
Click here for a searchable database of bad IP's, ie. ips that spam, are trojaned or are plain abusive according to our determination of our logs. In the event that the IP is for shared hosting, then the innocent others on the same IP have a problem. Get it fixed by applying pressure on your Host/ISP or move hosts, as your site will definately suffer, as it will be shunned because of the IP.
Use at your own risk
**Unmatched Entries** 66.232.38.8 (server.e3-kirkland.com[67.18.87.54]) - no such user 'anonymous' 66.232.38.16 (server.e3-kirkland.com[67.18.87.54]) - no such user 'anonymous' 66.232.38.13 (server.e3-kirkland.com[67.18.87.54]) - no such user 'anonymous' 66.232.38.19 (server.e3-kirkland.com[67.18.87.54]) - no such user 'anonymous' 66.232.38.12 (server.e3-kirkland.com[67.18.87.54]) - no such user 'anonymous' 66.232.38.7 (server.e3-kirkland.com[67.18.87.54]) - no such user 'anonymous' 66.232.38.14 (server.e3-kirkland.com[67.18.87.54]) - no such user 'anonymous' 66.232.38.18 (server.e3-kirkland.com[67.18.87.54]) - no such user 'anonymous' 66.232.38.11 (server.e3-kirkland.com[67.18.87.54]) - no such user 'anonymous' 66.232.38.15 (server.e3-kirkland.com[67.18.87.54]) - no such user 'anonymous' 66.232.38.6 (server.e3-kirkland.com[67.18.87.54]) - no such user 'anonymous' test4.wwbcity.net (server.e3-kirkland.com[67.18.87.54]) - no such user 'anonymous' 66.232.38.10 (server.e3-kirkland.com[67.18.87.54]) - no such user 'anonymous' 66.232.38.9 (server.e3-kirkland.com[67.18.87.54]) - no such user 'anonym
I personally feel sorry for the hosts on that machine, on 67.18.87.54. Nice to the providers really know whats going on on their networks, this is a prime example of "CLUELESSNESS", possible just negligence.Interestingly enough, they show up on at least 10 of our servers.
Prime example of Networks, that have lost control of their machines :
