Bots vs Browsers - Public Bot / User Agent Database & Commentary

Bots vs Browsers - database of 1,554,307 user agents and growing


AUG 26

2006

500 bots!



Catching up again, we have 11 new bots in the last 6 days, and we broke the 500 bot mark this week!

Here are the high points:
  • blogsearchbot-pumpkin-2 visited our sites for the first time. I'm not exactly sure what pumpkins have to do with blog searching, but this bot obviously connects them both.   related...
  • A particularly aggressive bot also visited us by the name of Charlotte/1.0b. In fact, this one is chomping on our site at this very moment at the rate of 1 page per 8 seconds, and has crawled 15,816 pages on our site in 3 days. This guy's hungry, but according to his site (www.betaspider.com), it obeys robots.txt if you want it all to go away. We're willing to take one for the team and let this bot crawl our site uninhibited by robots.txt. We do this so that we can observe and document all bot activities in the wild, without regards to robots.txt.   related...
  • In security news, page_verifier http://www.securecomputing.com/goto/pv scanned through our site looking for malicious software to report. They were kind enough to include a URL in the User Agent text for the bot, which explains more about the bot's purpose, and they also claim to follow the robots.txt exclusion standard.   related...
  • A new version of Exabot/3.0 showed up.   related...
  • A slightly altered DoCoMo/2.0 bot visited our site, displaying a very strange versioning pattern. They are on version 2.0, but the user agent string contains "P902i(c100;TB;W24H12)", which only varies slightly from another version we've seen, "P902i(c100;TB;W24H11)". This may not be a version number, but it seems to increment like one.   related...
  • This just looks shady: crawler root@localhost. Something tells me that root@localhost is not an email address...   related...
  • lwp-trivial/1.41 found our site and loooked around for a bit.   related...
  • HTMLParser/1.6 also found us and poked around. We've seen them before, but the vesrion is slightly modified this time.   related...
  • Sadly, we did not get a new findlinks bot version this week. Development must have finally come to a hault on their bot, as we are used to seeing at least one new version every week. As consolation, here is a link to their greatest hits.
We end this week with 42,284 user agents and 503 bots

Blog Archives:


FEB 29

2016

Leap Day Launch!











APR 18

2009

Where did Palm go?







JUL 13

2008

2,000 bots!





JUN 19

2008

And we're back...


JAN 21

2008

Happy 2nd Birthday!


OCT 25

2007

The Creature Post



JUL 22

2007

Hot Summer Bots




JUN 13

2007

Twiceler Strikes!


JUN 3

2007

Land of 1,000 bots




MAR 24

2007

March Bot Madness


MAR 10

2007

Got Spiders?




FEB 3

2007

February Stars


JAN 12

2007

Happy 1st Birthday!




DEC 12

2006

50,000 User Agents!



OCT 17

2006

2 weeks, 30 bots


OCT 2

2006

4 weeks, 23 bots



AUG 26

2006

500 bots!


AUG 20

2006

5 days, 8 bots











MAY 2

2006

A few new bots









JAN 12

2006

Day One


USER AGENT CATEGORIES

BOTS
BROWSERS
MOBILE BROWSERS
OLD SCHOOL
OTHER
SPAM / HACKS
  • Script Injections (298)
  • SQL Injection (7,697)
  • PHP Injection (1,435)
  • XSS Attacks (58)
  • Link Injection (250,860)