Bots vs Browsers - Public Bot / User Agent Database & Commentary

Bots vs Browsers - database of 16,268,725 user agents and growing


AUG 20

2006

5 days, 8 bots



In the brief 120 hours since my last post, we've encountered 8 new bots. At this rate, to make sure we don't get behind, here's the news on these new bots:
  • A new version of TurnitinBot/2.0 came by. According to their site, Turnitin indexes content on the web to compare to student's school work, to try and detect plagiarism. We are flattered that this bot felt the need to crawl our site, and would be even more flattered if a student ever plagiarized our content.   related...
  • In other news, WeRelateBot/0.9 searched our site for content related to ancestry. WeRelate.org owns this bot, and their site allows you to discover information about your family's history. It really is a smart move to crawl a website about bots to harvest content about human ancestry...   related...
  • Startup search engine "Sproose" sent a new crawler by, sproose/0.1. According to the Sproose home page, their "personalized search" technology is still in the works, so keep an eye out for this new service to launch.   related...
  • NimbleCrawler dropped in a new version this week, debuting NimbleCrawler 2.0.1.   related...
  • MSN showed up to the party with a new MSN Bot, msnbot-Products/1.0.   related...
  • Never seen this one before... Mozilla/5.0 crawl@digigetx.com Safe to say it's a bot from the "crawl" part of the User Agent. For more information, drop them a line at crawl@digigetx.com - they didn't include their email for nothing.   related...
  • Microsoft Research sent it's bot by, MSRBOT, for a look around our sites.   related...
  • And finally, our weekly findlinks version iteration came by like clockwork this week, now reporting version findlinks/1.1.3-beta8.   related...
The week concludes, putting our database at 41,818 user agents and 492 bots

Blog Archives:












APR 18

2009

Where did Palm go?







JUL 13

2008

2,000 bots!





JUN 19

2008

And we're back...


JAN 21

2008

Happy 2nd Birthday!


OCT 25

2007

The Creature Post



JUL 22

2007

Hot Summer Bots




JUN 13

2007

Twiceler Strikes!


JUN 3

2007

Land of 1,000 bots




MAR 24

2007

March Bot Madness


MAR 10

2007

Got Spiders?




FEB 3

2007

February Stars


JAN 12

2007

Happy 1st Birthday!




DEC 12

2006

50,000 User Agents!



OCT 17

2006

2 weeks, 30 bots


OCT 2

2006

4 weeks, 23 bots



AUG 26

2006

500 bots!


AUG 20

2006

5 days, 8 bots











MAY 2

2006

A few new bots









JAN 12

2006

Day One


USER AGENT CATEGORIES

BOTS
BROWSERS
MOBILE BROWSERS
OLD SCHOOL
OTHER
SPAM / HACKS
  • Script Injections (349)
  • SQL Injection (7,701)
  • PHP Injection (1,993)
  • XSS Attacks (66)
  • Link Injection (250,869)