Bots vs Browsers - Public Bot / User Agent Database & Commentary

Bots vs Browsers - database of 1,566,688 user agents and growing


SEP 7

2007

iTunes, Facebook, Google Earth, and more



We have introduced some new categories for some of the various bots and user agents we are seeing for the first time.

  • Google Earth and iTunes have browsers built into their systems, so their user agents will be tracked in these two new categories.
  • We've seen several RSS related user agents - mainly bots - and we'll be tracking these under this category going forward.
  • Finally, we have started tracking Sitemap user agents - some of these are from us evaluating sitemap tools, and some are visitors that we did not invite. Usually, these tools are used to build sitemaps for Google or Yahoo!, which both offer useful sitemap services to help webmasters publish all of their URLs directly to the search engines to help the web crawlers do a thorough job.

We hope that the new categories help with all of your bot research - enjoy!

Blog Archives:


FEB 29

2016

Leap Day Launch!











APR 18

2009

Where did Palm go?







JUL 13

2008

2,000 bots!





JUN 19

2008

And we're back...


JAN 21

2008

Happy 2nd Birthday!


OCT 25

2007

The Creature Post



JUL 22

2007

Hot Summer Bots




JUN 13

2007

Twiceler Strikes!


JUN 3

2007

Land of 1,000 bots




MAR 24

2007

March Bot Madness


MAR 10

2007

Got Spiders?




FEB 3

2007

February Stars


JAN 12

2007

Happy 1st Birthday!




DEC 12

2006

50,000 User Agents!



OCT 17

2006

2 weeks, 30 bots


OCT 2

2006

4 weeks, 23 bots



AUG 26

2006

500 bots!


AUG 20

2006

5 days, 8 bots











MAY 2

2006

A few new bots









JAN 12

2006

Day One


USER AGENT CATEGORIES

BOTS
BROWSERS
MOBILE BROWSERS
OLD SCHOOL
OTHER
SPAM / HACKS
  • Script Injections (300)
  • SQL Injection (7,697)
  • PHP Injection (1,975)
  • XSS Attacks (58)
  • Link Injection (250,863)