Bots vs Browsers - Public Bot / User Agent Database & Commentary

Bots vs Browsers - database of 16,268,670 user agents and growing


DEC 13

2013

Are the Bots beating the Browsers???



After years of observing Bots vs Browsers by analyzing user agent logs, the struggle appears to have swung heavily in favor of the bots in 2013. As late as 2012, the browsers (humans) were still within 1% of the majority of website traffic, but 2013 saw robot traffic increase with a vengeance.

According to an article published by Incapsula, robots are now responsible for 61.5% of all website traffic. The study last year measured much closer, at 51% of all website traffic coming from bots. This pushes human traffic down from 2012 at 49% down to 38.5% in 2013.

As far as the breakdown of what type of robots are responsible for all of this traffic, here is a quick list:
  • 31% Search Engines (i.e. the "good" bots)
  • 5% Scrapers
  • 4.5% Hacking tools
  • 0.5% Spammers
  • 20.5% Other Impersonators (basically all other non-humans)
You can read more about the findings here.

Blog Archives:












APR 18

2009

Where did Palm go?







JUL 13

2008

2,000 bots!





JUN 19

2008

And we're back...


JAN 21

2008

Happy 2nd Birthday!


OCT 25

2007

The Creature Post



JUL 22

2007

Hot Summer Bots




JUN 13

2007

Twiceler Strikes!


JUN 3

2007

Land of 1,000 bots




MAR 24

2007

March Bot Madness


MAR 10

2007

Got Spiders?




FEB 3

2007

February Stars


JAN 12

2007

Happy 1st Birthday!




DEC 12

2006

50,000 User Agents!



OCT 17

2006

2 weeks, 30 bots


OCT 2

2006

4 weeks, 23 bots



AUG 26

2006

500 bots!


AUG 20

2006

5 days, 8 bots











MAY 2

2006

A few new bots









JAN 12

2006

Day One


USER AGENT CATEGORIES

BOTS
BROWSERS
MOBILE BROWSERS
OLD SCHOOL
OTHER
SPAM / HACKS
  • Script Injections (349)
  • SQL Injection (7,701)
  • PHP Injection (1,993)
  • XSS Attacks (66)
  • Link Injection (250,869)