Bots vs Browsers - Public Bot / User Agent Database & Commentary

Bots vs Browsers - database of 16,250,331 user agents and growing


APR 16

2007

Spring Cleaning in the Web Traffic Logs



Springtime brings spring cleaning, so we're going to try and clean out the user agent logs since it's been 3 weeks (or more) since our last post. Here's what we found lurking around in our absence:
  • As far as mythical creatures in the traffic logs, we saw a gnome-vfs/2.16.2 neon/0.25.4 digging around and a Yeti/0.01 crawling through. The Yeti's user agent told us that it "check robots.txt daily and follows it", and is a variation of Naver. related...
  • We added 2 variations of the Larbin Web Crawler to our database this week - here's one and here's the other. related...
  • This one was unique - the facebookscraper/1.0 visited our sites, even though we are not facebook. I guess it got lost and scraped up on some other sites? related...
  • Someone is actually keeping a meta tags directory and is harvesting their content using metatagsdir/0.7. Seems like an odd hobby, but then again, running a blog for web robots isn't exactly normal. related...
This update puts us at 69,429 user agents and 939 bots.

Blog Archives:












APR 18

2009

Where did Palm go?







JUL 13

2008

2,000 bots!





JUN 19

2008

And we're back...


JAN 21

2008

Happy 2nd Birthday!


OCT 25

2007

The Creature Post



JUL 22

2007

Hot Summer Bots




JUN 13

2007

Twiceler Strikes!


JUN 3

2007

Land of 1,000 bots




MAR 24

2007

March Bot Madness


MAR 10

2007

Got Spiders?




FEB 3

2007

February Stars


JAN 12

2007

Happy 1st Birthday!




DEC 12

2006

50,000 User Agents!



OCT 17

2006

2 weeks, 30 bots


OCT 2

2006

4 weeks, 23 bots



AUG 26

2006

500 bots!


AUG 20

2006

5 days, 8 bots











MAY 2

2006

A few new bots









JAN 12

2006

Day One


USER AGENT CATEGORIES

BOTS
BROWSERS
MOBILE BROWSERS
OLD SCHOOL
OTHER
SPAM / HACKS
  • Script Injections (349)
  • SQL Injection (7,699)
  • PHP Injection (1,993)
  • XSS Attacks (66)
  • Link Injection (250,869)