Bots vs Browsers - Public Bot / User Agent Database & Commentary

Bots vs Browsers - database of 1,566,688 user agents and growing


DEC 23

2006

Have a Merry-Bot-Christmas



After moving to the new server, we synchronized our logs and found 69 new bots over the last few weeks, which pushed us past the 700 bot milestone!

And without further ado, the highlights:
  • Next up, YodaoBot/1.0 dropped in for a visit. We couldn't tell much about this bot, as it's home site appears to be in Japanese. However, it looks like they honor the robots.txt standard. related...
  • We ran across Twiceler in our logs, and visited their site. Their informational page was brief and to the point, saying "Twiceler is an experimental robot. Please contact costello@cuill.com if you have any problems. Twiceler should obey robots.txt.". I guess that says it all... related...
  • Dries Buytaert sent his personal bot over. According to his blog, he is a PHD student at the University of Ghent (Belgium), and lead of the Drupal project. related...
  • DepSpid/5.07, "The Dependency Spider", dropped in and provided a URL for it's site. Upon visiting, we found out (in verbatim) that "DepSpid is a distributed kind of a web crawler. The DepSpid spider visits domains, analyses links and finally calculates scores about the link dependencies between individual domains. Each spider job starts at the main page of a domain and then follows each link on that page retrieving more pages and analysing them, too. The spider stays within one domain. If it finds an external link it only checks if the linked domain is reachable but doesn't continue crawling into the external domain. Every unknown domain will be visited from another spider job at a later time. The DepSpid spider is currently under devlopment. Once it's running in production mode, the data collected by the spider will be publically available and will give webmasters a new kind of sight into their own or foreign domains." related...
We now have 51,490 user agents and 720 bots

Blog Archives:


FEB 29

2016

Leap Day Launch!











APR 18

2009

Where did Palm go?







JUL 13

2008

2,000 bots!





JUN 19

2008

And we're back...


JAN 21

2008

Happy 2nd Birthday!


OCT 25

2007

The Creature Post



JUL 22

2007

Hot Summer Bots




JUN 13

2007

Twiceler Strikes!


JUN 3

2007

Land of 1,000 bots




MAR 24

2007

March Bot Madness


MAR 10

2007

Got Spiders?




FEB 3

2007

February Stars


JAN 12

2007

Happy 1st Birthday!




DEC 12

2006

50,000 User Agents!



OCT 17

2006

2 weeks, 30 bots


OCT 2

2006

4 weeks, 23 bots



AUG 26

2006

500 bots!


AUG 20

2006

5 days, 8 bots











MAY 2

2006

A few new bots









JAN 12

2006

Day One


USER AGENT CATEGORIES

BOTS
BROWSERS
MOBILE BROWSERS
OLD SCHOOL
OTHER
SPAM / HACKS
  • Script Injections (300)
  • SQL Injection (7,697)
  • PHP Injection (1,975)
  • XSS Attacks (58)
  • Link Injection (250,863)