Bots vs Browsers - Public Bot / User Agent Database & Commentary

Bots vs Browsers - database of 1,566,166 user agents and growing


AUG 3

2008

Here comes August...



The last couple of weeks have been pretty routine around here. In our logs, we've sifted through a growing number of script injection hacks via user agent, some if which are getting quite creative with their HTML markup and JavaScript technique. We've also seen some new bots, and some old bots that are very active as of late.

Here's what we've seen:

  • In close relation to the robot WALL-E, his cousin swish-e turned up in our logs this week. Instead of cleaning up 700 years of trash on what is left of Earth, swish-e is an open source system for indexing web pages. The acronym "swish-e" stands for "Simple Web Indexing System for Humans - Enhanced". related...
  • More and more user agents for the Nintendo Wii have been showing up lately, so we decided to create a new category for this. The Nintendo Wii category is our latest category tracking user agents, so check it out to see any WII-related user agents that we've encountered in our logs. related...
  • We've noticed a great deal of activity recently from Russian search engine bot Yandex - in particular, from IP 77.88.25.28. They hit our sites over 15,000 times today, but the concentration per domain name and over time intervals was not quite enough to put them in range of being banned. In the two years that we've been tracking this bot, we've never seen the traffic rates this high. We'll keep a close eye on them over the next few weeks, as this trend may affect our readers as well. related...
  • Several of our consulting clients as of late have needed help ridding their sites of SQL injections. One thing that we have noticed in common when running our cleanup tools on their databases is that most of them end up with a table on their database called "t_jiaozhu". After Googling the term, we have found countless others with the same story - SQL injection hack, table created with weird name "t_jiaozhu". The point is, make sure your site is SQL injection attack-proof. If you think you may have been hacked or just aren't sure, check your database for the table "t_jiaozhu".

    Depending on your web architecture, be sure that all SQL calls are scrubbed either through common framework level cleansing or by home-grown means. Once you have secured your database from SQL injection, make sure you don't forget to protect yourself from script injection attempts as well. We've documented many of these script injection attempts that appear in user agents here.

    Anyways, off the soap box and back to the bots! related...
  • BobCrawl/Nutch-0.9 is a new form of Nutch that appeared in our logs, claiming to be a "Test/Development crawler". On a side note, in an effort to inform us that its URL and email are not available, they mis-spelled this in the user agent and put in "notavalable".

    Leave it to us to get caught up in the details. related...
  • Flatland Industries sent their web spider flatlandbot. Their website claims that the bot follows robots.txt exclusion standards, so if you don't want them around, be sure to let them know. related...
  • Here's a strange one - Blubberlutsch/1.0. There is absolutely no information currently on Google for this user agent. Results vary from site to site from "Donald Duck" to "Star Wars Attack of the Clones". My best guess is that it's German slang for something, so in the meantime, we'll call it a bot, and check back on it later. related...
  • We had our first close encounter / UFO sighting this week - UFO/77.7 (CoSMoS; Z; Pearl 256; peep) F!R3F0>< P\/\/NS y0!. related...
  • Another new search engine bot appeared from Isidorus/2.0. related...
  • Quite possibly the strangest user agent of the year, Nintendo64/1.0 (SuperMarioOS with Cray-II Y-MP Emulation) paid us a visit recently. related...
Opening up August, we have 192,894 user agents and 2,070 bots. Thanks for dropping by, and remember - only you can prevent injection hacks!

Blog Archives:


FEB 29

2016

Leap Day Launch!











APR 18

2009

Where did Palm go?







JUL 13

2008

2,000 bots!





JUN 19

2008

And we're back...


JAN 21

2008

Happy 2nd Birthday!


OCT 25

2007

The Creature Post



JUL 22

2007

Hot Summer Bots




JUN 13

2007

Twiceler Strikes!


JUN 3

2007

Land of 1,000 bots




MAR 24

2007

March Bot Madness


MAR 10

2007

Got Spiders?




FEB 3

2007

February Stars


JAN 12

2007

Happy 1st Birthday!




DEC 12

2006

50,000 User Agents!



OCT 17

2006

2 weeks, 30 bots


OCT 2

2006

4 weeks, 23 bots



AUG 26

2006

500 bots!


AUG 20

2006

5 days, 8 bots











MAY 2

2006

A few new bots









JAN 12

2006

Day One


USER AGENT CATEGORIES

BOTS
BROWSERS
MOBILE BROWSERS
OLD SCHOOL
OTHER
SPAM / HACKS
  • Script Injections (300)
  • SQL Injection (7,697)
  • PHP Injection (1,975)
  • XSS Attacks (58)
  • Link Injection (250,863)