Bots vs Browsers - database of 16,273,095 user agents and growing

AUG 3

2008

Here comes August...

The last couple of weeks have been pretty routine around here. In our logs, we've sifted through a growing number of script injection hacks via user agent, some if which are getting quite creative with their HTML markup and JavaScript technique. We've also seen some new bots, and some old bots that are very active as of late.

Here's what we've seen:

In close relation to the robot WALL-E, his cousin swish-e turned up in our logs this week. Instead of cleaning up 700 years of trash on what is left of Earth, swish-e is an open source system for indexing web pages. The acronym "swish-e" stands for "Simple Web Indexing System for Humans - Enhanced". related...
More and more user agents for the Nintendo Wii have been showing up lately, so we decided to create a new category for this. The Nintendo Wii category is our latest category tracking user agents, so check it out to see any WII-related user agents that we've encountered in our logs. related...
We've noticed a great deal of activity recently from Russian search engine bot Yandex - in particular, from IP 77.88.25.28. They hit our sites over 15,000 times today, but the concentration per domain name and over time intervals was not quite enough to put them in range of being banned. In the two years that we've been tracking this bot, we've never seen the traffic rates this high. We'll keep a close eye on them over the next few weeks, as this trend may affect our readers as well. related...
Several of our consulting clients as of late have needed help ridding their sites of SQL injections. One thing that we have noticed in common when running our cleanup tools on their databases is that most of them end up with a table on their database called "t_jiaozhu". After Googling the term, we have found countless others with the same story - SQL injection hack, table created with weird name "t_jiaozhu". The point is, make sure your site is SQL injection attack-proof. If you think you may have been hacked or just aren't sure, check your database for the table "t_jiaozhu".

Depending on your web architecture, be sure that all SQL calls are scrubbed either through common framework level cleansing or by home-grown means. Once you have secured your database from SQL injection, make sure you don't forget to protect yourself from script injection attempts as well. We've documented many of these script injection attempts that appear in user agents here.

Anyways, off the soap box and back to the bots! related...
BobCrawl/Nutch-0.9 is a new form of Nutch that appeared in our logs, claiming to be a "Test/Development crawler". On a side note, in an effort to inform us that its URL and email are not available, they mis-spelled this in the user agent and put in "notavalable".

Leave it to us to get caught up in the details. related...
Flatland Industries sent their web spider flatlandbot. Their website claims that the bot follows robots.txt exclusion standards, so if you don't want them around, be sure to let them know. related...
Here's a strange one - Blubberlutsch/1.0. There is absolutely no information currently on Google for this user agent. Results vary from site to site from "Donald Duck" to "Star Wars Attack of the Clones". My best guess is that it's German slang for something, so in the meantime, we'll call it a bot, and check back on it later. related...
We had our first close encounter / UFO sighting this week - UFO/77.7 (CoSMoS; Z; Pearl 256; peep) F!R3F0>< P\/\/NS y0!. related...
Another new search engine bot appeared from Isidorus/2.0. related...
Quite possibly the strangest user agent of the year, Nintendo64/1.0 (SuperMarioOS with Cray-II Y-MP Emulation) paid us a visit recently. related...