So it's been a while since we've posted our progress, so here goes:
We've been busy, and so have our web traffic logs - we've discovered 320 bots since our last post, and found over 31,000 other new user agents since then.
There's alot of noise in those logs, so here are the high points that we hope will interest you:
-
Yacybot has become one of our most active bots recently.
We now have 90 variations of this bot, and we have seen a fair amount of traffic from them.
These bots are from Yacy.net, a "distributed web search" that can be downloaded and run locally as "a scalable personal web crawler and web search engine".
related...
-
Sensis Web Crawler
dropped in - according to their site, "Sensis Corporation's Purpose is to provide distinctively elegant, innovative technical solutions in the service of humanity."
related...
-
Over the past two months, we've had many flavors of Nutch pass by - below is a taste of what we've seen
related...
-
Where do all the bad bots go? We found out that there is in fact a "Bot Hell"
(Bothell, WA)
located about 12 miles southwest of Seattle, Washington.
We're pretty sure it's *not* pronounced "bot-hell", but it's still funny enough. Maybe bots like
devil in disguise (v. 2.0.1a)
and HellBoundHackerOS
reside there...
-
Here are a few random bots from the last month that interested us:
-
mixi-crawler/2.00
is from Japan. I spent some time on their site, but my Japanese is non-existent, and even with the help of Google Translate, I could not figure out what the site does
related...
-
ZONGOLBOT the Web Spider is a bot from the UK search engine of the same name
related...
-
flatlandbot/allspark
is a spider from Flatland Industries. To their credit, they provide not only robots.txt support and an email to contact them at, but they also have a phone number to dial if you are having trouble with one of their bots.
related...
-
Two new versions of Zontirbot showed up recently
related...
This brings us to
252,674 user agents and
2,828 bots.