Springtime brings spring cleaning, so we're going to try and clean out the user agent logs since it's been 3 weeks (or more) since our last post. Here's what we found lurking around in our absence:
-
As far as mythical creatures in the traffic logs, we saw a
gnome-vfs/2.16.2 neon/0.25.4
digging around and a
Yeti/0.01
crawling through. The Yeti's user agent told us that it "check robots.txt daily and follows it", and is a variation of Naver.
related...
-
We added 2 variations of the Larbin Web Crawler to our database this week -
here's one
and here's the other.
related...
-
This one was unique - the facebookscraper/1.0
visited our sites, even though we are not facebook. I guess it got lost and scraped up on some other sites?
related...
-
Someone is actually keeping a meta tags directory and is harvesting their content using
metatagsdir/0.7.
Seems like an odd hobby, but then again, running a blog for web robots isn't exactly normal.
related...
This update puts us at
69,429 user agents and
939 bots.