Springtime brings spring cleaning, so we're going to try and clean out the user agent logs since it's been 3 weeks (or more) since our last post. Here's what we found lurking around in our absence:
As far as mythical creatures in the traffic logs, we saw a
digging around and a
crawling through. The Yeti's user agent told us that it "check robots.txt daily and follows it", and is a variation of Naver.
We added 2 variations of the Larbin Web Crawler to our database this week -
and here's the other.
This one was unique - the facebookscraper/1.0
visited our sites, even though we are not facebook. I guess it got lost and scraped up on some other sites?
Someone is actually keeping a meta tags directory and is harvesting their content using
Seems like an odd hobby, but then again, running a blog for web robots isn't exactly normal.
This update puts us at 69,429
user agents and 939