Our sites have been covered up in traffic from
twiceler robots this month. I finally had to break down and add an entry to our
robots.txt file to disallow them from the root. For those of you who don't know, we've always been very liberal with our robots.txt file here on Bots vs Browsers.com to encourage a variety of robots to visit us and leave their information and so we could observe bot behavior in the wild without any disturbances.
In addition, I wrote an email twiceler's address mentioned on their website asking them to stop hitting our sites. I'll make sure and post an update on their reply to see how responsive they are. The next step is for me to ban their IP and move on, but I like to give people the chance to do the right thing.
related...