Bots vs Browsers - Public Bot / User Agent Database & Commentary

Bots vs Browsers - database of 16,271,499 user agents and growing


JUN 13

2007

Twiceler calls off the bots



Twiceler was more than helpful with my request to slow down on our sites. Within half an hour of my email, here is their reply:
Sorry for any inconvenience. We should not be crawling your sites that fast. I will stop crawling everything in your IP space immediately.

I will check that we understand your robots.txt, but robots.txt often has significant time delay as the standard says that you should cache the robots.txt file for seven days.

Thanks, twiceler! That was surprisingly helpful, and a very fast response.

related...

Blog Archives:

MAY 19

2021

Goodbye Internet Explorer

MAR 17

2016

User Agent Debuts for Brave and Servo Browsers

JAN 20

2016

10 Years Old, and Big Changes for Bots vs Browsers!

JAN 6

2016

Internet Explorer : Out with the old, in with the new!

DEC 11

2015

Microsoft Edge browser extensions on the way!

DEC 13

2013

Are the Bots beating the Browsers???

FEB 3

2013

Tablets, Slates, and Touch-enabled Devices - Part 1 : The iPad

JAN 15

2013

7 Years Old, One Million User Agents!

JAN 12

2010

Happy 4th Birthday, Bots vs Browsers!

JUL 12

2009

300,000 User Agents!

MAY 31

2009

Summer Bots make a splash!

APR 18

2009

Where did Palm go?

FEB 18

2009

Happy Belated 3rd Birthday to Us!

NOV 23

2008

Bots should be thankful

OCT 22

2008

Would you say I have a plethora of new bots?

SEP 6

2008

200,000 User Agents!

AUG 3

2008

Here comes August...

JUL 13

2008

2,000 bots!

JUL 6

2008

Dog Days of Summer Bots

JUN 23

2008

Another one bites the dust

JUN 22

2008

Bots don't run out of gas

JUN 19

2008

And we're back...

JAN 21

2008

Happy 2nd Birthday!

OCT 25

2007

The Creature Post

SEP 7

2007

iTunes, Facebook, Google Earth, and more

JUL 22

2007

Hot Summer Bots

JUL 6

2007

The Apple iPhones have landed!

JUN 13

2007

Twiceler calls off the bots

JUN 13

2007

Twiceler Strikes!

JUN 3

2007

Land of 1,000 bots

APR 22

2007

April Bots Bring May Traffic... (we hope)

APR 16

2007

Spring Cleaning in the Web Traffic Logs

MAR 24

2007

March Bot Madness

MAR 10

2007

Got Spiders?

FEB 24

2007

Some New Faces (bots, of course)

FEB 10

2007

Winks, DataFountains, Hijacks, and Tool - what's the connection?

FEB 3

2007

February Stars

JAN 12

2007

Happy 1st Birthday!

JAN 1

2007

The 2007 bots are in!

DEC 23

2006

Have a Merry-Bot-Christmas

DEC 12

2006

50,000 User Agents!

NOV 4

2006

2 weeks, 19 new bots

OCT 17

2006

2 weeks, 30 bots

OCT 2

2006

4 weeks, 23 bots

SEP 9

2006

Playing catchup with the bots

AUG 26

2006

500 bots!

AUG 20

2006

5 days, 8 bots

AUG 15

2006

Two weeks, 15 new bots

AUG 2

2006

Another week, another gang of new bots

JUL 25

2006

40,000 User Agents!!!

JUL 4

2006

New category, bot revisions, and 6 new bots for July 4th

JUN 25

2006

New batch of imports!

JUN 18

2006

5 new bots, just in time for Father's Day!

JUN 4

2006

8 new bots for the database

MAY 19

2006

A couple new sightings

MAY 9

2006

Quiet week, one new bot

MAY 2

2006

A few new bots

APR 20

2006

A quick bot-sighting update

APR 12

2006

More user agents, more bots

MAR 19

2006

5,000+ New User Agents

MAR 9

2006

42 more bots, lots of new user agents

FEB 19

2006

New Bots and Browsers just addded!

JAN 25

2006

Welcome to Bots vs Browsers!

JAN 17

2006

Welcome to the User Agents Database that has no name yet!

JAN 12

2006

Day One

USER AGENT CATEGORIES

BOTS
BROWSERS
MOBILE BROWSERS
OLD SCHOOL
OTHER
SPAM / HACKS
  • Script Injections (349)
  • SQL Injection (7,701)
  • PHP Injection (1,993)
  • XSS Attacks (66)
  • Link Injection (250,869)