I added a few to the default set you get with the pepper. Because you might want to know when Technorati crawls your page, or if Bloglines is getting the right pages, you can use these to check what those bots are up to.
(There’s actually no really pressing need to track every bot that hits your site. You’ll only need to see the crawlers if you’re troubleshooting or trying to figure out who’s stealing all your bandwidth.)
Default bots:
- googlebot|Googlebot
- yahoo! slurp|Yahoo! Slurp
- msnbot|MSNBot
- ask/teoma|Ask/Teoma
- ia_archiver|Alexa (ia_archiver)
- archive.org_bot|Internet Archive (archive.org_bot)
- gigabot|Gigabot (Gigablast.com)
- mozdex|Mozdex
Other names for default bots:
- jeeves|Ask.com
- slurp@inktomi|Inktomi (Yahoo!)
Other bots of varying interest:
- sphere|Sphere
- technorati|Technorati
- bloglines|Bloglines
- tailrank|Tailrank
- polar4|TTLB (The Truth Laid Bear Ecosystem)
- lycos|Lycos
- scooter|Altavista
- quantcast|Quantcast
- SBIder|SiteSell
- voyager|Kosmix (Spammer)
- fast-webcrawler|AllTheWeb
- turnitinbot|Turnitin.com
- findexa|Findexa
- findlinks|NextLinks
- gaisbo|Gais
- zyborg|WiseNut
- surveybot|WhoisSource
- blogsearch|BlogSearch
- pubsub|PubSub
- syndic8|Syndic8
- userland|RadioUserland
- become.com|Become.com
Please add any that you’d think are important or useful.
