The Bot Project - open source community bot tracking initiative
jaxta - May 8, 2008 - 10:57
As the title says, it's an open source, community driven bot tracking site that aims to build a bigger picture of search engine bot activity.
The Bot Project is a Drupal site obviously with some custom code for the bot tracking stuff. Most of the modules used are standard as is the tempate (with a few minor mods). Charting uses Open Flash Chart (http://teethgrinder.co.uk/open-flash-chart/).
There will be a Drupal module to view your site(s) stats from within your site - we hope to have that completed in June.
We will have .NET clients available soon but for now, there's just a PHP client available.
Feedback and comments welcome.
really useful project
I've had a hand building this and you'll be staggered by whats indexing, spidering AND scraping your site.
The amount of "experimental" search bots thats are out there. Well anyway, if nothing else it will make you realise your little robots.txt needs love and attention too.
I'd try it out on a few of your sites to begin with to see what results you get.
Matt
www.interadvertising.co.uk
www.crawlscore.com