By flipp3r on

My site has been online for 4 years, so perhaps that's part of it, but I was getting hit by bots almost non-stop last week. I would have up to 1000 "guests" on the site, so you know that allot had been hitting the site. I run my server at home, so anything that is sucking up bandwidth is going to get my attention, but this was just wasteful, and rude. I have been doing some 'deny from ...' lines in my .htaccess, but with dynamic IPs it just wasn't keeping up. Now I bots by detecting bogus (or known spammer) user agents via Apache I now have gotten that down to 20-30 at any time. Now when I tail my httpd-access.log it's nice and calm! ;) I did this within my httpd.conf file in Apache, but if you don't run your own server, you can add to the .htaccess file, learn how to do it here:

http://www.jcornelius.com/articles/deny-apache-user-agent/

f

Comments

butler’s picture

butler commented

Hi and thanks for the link... I am not the worlds greatest programmer but just spent a bunch of time researching email obfuscation, eventually coming up w/ a bastardized little php app that combines several methods. No sooner did I finish with that than the idea of denying access via user agent kind of hit me and I wound up searching Drupal, finding your post. Perhaps the way to approach fighting spam bots which are overwhelming the internet via a swarm modus operandi is to swarm back... and a community like this one could swarm. What I mean to say is that the downfall of deny access schemes is, as you mention, dynamically generated info-it won't be long before the bot coders start applying that to user agent names. It seems to me that dynamism is the key to a solution... there must be a way to dynamically generate a list of bad user agents from MySQL which could be updated by a large community on a momentary basis, and applied dynamically via PHP/Apache. I am probably not really up to more than a contributory role, but, could something like this become part of Drupal? Anybody think it would be worth making this into a topic?