By dzhu on

Some robots or aggregators come to our site and try to load non-existing pages, for example:

/taxonomy/term/123/0/URL Required
/node/.example.com
and so on, and so on...

And there are about ten such requests in one minute, hundreds at all from one IP. Different path are tried to be loaded.

Is there some way (using module, for example) to block such IPs automatically, if it generates too many errors, like 403 error?

Or, as an alternate solution,is there a way, how to limit number of requests from one IP per minute?

⋅ Categories: Drupal 6.x

Comments

Steve Halleman’s picture

Steve Halleman commented

Under 5.x it is under Administer->Logs->Top Visitors, or www.example.com/admin/logs/visitors

dzhu’s picture

dzhu commented

I do can find top visitors, and block some of them manually, bu I actually want to find a way to block only robots, which try to access a lot of incorrect path on the site and do that automatically.

yelvington’s picture

yelvington commented

Be careful -- you don't want to accidentally ban Google's bot just because it's following bad links that might be buried on your site.

You might want to try http://drupal.org/project/badbehavior. I have not used it.

lomo’s picture

lomo commented

Some bots try to link to nonexistent content just to get an idea of how your site responds to it. But to get hundreds of them at once sounds excessive and shouldn't be done by a normal search engine bot.

As yelvington says, be careful not to block the "good bots" but it does sound like you've got some issues with bots that might be searching for vulnerabilities or other "unfriendly" activity.

See you at the Drupalcon!

dzhu’s picture

dzhu commented

I think it may be some site downloaders, misconfigured in some way. The user agent ID is: "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)", sometimes "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; MRSPUTNIK 2, 0, 1, 90 SW)", from the last such IP.

Or, is there a way, how to limit number of requests from one IP per minute?

dzhu’s picture

dzhu commented

It is interesting project, but I'm actually looking for way to do blocking according to errors in Drupal error log. Or, at least, monitor 403 errors somehow and block IPs, which generates most of that errors.

dman’s picture

dman commented

I imagine rules would be the thing for this
if user accesses a page that triggers a 403
then add their IP to the ban list
There's a lot of danger of flase positives of course - including locking yourself out permanently if you ever accidentally hit a bookmark to the admin section without being logged in.

I had a quick look at rules, but I can't see the actual triggers you'd need. May need to build them yourself. But rules.module sounds like the best structure to be doing this under.

.dan. is the New Zealand Drupal Developer working on Government Web Standards

lomo’s picture

lomo commented

But maybe you can set it so that if a user is REPEATEDLY requesting pages that result in a 403 error, then the IP is banned for a period of time. Beware that, as I said, some search engine bots that you do want to have accessing your site, will send requests for pages that don't exist, just to make sure that your site responds in a predictable way and see what your "403" looks like.

See you at the Drupalcon!