Crawler Rate Limit allows you to limit requests performed by web crawlers, bots and spiders. (Option to additionally rate limit non-bot requests (regular website traffic) has been added in version 3.0.
There are number of crawlers, bots and spiders that are known for excessive crawling of websites. Such requests increase the server load and often affect performance of the website for regular visitors. In extreme cases they can even bring the site down.
This module detects if the request is made by the crawler/bot/spider by inspecting the UserAgent HTTP header and then limits number of requests crawler is allowed to perform in the given time interval. After the limit is exceeded server will respond with HTTP code 429 (Too many requests). Crawler is unblocked at the end of that time interval and new cycle beings. If crawler exceeds the rate limit again, it will again be blocked for the duration of that time interval.
Features
- targets only web crawlers, bots and spiders; regular browser traffic is not affected
- version 3.0 can optionally rate limit regular website traffic (non-bots)
- number of allowed requests in a given time interval is configurable
- limits requests based on the UserAgent string - if the same crawler uses multiple IP addresses all those requests will count towards the single limit
- minimal effect on performance
- rate limiting is performed in a very early phase of request handling
- uses Redis as a rate limiter backend (version 2.0)
- uses Redis, Memcached or APCu as a rate limiter backend (version 3.0)
Requirements
- version 2.0 of the module requires Redis Drupal module
- version 3.0 supports multiple backends: Redis, Memcached and APCu. Either one of those three backends is required.
- jaybizzle/crawler-detect
- nikolaposa/rate-limit
Installation
Crawler Rate Limit manages its dependencies via composer. Just copying the module into the modules folder won't work. Use composer to install the module along with all the dependencies.
cd /root/of/your/drupal/project
composer require drupal/crawler_rate_limit
Configuration
Refer to the README.md file that comes with the module.
How to update to version 3
New features introduced in the version 3 require changes to the module configuration in settings.php file. Version 3 is backward compatible with versions 1 and 2 and it will continue to work with old settings. However, it's recommended to update settings in order to take advantage of the new features. For details how to update refer to the README.md file that comes with the module.
Project information
- Module categories: Administration Tools, Performance, Security
- 448 sites report using this module
- Created by vaish on , updated
- Stable releases for this project are covered by the security advisory policy.
Look for the shield icon below.
Releases
Supports mulitple backends (Redis, Memcached, APCu) and rate limiting of non-crawler requests. Requires update to settings.php in order to enable new features.
Same functionality as version 1.0 with added support for Drupal 10