Crawler Rate Limit allows you to limit requests performed by web crawlers, bots and spiders. (Option to additionally rate limit non-bot requests (regular website traffic) has been added in version 3.0.

There are number of crawlers, bots and spiders that are known for excessive crawling of websites. Such requests increase the server load and often affect performance of the website for regular visitors. In extreme cases they can even bring the site down.

This module detects if the request is made by the crawler/bot/spider by inspecting the UserAgent HTTP header and then limits number of requests crawler is allowed to perform in the given time interval. After the limit is exceeded server will respond with HTTP code 429 (Too many requests). Crawler is unblocked at the end of that time interval and new cycle beings. If crawler exceeds the rate limit again, it will again be blocked for the duration of that time interval.

Features

  • targets only web crawlers, bots and spiders; regular browser traffic is not affected
  • version 3.0 can optionally rate limit regular website traffic (non-bots)
  • number of allowed requests in a given time interval is configurable
  • limits requests based on the UserAgent string - if the same crawler uses multiple IP addresses all those requests will count towards the single limit
  • minimal effect on performance
    • rate limiting is performed in a very early phase of request handling
    • uses Redis as a rate limiter backend (version 2.0)
    • uses Redis, Memcached or APCu as a rate limiter backend (version 3.0)

Requirements

Installation

Crawler Rate Limit manages its dependencies via composer. Just copying the module into the modules folder won't work. Use composer to install the module along with all the dependencies.

cd /root/of/your/drupal/project
composer require drupal/crawler_rate_limit

Configuration

Refer to the README.md file that comes with the module.

How to update to version 3

New features introduced in the version 3 require changes to the module configuration in settings.php file. Version 3 is backward compatible with versions 1 and 2 and it will continue to work with old settings. However, it's recommended to update settings in order to take advantage of the new features. For details how to update refer to the README.md file that comes with the module.

Supporting organizations: 
Development and maintenance

Project information

Releases