Performance and Scalability

Boost

Boost provides static page caching for Drupal enabling a very significant performance and scalability boost for sites that receive mostly anonymous traffic. For shared hosting this is your best option in terms of improving performance. On dedicated servers, you may want to consider Varnish instead.

Hierarchical Select

Description

This module defines the "hierarchical_select" form element, which is a greatly enhanced way for letting the user select items in a hierarchy.

Hierarchical Select has the ability to save the entire lineage of a selection or only the "deepest" selection. You can configure it to force the user to make a selection as deep as possible in the tree, or allow the user to select an item anywhere in the tree. Levels can be labeled, you can configure limit the number of items that can be selected, configure a title for the dropbox, choose a site-wide animation delay, and so on. You can even create new items and levels through Hierarchical Select!

For a good overview of what Hierarchical Select can do, look at this demo!

I'm focusing on maintaining only the Drupal 7 version of Hierarchical Select. That means I'll only commit RTBC patches for the D6 version relatively quickly. If a co-maintainer steps up to help maintain the Drupal 6, I'd be happy to accept him/her! :)

Drupal 7 version

The Drupal 7 version integrates with:

• Taxonomy (Drupal core)
• Menu (Drupal core)

Memcache API and Integration

Integration between Drupal and Memcached with the following features:

• An API for using Memcached and the PECL Memcache or Memcached libraries with Drupal.
• Memcache backends for the following systems (all drop-in):
• Caching (memcache.inc)
• Sessions (memcache-session.inc)
• Locking (memcache-lock.inc)
• A module that provides a comprehensive administrative overview of Drupal's interaction with Memcached and stats.
• A set of tests that can be run to test your memcache setup.

Varnish HTTP Accelerator Integration

This module provides integration between your Drupal site and the Varnish HTTP Accelerator, an advanced and very fast reverse-proxy system. Basically, Varnish handles serving static files and anonymous page-views for your site much faster and at higher volumes than Apache, in the neighborhood of 3000 requests per second.

This module provides admin-socket integration which allows Drupal to dynamically invalidate cache entries, and also lets you query the Varnish admin interface for status, etc.

Compatibility

At the moment, using Varnish to any effect at all requires Drupal 7 or PressFlow.

We currently support Varnish 2.0, 2.1 and 3.

Entity cache

Entity cache puts core entities into Drupal's cache API.

Due to the entity loading changes in Drupal 7, no core patches are required.

How to use it

This module has no user interface, if you enable it, entities will start being cached, if you disable it, they'll stop.

Supported core entities

• Nodes
• Comments
• Taxonomy terms
• Taxonomy vocabularies
• Files
• Users

Support for contrib entities:

There is no automated support for contrib entities, to use entitycache for a contrib or custom entity:

• Create a cache_entity_$myentity table/bin.
• Specify EntityCacheDefaultEntityController as your controller class in hook_entity_info(), or extend it.
• Never update your entity tables outside of save/delete API calls.

Tests

Entity cache is now set up for automated testing on Drupal.org, see current HEAD status.

This module is being developed for Drupal 7, and will never be backported to Drupal 6. For Drupal 6 you may want to look at this pressflow merge request or http://drupal.org/project/advcache.

Javascript Aggregator

The aim of the JavaScript Aggregator module is to improve performance of your site with less server requests and bandwidth per page. In Drupal 5, all the JavaScript files will be aggregated into one file and optionally minified. JavaScript aggregation was brought into core with Drupal 6, so the Drupal 6 version of this module goes one step further to minify that file.

It basically does the following:

• Takes the $scripts variable and removes .js files using a regular expression
• Generates a unique filename for any set of .js files
• Creates and caches one file containing all .js files in files/js
• Optionally minifies the JavaScript using JSMin
• Returns a modified $scripts variable with link to cached file

Elysia Cron

!NEW Release 2.0 is finally out, with D7 support, code & performance improvements and a lot of new features!

Elysia Cron extends Drupal standard cron, allowing a fine grain control over each task and several ways to add custom cron jobs to your site.

• [NEW IN 2.0] Set the timings and frequencies of each cron task (you can run some jobs every day at a specified hour, other only monthly and so on...). For each task you can simply choose between some frequently used options ("once a day", "once a month" ...), or use a powerful "linux crontab"-like syntax to set the accurate timings. You can even define your frequently used options to speed up site configuration.
• Parallel execution of cron task: you can group jobs in channels and execute then simultaneously: so a task that takes a lot of time to execute won't block other tasks that need to be executed every 5 minutes...
• You can disable all tasks, an entire channel or a single task.
• Change the priority/order of task execution.
• Manual force the execution of a cron tasks.
• Detailed overview of cron status with time statistics for single tasks and channels.
• [NEW IN 2.0] powerful API for module developers: you can define extra cron tasks for your modules, each one with own default timings (site administrators can override them by configuration, other modules via hook_alter). Elysia Cron 2.0 gives a brand new API support (compatible with 1.0 version) with a lot of features.
• Administrators can define custom jobs (call to functions with parameters), via the "script" option.
• Several optimization for frequent cron calls and error handling.
• Protection from external cron calling by cron_key or allowed host list.

Elysia has no dependencies with contributed modules, and doesn't need to patch the core: it can be used in minimal Drupal installation with only core modules.
It also can be used in a Drupal install profile.

3rd party integration:

• [NEW IN 2.0] Ping feature, for external tracking services like host-tracker to tell whether cron is functioning properly on your site.
• [NEW IN 2.0] Drush support: you can call "drush elysia-cron" to manually run extended cron.
• [NEW IN 2.0] CTools support for exports/backup of task settings.
• [NEW IN 2.0] Features support.

CSS Gzip

CSS Gzip will no longer be updated/supported and is considered obsolete. All future development is being done in the Advanced CSS/JS Aggregation module.

Node Hierarchy

Node Hierarchy allows nodes to be children of other nodes creating a tree-like hierarchy of content. The module offers:

• 1-Click creation of hierarchical menus
• Hierarchical breadcrumbs
• Automatic hierarchical urls using Pathauto (and token module)
  (eg: q=aboutus/history/beginning).
• Automatic creation of hierarchical menus if desired.
• Optional Views integration.
• Optional Node Access integration.

HTTP Parallel Request & Threading Library

This is a library module. It provides no out of the box functionality other then providing an API that other modules/code can use. Other projects might require/recommend this module. Install HTTPRL only if other modules recommend or require it.

What does httprl do?

Using stream_select() it will send http requests out in parallel. These requests can be made in a blocking or non-blocking way. Blocking will wait for the http response; Non-Blocking will close the connection not waiting for the response back. The API for httprl is similar to the Drupal 7 version of drupal_http_request().

As a bonus, a simple threading library is built on top of the parallel http requests functionality. This allows you to split a job and have multiple http "workers" running this split job in parallel. Anything that takes a long time to do can greatly benefit from using threads.

Benchmarks

Modernizr

Modernizr tests which native CSS3 and HTML5 features are available in each browser and makes the results available to you in two ways: as properties on a global Modernizr object, and as classes on the <html> element. This information allows you to progressively enhance your pages with a granular level of control over the experience.

This Drupal module provides deep integration with the Modernizr JS library, allowing other modules or themes to register tests, load additional assets as needed, and even create new copies of the Modernizr library when a website's requirements change. Read more below.

APC - Alternative PHP Cache

The Alternative PHP Cache (APC) is a free and open opcode cache for PHP. Its goal is to provide a free, open, and robust framework for caching and optimizing PHP intermediate code. Besides a opcode cache it provides a user cache for storing application data. This module uses the APC user cache as a cache backend for Drupal.

Use APC for caches that do not change often and will not grow too big to avoid fragmentation. The default setting of APC will allow you to store 32 MiB for the opcode cache and the user cache combined. Make sure you tweak this according to your website's needs. An example configuration could be to cache 'cache' and 'cache_bootstrap' in APC; 'cache_field' and 'cache_menu' in Memcached and store 'cache_filter' in the database

Drupal 6

The APC backend can be run for drupal 6 using cache backport.

Maintainers

• Drupal 7: R.Muilwijk (sponsored by Trinoco)
• Drupal 5&6: Slantview

Installation instructions

Cache Router

About

CacheRouter is a caching system for Drupal allowing you to assign individual cache tables to specific cache technology. CacheRouter has an option to utilize the page_fast_cache part of Drupal in order to reduce the amount of resources needed for serving pages to anonymous users.

Drush integration #642544-12: Solution for Drush integration problem

NO DRUPAL 7 VERSION OF THIS MODULE WILL BE RELEASED. See #593238: Do not release Drupal 7 version of this and move the engines into separate projects for details.

Cache Router supports the following technologies natively:

• APC
• Database
• eAccelerator
• File
• Memcache
• Memcached (Alpha)
• XCache

Node.js integration

This module integrates Node.js with Drupal.

It provides an API that other modules can use to add realtime capabilities to Drupal, specifically enabling pushing updates to open connected clients.

Block Cache Alter

Alter cache settings per block. Cache settings per block are now set in code, but if you don't like the default - usually none - you can now easily change this. Install this to speed up block rendering for authenticated users.

The Drupal 6 version works again after #1173012: Blocks lose settings during update.php and cache clears got committed to Drupal core. A patch is available here #1194880: _block_rehash() overwrites the caching mode set by Block_Cache_Alter module and is also in the 6.x-1.5 release.

Drupal Queue

Backport of Drupal 7 Queue API.

Note: if you use Drupal Queue, make sure to configure Drupal cron and Drupal Queue's own cron entry point. See README.txt for details.

Drupal Queue Drupal 7 issues

• #391340 Job Queue API
• #578676 Use queue for cron

Based on an initial patch to Drupal 7 core by chx, dww, neclimdul et. al.

CSS Embedded Images

How does it work?

According to Wikipedia, "the data URI scheme is a URI scheme that provides a way to include data in line in web pages as if they were external resources." That means that rather than having a URL telling the browser where to find the image you actually have the image data right there on the page so the browser does not have to make an extra request to get it. This particular module is only concerned with images referenced in CSS, so for an example let's look at a declaration in Garland's CSS:

When we embed the image into the CSS, it becomes:

The actual process is a bit more involved than that, but this is the general concept. By default, embedded images are placed in a separate CSS file to allow for parallel rendering of other site styles while the relatively large image data is being downloaded.

DB Tuner

Wondering why your views run so slow? Why after switching to InnoDB, MySQL isn't running any better? This module has the answers!

Indexes

It will show what CCK columns get used in a view filter or relationship and give you the option to add an index on it so the views query will run faster. Also can add indexes to core modules that could use one.

Engine

Ability to convert your database from MyISAM to InnoDB in order to eliminate table locks on data writes.

Collation

Are half your tables latin1_swedish_ci and the other half utf8_general_ci? Use this to change them all to utf8_general_ci.

MySQL Tuner

Have you never edited your my.conf file? This will give you hints at what values to tweak in order to get MySQL running better. Special thanks to MySQL Tuner, since this part of the module is a port of that project.

Future Plans:

• Figure out which experimental indexes are good & which ones are bad
• #798756: Better index suggestions with filters and sorts. Manage Multi Column Indexes
• #799230: Run explain on every views query
• #799236: Run explain on every query in the slow query log
• #817688: Engine Selection Per Table; Suggest you put site into maintenance mode

Similar Projects

MySQL Report
Shadow - SQL queries / views optimization
Materialized View API

Cache Actions

Cache actions provides rules actions for clearing drupal caches.

It currently provides actions for:

• Clearing Drupal cache bins
• Clearing CSS/JS cache
• Clearing The cache of specific views
• Clearing The cache of specific panel pages
• Clearing The cache of specific mini panels

Dependencies

Cache actions only depends on the rules module. If you don't have views or panels, you only get the option of clearing the drupal cache bins and the CSS/JS cache. When views or panels are present, you get actions for clearing the cache of specific views and panels as well.

Compatibility

This module is compatible with:

• Panels 3
• Views 2 and 3. The views 2 support is the best tested currently, but everything seems fine
  with views 3 as well.

Version 2

The new version of cache actions offers a lot of new features:

• Clearing The cache of more than one view at a time
• Clearing The cache of specific panel page variants
• Clearing The cache of specific panel page panes
• Clearing The cache of specific mini panel variants
• Clearing The cache of specific mini panel panes
• Unit tests

Authenticated User Page Caching (Authcache)

The Authcache module offers page caching for both anonymous users and logged-in authenticated users. This allows Drupal/PHP to only spend 1-2 milliseconds serving pages, greatly reducing server resources.

Please note that enabling authenticated user caching will require modifying how your user-customized content displays on your pages. You should be an experienced Drupal developer if you choose to implement the full functionality of this module.

How does it work?

Authcache saves the final rendered HTML of a page to serve visitors. A separate cache is created for each user role as defined by the administrator, so some roles can be excluded if necessary.

Authcache places priority on serving pages to the visitor as fast as possible. After a page is loaded by the browser, a second HTTP request may be performed via Ajax. This initiates a lightweight DRUPAL_BOOTSTRAP_SESSION that allows SQL queries to be executed (such as updating the user "history" table or node "statistics" table), and returns any user-customized data to the page (such as form tokens or default values on a contact form).

What type of caching is used?

Authcache on Drupal 6

AmazonS3

The AmazonS3 module allows the local file system to be replaced with S3. Uploads are saved into the drupal file table using D7's new file/stream wrapper system.

Requirements:

• AWS SDK for PHP
• Libraries API (7.x-2.x branch)

You can switch it on as the default file system scheme, or individually for file and image fields.

Image styles are delivered first through the private file system, which generates a derivative on S3 and is served from S3 thereafter.

Also available on github.

CORS Upload
http://drupal.org/project/amazons3_cors

Requirements
You will need to set allow_url_fopen to on your PHP settings. This option enables the URL-aware fopen wrappers that enable accessing URL object like files. See http://uk.php.net/manual/en/function.fopen.php

SSL Issues
Some curl libraries, such as the one bundled with MAMP, do not come with authoritative certificate files. This might cause an error such as "There was a problem using S3". http://dev.soup.io/post/56438473/If-youre-using-MAMP-and-doing-something
This may also cause issues when using buckets with "." in their names. http://drupal.org/node/1673888

Fast 404

What and Why

Drupal has expensive 404 errors. On an 'average' site with an 'average' module load, you can be looking at 60-100MB of memory being consumed on your server to deliver a 404. Consider a page with a bad .gif link and a missing .css file. That page will generate 2 404s along with the actual load of the page. You are most likely looking at 180MB of memory to server that page rather than the 60MB it should take.

That's where Fast 404 comes in. This module combines a very common method of handling missing image/file 404 errors (discussed here and planned for Drupal 8) with a method created by dpardo (a co-maintainer of this project) to deliver super fast 404 error pages for both missing images and bad paths. Depending on which method of implementation you choose (aggressive or super aggressive) you can deliver 404 errors using less than 1MB of memory on your server.

Drupal 7 Core Updates

Drupal 7 core has updated to add a rudimentary version of what this module implements. It allows you to set an excluded set of paths, a list of extensions to Fast 404 on, as well as the plain HTML that is delivered.

Issue describing: http://drupal.org/node/76824
Patch added: http://drupal.org/files/issues/404_fast_paths_7x-76824-328.patch

Ultimate Cron

The Ultimate Cron handling for Drupal.
Runs cron jobs individually in parallel using configurable rules, pool management and load balancing.

Documentation has begun! http://drupal.org/node/1666944 Looking for help: Please contribute if you have input.

Features

• Works out-of-the box in most cases (or aims to)
• Parallel exection of cron jobs
• Configuration per job (enable/disable, rules, etc.)
• Multiple rules per cron job
• Pool management and load balancing using Background process
• Support for Drupal Queue (also in Drupal 6 using the Drupal Queue module)
• Overview of cron jobs
• Log history of cron jobs
• Status/error messages per cron job, providing easy debugging of troublesome cron jobs
• Uses hook_cronapi() (Elysia Cron compatible, NOT 2.x, please use hook_cron_alter() for similar functionality)
• hook_cron_alter() for easy adding/manipulating cron jobs
• Background process driven poormans cron with a granularity of 1 minute
• Drush support (list, start, enable/disable jobs from the command line)

Views Litepager

Note: Use the 2.x releases with Views 2 and the 3.x releases with Views 3.

The Views Litepager module solves a problem of scalability for sites with large amounts of content. Drupal's core pagination system creates a pager navigation that shows exactly how many pages of content exist for the content list. This requires that a COUNT query be executed based on the query used to generate the list.

While COUNT queries are blazingly fast on tables with MySQL's MyISAM engine, they are painfully slow when using InnoDB tables which is the recommended engine type for high traffic Drupal sites. The COUNT queries quickly degrade the more rows a table has.

The Views Litepager module solves this problem for Views pagination by providing a pager option that does not require a COUNT query to be executed. This "Lite" pager is only slightly less useful than Drupal's core pager in that it does not allow you to navigate to the "last" page and does not show how many total pages of content there are. But for large sites, this small cost in features is worth the boost in performance by ridding your pages of the painfully slow (and sometimes crippling) COUNT queries.

Previous/Next API

An API for browsing next/previous nodes without overloading your database server.

Description

This module allows you to know the previous or next nodes for any given node. This is very useful for providing navigational links to the user without the expensive queries required to dynamically deduce such information on the fly.

The use case is two fold:

Usability/Navigation

For example, on a site with a gallery of images, you want to show a next/previous link with a thumbnail under each image. Your site's visitor click on the link to show new content or browse it.

Scalability

Although the previous and next nodes can be deduced with some SQL work, the queries to do so are very heavy on the database, and can bring a site to its knees. This module solves this problem by storing the previous/next node in a table so lookups are fast. Once the module is installed, it will build this index backwards via cron until all nodes have been indexed. See the "More Info" section below for a detailed post on the positive scalability impacts of implementing this module.

Configuration

The module can be restricted to certain content types to be included in the previous/next indexing. For example, you want the site's visitors to browse through video and image nodes only, but not blogs and regular pages.