By bengtan on

Using Drupal as an application framework for a massively multisite cms

ProsePoint ExpressProsePoint Express (http://www.prosepoint.net) is an online service that allows users to build their own newspaper or magazine website. It is a massively multisite cms built on top of Drupal.

The software is implemented as a collection of custom Drupal modules and uses the Drupal API for many of the functions that are traditionally provided by an application framework.

ProsePoint Express differs from the norm in several respects:

  • As a Drupal site, ProsePoint Express pushes the boundaries of what is possible. Whereas most Drupal sites are content based, the main focus of ProsePoint Express is application functionality.
  • As an SaaS webapp, ProsePoint Express is remarkable for being built on top of a CMS rather than a conventional application framework.

Hence, ProsePoint Express should be a useful data point for being a bit of an outlier. This article will present some commentary and insights gained from the building of ProsePoint Express. Hopefully they will be of interest to some people.

Throughout the rest of this article, it is useful to remember the following:

ProsePoint Express was created with ease of use, simplicity, and a 'just works' philosophy in mind. It is aimed at lay people for creating newspaper and magazine websites out of the box. This has influenced architectural and design decisions during development.

Hence, it is a streamlined system, but at the expense of full configurability or excessive user options. ProsePoint Express is focused on the 20% of features and settings that provide 80% of the value. It does not try to be a content management system that can do everything, particularly when doing everything would mean overwhelming users or require too much technical knowledge.

Having said that, newspapers and magazine websites are inherently complex. ProsePoint Express attempts to simplify as much as possible, but some baseline complexity does remain.

Description of the service

Frontpage of demo siteProsePoint Express features a separate, virtual, multisite, content management system implemented on top of Drupal. It is focused towards building and managing newspaper and magazine websites and provides commonly used features out of the box.

Lets expand on that a bit more.

  • ProsePoint Express is separate because it is a distinct cms in its own right (albeit not as full featured as Drupal) with its own content types, widgets and plugins.
  • ProsePoint Express is virtual because it does not exist on its own. It relies on the host system (in this case, Drupal) for bridging to the outside world.
  • ProsePoint Express is multisite because it natively allows for the quick and easy creation of many sites, each with its own web address.

(For the purposes of avoiding ambiguity, a user site on ProsePoint Express shall be referred to as a minisite. The pages of a minisite shall be referred to as mininodes.)

ProsePoint Express features a custom node type called site. A site node contains all the settings and content for a user's minisite. Obviously, a site node contains a lot of information so this is spread over several database tables. In turn, each minisite has its own set of mininodes which comprise the pages of the site.

Essentially, ProsePoint Express comprises a 'master' website which is itself, and many subordinate minisites. However, each minisite is a fully fledged user-defined website in its own right with its own settings and content.

ProsePoint Express is a collection of about 20 (as of writing) custom Drupal modules structured in a similar way to Drupal. There is a small set of modules forming a core, and the rest of the functionality is provided by add-on modules. Drupal concepts like content types, blocks, and modules often have a direct analogy in ProsePoint Express (albeit with different names).

Scalable, efficient, secure multisite functionality

Sample story screenshotProsePoint Express was the result of a search for a solution to host many Drupal websites in a way that was massively scalable, efficient and secure for use by untrusted third parties. Existing methods didn't meet this criteria or would not scale to the level we were planning for. In the end, however, we didn't meet the initial objectives either, and diverged from hosting Drupal sites to hosting Drupal-like sites. The minisites created by ProsePoint Express, while they look Drupal-ish, are not standalone Drupal websites.

Here are some comments regarding the solutions that we did look at.

Solutions (including Aegir) that used Drupal's native multisite feature (of having separate sites directories) were not secure across sites. There is no way to stop code in one site from accessing another site. The only way such a solution would work if all the site owners were trusted and knew each other.

Solutions that deployed apache-style virtual hosts for each site were clunky and troublesome to deploy and administer on a large scale. Each virtual host requires non-negligible resources that, whilst individually small, quickly add up after the fist hundred or so sites. We were doubtful about the ease of scalability past this point.

Domain.module doesn't provide true multisite functionality. It just presents different subsets of the same site. For example, you can't reuse the same url alias across different sites. The core contact form displays the same contact categories across all sites and cannot be changed. Whilst Domain is adequate for many use cases out there, it wasn't for ours.

(Please note these comments were true at the time of investigation, but things may have changed since then.)

ProsePoint Express achieves multisite functionality by including a code snippet into settings.php which implements the functions custom_url_rewrite_inbound() and custom_url_rewrite_outbound(). For a given http request, the HTTP_HOST value is inspected and used to find which minisite is to be served. If a minisite is found, the inbound and outbound urls are rewritten accordingly. Everything is done internally within Drupal and there is no need to populate the filesystem with declarative files.

Modules not used

Sample layout demoProsePoint Express contains a sprinkling of standard Drupal modules and a lot of custom modules. Partially, this is because there weren't any modules available that provided the functionality we needed. This is understandable. Creating a content management system within Drupal is quite a niche requirement.

However, we didn't try very hard in re-using too many standard Drupal modules (such as CCK or Views) to build the site. Why?

From our experiences in building Drupal websites and the ProsePoint distribution, we have developed the view that it is very difficult, if not almost impossible, to build a tight, well executed, quality website application on Drupal using standard modules.

Most of the well done Drupal sites out there have a lot of custom code, either as custom modules for specialised functionality, or a lot of site-specific themeing, or both.

It is possible to build a functional Drupal website with standard modules, but you'd only get 80% of the functionality and ease-of-use that you want. Drupal modules, in order to be generic and reusable, have to compromise and be adaptable to a wide variety of operating requirements. In trying to cover many things, it is hard to cover a few things well.

This is why a lot of Drupal modules have to be over-configured in order to be used on a site, instead of having modules that just work and works well. Everyone's idea of what constitutes 'works well' is different because their situation is different, and it sure won't be exactly the same as yours.

When there are modules written by unrelated parties that interact with each other, this makes the end result even more loosely coupled and less integrated.

(Another factor which excerbates this issue is using modules for purposes they were never designed for, but this is true for all software.)

If an 80% website is adequate, then just use standard Drupal modules with a minimum of themeing.

However, in the cut-throat world of webapps and fleeting user attention spans, 80% is not adequate. To get to the 90%, 95% or 99% level, you need site-specific functionality and customisations. It is not enough to have a website that works. It is necessary to have a website that works well.

There is a comment I came across which expresses this succintly.

CMS solutions are good for only one thing: giving people a bunch of boilerplate functionality when that's all they can afford. Building a product on top of a CMS is a sure route to a mediocre product surrounded by dead-ends.
          - http://news.ycombinator.com/item?id=1989177

Whilst my experience with Drupal hasn't been as bad, I understand where the author is coming from. In the past, I have run into dead-ends before, but successfully worked my way out each time with custom modules or themeing.

Hence, when building ProsePoint Express, we already expected to be writing most of the functionality as custom modules since we wanted a 'much better than 80%' application (and we know Drupal internals well enough). This has turned out to be a good decision so far.

Modules used

Story carouselsSince ProsePoint Express is mostly custom code, we only use a handful of standard Drupal modules. They tend to be either self-contained modules we can just bolt on, or modules with an API which are invoked by the custom modules.

Here is the list of modules:

Some comments:

We programmatically created imagecache presets with widths from 20 pixels to 940 pixels in 10 pixel increments. With some other presets for other purposes, there are over 120 presets in total. Imagecache works fine.

We initially developed ProsePoint Express on Drupal 6.x, but then moved to Pressflow in order to use Varnish. Varnish has been great.

There is no CCK or Views. We're trying to resist using these modules on the master site. From past experience, once one of these modules is used, the kitchen sink also goes in. So far we've been able to keep the floodgates closed.

Website Builder is the collection of custom modules that comprises the newspaper cms in ProsePoint Express. It has a modular architecture and the base cms can be used to create a website builder for other industry segments.

Node subscriptions is an Ubercart module we wrote to suit our billing model.

The subscriptions billing system

Billing was an interesting problem. We could have written our own billing system, but it would have been a lot of work and it wasn't a strategic competitive advantage to do so. Ubercart already does ecommerce, so we tried to use that.

Initial attempts to force Ubercart into submission didn't work. The problem with using Ubercart for subscriptions billing is that ... well, Ubercart is designed for retail, and subscriptions is different from retail.

The workflow is different. The concepts are different. There is no such thing as a cart in subscriptions. Users don't get to choose what 'subscriptions' to add to a cart. Either they pay for all their subscriptions for a billing cycle, or they don't. There is no pick and choose.

We went back to the drawing board.

Our billing model was to charge users for each minisite on a monthly cycle. Since minisites were nodes, we wanted to charge per node per month.

(Without roles. I vaguely remember there are some Ubercart pay-per-node modules that used roles. For ProsePoint Express, that just adds unnecessary and troublesome complexity. We're not charging for the ability to edit some nodes, we charge for the node itself. Roles don't enter into the picture at all.)

Every month, there should be an order generated for each user which lists their node subscriptions for that month. And the user needs to be able to pay them. Ubercart expects users to pay during checkout, at the point of order creation. For subscriptions, this is wrong. There is no checkout stage. Orders are generated, and then they paid afterwards.

We wrote a new custom module Node Subscriptions to implement this billing model. Besides generating renewal orders every cycle, it also allows retrospective paying of outstanding orders by users. Retrospective payment replaces the normal checkout/review/submit flow of Ubercart's retail sales model, and does away with the need for Ubercart's core Cart module. ProsePoint Express operates with the Cart module disabled.

Using Drupal as a web application framework

Our initial reason for building ProsePoint Express on Drupal was because we wanted to make Drupal scale massively and securely to many sites on a single vhost. Once that was deemed infeasible, we amended our objectives to create many Drupal-like minisites instead, so we kept Drupal as the platform.

Using Drupal as a framework has given us some freebies and saved us a lot of work. These include, but are not limited to:

  • User accounts
  • Image and flash handling
  • Ecommerce and billing
  • Ticketing system
  • High performance caching (Varnish)
  • Forms API
  • Modular architecture
  • Themeing infrastructure
  • Tried and tested code design patterns for reuse

As we add more features to ProsePoint Express, we will continue reusing standard Drupal modules (if suitable), or borrowing their code. It has been much more productive learning from how Drupal implements a particular feature or borrowing the same technique, than always coming up with something from the ground up.

If we had used another web framework, we would likely have had some freebies too, but likely not as much. What other framework provides an ecommerce and ticketing system that can be dropped in?

There is a caveat though. Using Drupal as a framework only works if you know Drupal very well. There is a bit of a learning curve for development (which is different from a learning curve as an end user). Be prepared to have custom modules written for strategically important features if you want a 'better than 80%' website.

ProsePoint Express has been an interesting case study in building a webapp using Drupal as a framework. Drupal has always been touted as more than just a content management system. It's flexiblity and extensibility results in a platform upon which other arbitrary applications can be built. ProsePoint Express is just an example of this.

Credits

ProsePoint Express is a project of ThinkLeft built by bengtan in Perth, Australia.

⋅ Categories: Drupal 6.x