Origo - Distributed Software Development

Origo is an open, modular and extensible software development, management and distribution platform. Aimed at software developers, Origo provides a set of services for combining, integrating and facilitating the development process over a network. Origo provides services like source control management, issue tracking, statistics, release hosting, wiki, blog and features for communicating and networking with other developers. An extensive API can be used to integrate Origo into other applications and development processes.

Origo was conceived as a research project of the Software Engineering chair at ETH Zürich, and has resulted in several theses. Recently ownership was transferred into an ETH spinoff, Oriact.

Released in August 2007, the platform as of August 2009 hosts 3,000 projects with over 6,500 developers worldwide.

Origo's Features

Origo provides all the common features of a source control platform:

• Source Control Management & hosting using Subversion
• Release Hosting
• Both open- and closed source projects
• A wiki for presentation and documentation
• Blogs and Forums
• Bugtracker ("Issue Tracker")
• Community wiki pages
• Jabber server
• Networking by befriending other users
• Process tracking through workitems

A user can own or be a member of any number of projects. While all projects are separate, the user can conveniently track the progress of the projects centrally, as all actions (source commits, wiki-edits, etc.) are tracked by Origo as "workitems". Email notification for each project and each action can be configured. The concept of "simultaneous separation while centrally tracked" projects posed the biggest challenge for this site.

Why Drupal?

When looking for a front-end framework that would allow to build Origo, it was clear from the start that Origo had to be ready for the masses (because hosting a project on Origo is free), and that stringent security requirements were necessary. Therefore a solution was needed that

• keeps the design separate from the content and is thus scalable
• allows updating things in one place for all projects hosted on Origo
• takes security seriously and integrates in Origo's development process

The only CMS out there that met all three of these requirements, and additionally allowed integration of great functionality from other developer's modules (there was plenty of work to implement Origo anyway, and being able to re-use existing solutions wherever possible was most welcome) was Drupal - thanks to the Drupal community, amazing work!

Architecture & Webpage

Origo uses a two-tiered architecture: a back-end handling the business logic and most of the data, and a front-end for user interaction. An API connects both layers using XML-RPC calls. Part of the API is also open to 3rd-party applications, e.g., Origo's Eclipse plug-in. Origo's Web front-end is implemented using Drupal, including contributed and custom modules, and set up in a (huge) multi-site installation: Each hosted project gets its own Drupal site with its own database, but all sites share the same modules. This enables Origo to completely separate projects from each other while building upon a common code base for its modules. This also allows for tight controlling of each project's data. All project-independent functions, such as user-related functions, are managed by the back-end and relayed back to the website.

Drupal with a Twist

From a technical viewpoint, integrating multiple Drupal sites with a back-end is one of the main challenges of Origo. Fortunately, this is where Drupal's modules - and APIs - come into play. Using a range of contributed and custom developed modules, Origo's front-end provides the following main features:

Authentication

User authentication is handled by the back-end and replicated to Drupal. The custom module uses XML-RPC calls to authenticate users in the back-end. The authentication module works by implementing hook_form_alter, where Drupal's validation- and submit functions are swapped against custom ones that authenticate with the back-end. Also, functions from Drupal's session.inc are overwritten with slightly changed ones that synchronize session information with the back-end. This interception of Drupal's login mechanism is leveraged to implement a single-sign-on system: users that change between projects, i.e. Drupal sites, are automatically logged in at the new site, and the still valid back-end authentication recreates a Drupal session at the new site-instance.

The module also provides an XML-RPC framework for other Origo-API functions, both authenticated (that require a valid user-session) and anonymous API calls.

Wiki

Origo aims to provide full MediaWiki functionality. It uses multiple modules to accomplish this:

• wikitools to provide wiki nodes in Drupal
• Custom module mediawiki_filter wraps MediaWiki's library for parsing and rendering MediaWiki syntax. The MediaWiki filter is enabled for all of Origo's content: issues, blogs, forum posts, etc.
• Diff for comparing revisions or previewing changes
• GeSHi Filter for syntax highlighting of code blocks
• Image, because they speak a thousand words
• Image Assist to facilitate image insertion (locally patched to use MediaWiki syntax)
• Custom module section_edit which provides the ability to edit individual sections of a wiki page
• Custom module developer_pages enables users to restrict access of a wiki node to project members.

All these modules - including the custom ones - interact purely with Origo's front-end. Therefore, this wiki setup could be used in other Drupal installations.

Issue tracker

Origo uses its own bug tracking module which integrates with its back-end. Issues are a custom node type with its own vocabulary. This way, issues can be tagged, indexed and searched using Drupal's taxonomy module. An issue's metadata (such as open/closed-, assignment- or resolution-status) is added to a node's tags, too, and thus seamlessly integrate with other vocabulary data. The Issue Tracker's main page provides an interface to this data, and besides displaying a paginated list of all issues, gives the user the possibility to search and filter these tags. The different filters can be customized in any way, and the resulting combination can be saved.

issue_tracker implements quite a few Drupal hooks, e.g. hook_view, hook_form_alter, hook_update, hook_insert, hook_nodeapi, etc. This is necessary to intercept node processing at various stages, and synchronize issue-nodes with data from the back-end.

Origo Home: Communicating, tracking, releasing

origo_home is another custom module providing a bulk of Origo's features. It handles all user related functions such as profile, settings, networking (user friendships and communities), as well as workitems (including RSS feed). It offers input masks for requesting projects and communities, and displays an aggregated view of issues reported by the user. It also provides the interface to Origo's release hosting, i.e., displaying the list of a project's releases, as well as forms for uploading and categorizing releases. Furthermore, it implements an interface for the system's administrative functions, such as mass mailings to all users.

While most of origo_home's functions are straightforward form processing, its main task is bringing the front- and back-end together. On the Drupal side this results in intercepting all processed nodes: hook_nodeapi is crucial for this. It checks which action is performed on what type of node, and registers this action accordingly with the back-end.

Restricted content

Origo implements a hierarchical role model. The most important roles are project owners, project members and authenticated users. These roles - with their associated permissions - are used throughout Origo and give different users different possibilities of how to use Origo: posting blog entries, moderating issues, editing wiki pages, etc.

Other modules

Besides the modules mentioned above, Origo uses the following contributed modules:

• Captcha for blocking spambots from user-registration
• Customerror for blending in errors with Origo's design
• Google CSE for providing site-wide search
• Pathauto and Token for generating better readable URL's
• Trash for nondestructive deletion of nodes
• Views for generating node lists: a browsable list of content-nodes, filterable by type, and a list of taxonomy terms.

Back-end

Integrating, coordinating and tracking all steps of the software development workflow requires a powerful back-end. Origo uses a decentralized approach using several processing units (called nodes) potentially running on separate machines, which communicate using Apache's ActiveMQ message broker system. API nodes receive XML-RPC calls, generate a message of an appropriate message type and relay them to the relevant nodes, e.g. a storage-node for database access, or mail-node for sending emails.

Challenges

Managing Origo

Managing and maintaining such a large-scale multi-site Drupal installation - Origo's web front-end currently combines nearly 3000 Drupal multi-site instances - poses some unique problems, as one needs to automate Drupal administrative functions. Origo uses a combination of shell-, Python- and PHP-scripts to solve this: some triggered by cronjobs or the back-end, some manually.

Drupal Installation

A usually simple action such as adding a new instance to the multi-site installation is a complex process with Origo: While the creation of the front-end needs to be coordinated with other services (a subversion repository is set up, databases created, etc.), there's also the need to tailor the new instance to Origo's needs. This is achieved through a custom installation profile, which enables the needed modules, activates the proper theme, creates navigation links, and also adds a few default nodes to the new Drupal instance. Origo needs to automate these steps, i.e., perform command-line installations: for this, a modified version of the Drupal CLI utils are used (which screen-scrape Drupal's installation forms).

Updates

Similar to installations of new projects, Origo orchestrates updates - security updates or releases - from the command line. For a few months now, Origo has used the wonderful Drush module (well, more of a "utility") to access Drupal functions via command line. Drush is used extensively for installing new modules, security updates, but also for executing custom SQL scripts (e.g. for deploying bug fixes in custom modules), emptying caches, etc.

API Synchronization

Origo's open API allows to access most functions from 3rd party applications, e.g., we provide a Mylyn plug-in to manage one's project's issues from within Eclipse. This mandates the propagation of data posted to the back-end via API calls to the front-end. The custom module issue_tracker implements a small XML-RPC interface to this effect, with which a project's issue nodes can be created or manipulated.

Future Work

In accordance with Drupal's motto, development on Origo never stops. Here's a quick overview of future prospects

• Streamlining Drupal Databases
  Currently all Drupal instances have their own, completely separated databases. Some of these could be shared between projects, e.g. a common user-table for all instances.
• Sandbox
  A sandbox project where users can test Origo's functionality. This not only concerns the Web- but also the back-end.
• Project time management
  Extend Origo with time management, where nodes (especially issues) can be assigned to a timeline, and be managed centrally.

Contact

You can contact the Origo team by either emailing info@oriact.com, or posting on the message board - or by posting comments below.

Associated Drupal users: bayt, roetzi, tario and bherlig.