Data Architecture Overview

Importing feeds with the Feeds module

The core purpose of Managing News is to aggregate information. Each source comes in the form of a URL or a local file. Managing News uses a module called Feeds to first harvest feeds from local OPML files, and then use these feeds to pull in feed items from those URLs.

Understanding how Feeds works is important when extending Managing News as a platform. Lets look briefly as how these two Feeds configurations work.

How data is imported from an OPML file

The aggregation process begins when an admin user goes to Import > Opml (import/opml) to import an OPML file. When the Import button is clicked, the Feeds module converts the OPML file into Feed nodes.

You can view this Feeds configuration by going to Administer > Site Building > Feeds (admin/build/feeds) and clicking the Override link. The page will look something like this:

As you can see from the settings, there are 4 steps involved with any feed import. The Getting started (5) section has some good information about each of these steps in general, but here are a few important details about this specific import configuration:

The Basic Settings (1) are set to import the OPML feed when it is created since it will be a one-time import (unlike an RSS feed, which would be a regularly scheduled import)
The Fetcher (2) setting are set to pull from an uploaded file.
Once the file has been uploaded, the Parser (3) takes over and uses the OPML parser to convert the data into a common format that can then be used by the Processor.
The Processor (4) is set to take each item and create a Feed node from it.

Because this feed is not attached to a particular node type (1), a link is added to the Import (import) page to a form where the OPML file can be uploaded.

How data is imported from a feed

Once a feed is added, either through the OPML process outlined above or by manually adding a feed, the next part of the process is fetching data from the feed URLs and importing them into Managing News. Feed items are not imported as nodes, but rather as raw data into a table provided by the Data module, which we will discuss in the next section.

Let's take a look at the feed settings by going to Administer > Site Building > Feeds (admin/build/feeds) and clicking the Syndication link:

You can see that this import configuration is set up quite a bit differently than the OPML one:

In the Basic settings (1), the Attach to: Feed item indicates that the importer will look for the URL of the feed in Feed nodes.
When a cron run occurs, the page this URL points to will be fetched, as indicated in (2).
Once a page as been fetched, the Parser (3) determines what type of feed is being pulled in (RSS, RSS 2, or Atom format) and pulls out important information. In this case, it will also grab taxonomy terms from the title and description, which can then be used for [geo-tagging[920726#geo-tagging), or tagging for display in specific channels.
Finally, the feed items are added to a data table (4).

If you are coming from a Drupal background, a logical next question is why aren't feed items stored as nodes? Read on to find out more.

How the Data module is used to store feed items

Data is a module that standardizes the creation, modification and integration of customized data tables in Drupal. Often, the node system is used for storing this kind of data, but the node system is also very heavy. The reason why Data is used for feed items is that it is much faster than the node system, but still maintains a good level of flexibility. For example, Data exposes tables to Views. Data is also better suited for large one dimensional data items as it does not bring the overhead of resource-intensive modules like CCK.

Data structures are also extensible through the Data user interface. See Customizing Imported Feed Data for an example of capturing a new author field from a feed and displaying it as part of the feed with Views

How data is displayed with the Views module

Views is a module that allows for flexible output of data, and nearly every display of feed content - including maps - are all based on Views. If you go to Administer > Site Building > Views (admin/build/views), you will see a list of all of the active and inactive views on the site.

Click on the Edit link next to the mn_views view to for some insight into how the view for the feeds you see on in the Feeds section is constructed:

Edit view link

You will see the following Views configuration:

Edit view configuration

Views is a complex module, and resources exist to help you familiarize yourself with how to build and modify views. The import part to note here is that this configuration page where you would go to add modifications to the output of data. In the Customizing Imported Feed Data section mentioned above, there is an illustration of using Views in combination with the Data module to add an authored by section to feed items.

Section (1) in the screenshot above lists all of the displays for a particular view. Displays within a single view are typically related by having several common settings, including the type of data being displayed. In this case, each item in the list displays a feed in a unique way. For example, there is a Page: News display which shows a heading with a short blurb of text, while the Mode: Map displays the same data layered on a map.

Section (2) is the configuration area for the display. Here you can add new fields and modify the output of the display in several ways.