Integrate large sortable database with regular flat-file (csv) updates -- Is CCK + Views the right recipe?

rjacobs - February 21, 2009 - 00:17

Hello,

I am working with a small nonprofit organization helping them migrate an old static/piecemeal site into a CMS. We are quite confident that Drupal is the right tool for the job, and I already have intermediate Drupal experience. We have mapped out most of the necessary specifics for their new Drupal framework, and related content migration processes, but there is one element from their current setup that I am not quite sure how to handle with Drupal, and I am seeking any insights I can find.

Their current site has a feature/section called the "payroll database", which is a simple listing of salary information for various government offices (made available by the Freedom of Information Act and part of their mission to encourage transparency in government). Site visitors can view, in a small table, a listing of salary details for government employees based on their office (each row contains "employee", "position", "salary", etc. headings). Currently this is powered by a custom ASP script and a MSSQL database. They populate the actual data via excel or access files provided by government.

Getting this running in a new Drupal site would seem trivial for CCK + Views, but I have the following concerns:

  • This payroll database will grow to literally tens-of-thousands of entries. With CCK + Views, each of these would be a node, and I fear that so many "payroll" nodes would be tricky to handle (admin and maintain) within a site that otherwise has only has a few hundred nodes of other content types.
  • We need an easy way to import large volumes of this payroll info from something like CSV flat-files, on a regular basis. I can't find a very easy way to do this with Drupal 6 (all other site specs point us to v6), and it seems that the "node import" module does not even support CCK-types for Drupal 6 yet.

On idea that came to mind is to run this payroll database in a separate Druapl 5 site (perhaps on a separate subdomain). This would, in theory, address both of the concerns above, as it would isolate the payroll content in a totally separate system, and furthermore, the "node import" module for Drupal 5 will support CCK content. But if we go this route, I'm not sure how to integrate data between the two sites in a useful way (or if that would even be possible).

We are hoping to avoid the need to hand-code this payroll database feature, as there are obviously a number of advantages of getting it running within a Drupal installation (scale, future feature additions, etc.).

We would really appreciate any insights or pushes toward other modules and recipes that might work. Many thanks!

Ryan

Node import with CCK is coming to D6 soon

smithn.nc - February 25, 2009 - 21:40

I have a client that's looking for something similar. As a heads up, the node import dev says that a new D6 beta with some CCK support should be out sometime day.

I'm still concerned about creating that many nodes over and over, myself. If/when they ever give me the go ahead, I am considering coding a module that would manage the information from an imported CSV without using nodes. Then again, I'd really miss being able to use Views and the built-in search features. :(

Thanks for that comment. Even

rjacobs - March 2, 2009 - 17:53

Thanks for that comment. Even if node import for D6 incorporates CCK support, I'll still be a little concerned about having so many nodes created for what is essentially just tabular data, and will have to "live" among only a few hundred normal page nodes.

You are right that a custom solution (or module) would be quite a fine option, but as you pointed out, it would circumnavigate what is essentially one of the best and most powerful parts of Drupal (CCK + Views).

I'm really thinking that a site section like our payroll list deserves a separate database for its storage, no matter what. So it makes the ideas of running a parallel Drupal 5 site just for this more and more appealing. I just wonder if there would be a way to display a view from this Drupal 5 site within our main Drupal 6 site so that the user is always within one site theme, etc.

separate database with Drupal 6?

rjoy - March 3, 2009 - 01:09

Depending on your comfort level with the Drupal 6 branch of Node Import, you might be interested in the Embed Widgets module. There is not a Drupal 5 branch of this module that I'm aware of, but the D6 dev branch has proven stable in a pre-production site I've been working on. To clarify, I'm using the embed widgets module to export calendar displays (12 months worth at about 1,500 entries per month) to an existing HTML site while the new Drupal site is being developed.

This, along with multi site or another method for data sharing might be worth a look.

 
 

Drupal is a registered trademark of Dries Buytaert.