The IT Dashboard is a management tool for the United States federal government CIOs to use in overseeing their IT spending. To support transparency and open government initiatives, Federal decision makers also chose to make the IT Dashboard available to the general public. In the IT Dashboard, members of the public can view the same data about performance that government officials use.

Background

In June 2009, Federal IT leaders asked two questions about their IT budgets. These questions are common in the commercial world, but have been difficult to answer in the public sector:

  • Where are our IT dollars going?
  • Are we getting value from our investments?

The answer to these questions is a website that gives insight into the acquisition and management of the federal government's $75 billion-plus IT portfolio. The result is the IT Dashboard, a site built in collaboration with the government's most senior IT leaders and nearly every federal agency.

Purpose of the IT Dashboard

The IT Dashboard provides full lifecycle performance information on IT projects' scope and status. The data allows users to see project status, spot problems and trends, manage performance and quality expectations, and establish accountability. The tool is useful to the public and to government IT leaders who plan and manage IT investments. The home page screen shown above gives an overview of the tool.

IT Dashboard’s value comes from several facets:

  • Mashing-up data from multiple data sets (IT project justification, schedule/status, expenditures, contractors.)
  • Ensuring that real-time and complete data is available.
  • Presenting the data and comparisons in ways that are visual and easy to drill into.
  • Providing only high-level data that doesn't overwhelm the viewer.

Site Design and Functionality

Originally conceived as a PHP application, the IT Dashboard evolved to a Drupal-based application, employing the Drupal API, several modules, and basic theming. While not all functionality is built in a customary Drupal way, we have modified all pages so that they can be accessed using menu hooks. The advantage of using such hooks is that now all those pages can use the Drupal API, and contributed modules, without worrying about whether or not Drupal has been properly bootstrapped.

One of the primary requirements was that REI Systems deploy the IT Dashboard rapidly. Since the source datasets were already available, developers avoided the need for a data migration step by working with the existing data model. With a working data model and Drupal’s easy-to-use components, we went to initial release in under six weeks. Since the launch in 2009 active development has been ongoing.

Another requirement was that the IT Dashboard incorporate captivating and practical data visualizations. To meet these requirements several visualization libraries are used across the site. FusionCharts drives most of the bar, pie and line graphs throughout the site. Google Motion Charts and JuiceKit power the more advanced Tools section of the site.

Initial plan was for agencies to enter monthly updates through data entry screens. However, this approach quickly led to delays and data quality issues. To address these issues, we introduced an XML API, allowing for system-to-system integration between the IT Dashboard and the systems used by more than 26 Federal agencies.

Access authentication for data providers is handled by integrating with an existing security system, included within the OMB's MAX system -- the tool used by federal agencies to enter and report official budgetary data. This integration was accomplished by extending the Drupal Central Authentication Services module (CAS), which controls role-based data access at both the application and database levels. The application makes certain data visible to the user only if they have appropriate permissions; otherwise, the user is limited to viewing only the publicly available data and graphics. User credentials are propagated to the database and are stored in a database session. This information is used by the Database Views module to enforce authentication and authorization rules.

The use of the FusionCharts library, which uses the Adobe Flash plug-in for rendering, meant that many mobile users were not able to easily access the same information as desktop users. To address the growing need for mobile use, a mobile-friendly version of the website was rolled out using the jqTouch library and the javascript visualization library flot.

Apache SOLR powers the search capability. A private SOLR index is used for authenticated users and a public SOLR index with a subset of the data is used for anonymous users.

Infrastructure

To support easy scalability and accessibility, the IT Dashboard resides in the cloud.

The federal government selected a technical environment that is robust; provides Infrastructure as a Service (IaaS); offers high levels of security, scalability, reliability and accessibility; and has a security plan and existing capabilities to protect publicly accessible web sites that disclose government information.

The IT Dashboard leverages a Content Delivery Network (CDN) structure for the public portion of the website to improve overall system performance. Business data is stored in a separate MySQL database; the Drupal database contains only Drupal-specific configuration information, along with static content.

The infrastructure was selected to support a tool open to use by more than 26 government agencies, members of Congress, the IT vendor community, and the general public - whomever may have some interest in the federal government IT investments.

Module Use

Contributed Modules

Major modules that were used were:

  • CAS - Integrates with existing OMB CAS implementation named MAX.
  • CCK - Used for data entry and file uploads.
  • ClamAV - Scan attachments for viruses.
  • Context - Used for page layout.
  • Skinr - Allows easy application of a pre-built style library to blocks.
  • ThemeKey - Eases transition from old theme to new theme on per page basis.
  • Views 3 - Allowed us to plug in our own query backend, necessary to access separate application database.

Custom Modules

The following custom modules were developed for the IT Dashboard:

  • Data Controller - A Views 3 Datasource plugin that has support for aggregations and efficient query generation.
  • PHP Dataset - Describe data sets for views using custom PHP code. We are using this to drive views that require data from the application database.
  • Views FusionCharts - Display style plugin to render view output as a FusionChart. This module is compatible with Views 2 and 3.

Challenges and Lessons Learned

We learned one significant lesson through this effort, namely that Drupal APIs were more versatile than we had originally believed. Our work on the IT Dashboard started initially with a PHP application, not using any Drupal APIs. We found several problems with this approach, each of which make system maintenance more complex and error-prone:

  • The logic in SQL queries tended not to be re-used.
  • Security items need to be addressed separately for each page viewed.
  • There was not clear separation between logic and presentation.

Upon observing these problems, and gaining confidence in Drupal APIs, we revised several pages to use the Views module, which allowed us to:

  • Centralize the logic of most queries in View Default.
  • Establish a clean separation between View logic and View display.
  • Most SQL-injection and XSS issues are already addressed in the code of the contributed modules.

As with any complex development effort, we met several challenges, and learned lessons that may benefit others in the community. The most significant challenges included:

  • Reconciling preferences of different stakeholders for different visualization and layouts in order to take advantage of Drupal's strength in a consistent presentation style.

    Drupal offers many ways to utilize templates and generic modules to implement a consistent look and feel across many screens. However the quick turnaround and occasionally conflicting preferences from different stakeholders made it difficult to develop an overall design that leveraged this Drupal strength. We have made a number of revisions since launch to improve consistency and we plan to address this challenge by spending more time up-front on mock-ups, and facilitating agreement across stakeholders on the end-state vision.

  • Legacy system includes a data model that isn't suited to new uses.

    The IT Dashboard has a separate application database that originated from a previous system. We addressed the challenge of accessing this database by using a custom query plugin for the Views 3 module.

  • Facilitating expansion/collapse of table data detail to suit user preferences (an example of custom display functionality).

    Many IT Dashboard screens have custom functionality, which could not be satisfied by the Views module at a basic level. For example, some screens offer a table view of data. However, the desire not to overwhelm all users with the full set of data led to a decision that some rows should offer an "expand" button to allow drill-down capability at the user's option. Implementing this customized functionality required custom coding.

Project Team

The site is currently operated by a nine member team, composed of the following roles:

  • 4 Drupal developers
  • 1 database developer
  • 1 QA engineer
  • 2 business analyst
  • 1 project manager

Additional Information

REI Systems is a leading provider of web-enabled, database-driven knowledge management and performance support solutions for federal agencies, state and local governments, and the commercial and industrial sector. For more information, please visit us at http://drupal.reisys.com.

Comments

rick_deputy’s picture

Its usability, contrast could have been much better. Twitter/FB links are wrongly placed. I can list 50 things overlooked in Drupal implementation that any expert would have paid attention to.

Rick

mherchel’s picture

Damn Rick! A little harsh, don't you think? I think that if the your first criticism is the placement of the social network links, then it can't be that bad. I also think that without knowing the specific use-cases that their designer/developers are targeting, its hard to judge.

I personally think the website looks great. Great job guys! Browsing down into the data (without really knowing what I'm getting into) seems to look great and be easy to use. I personally think any type of transparency in government data, spending, and resources is a good thing.

Can you guys elaborate a bit more on the infrastructure? Where does it pull the data from? How does the data get there? What type of databases is this system using?

rjivan’s picture

Thanks mherchel. The initial plan was for agencies to enter monthly updates through data entry screens. However, this approach quickly led to delays and data quality issues. To address these issues, we introduced an XML API, allowing for system-to-system integration between the IT Dashboard and the systems used by more than 26 Federal agencies. More information about the IT Dashboard is available in the FAQs at http://itdashboard.gov/faq-public and http://itdashboard.gov/faq-agencies
Regarding infrastructure, the IT Dashboard is using MySQL.

Marko B’s picture

Grea site Rjivan, like your overview. Btw how and where do i apply site to drupal showcase?

aellison’s picture

You can create a showcase post here:

http://drupal.org/forum/25

braino’s picture

Wow! I can't image the complexity of integrating 26 back-end systems, many of which are probably archaic, while not breaching the security of those systems. Well done! This site looks fantastic for a government site.

For a real level of transparency, it would be nice to include the relationship between vendors receiving government expenditures and the politicians behind those expenditures. For example, vendor zip codes could be categorized by congressional district. This would not tell us which vendors donated to, or are related to, which politicians, but it is a start. By the way, how did you get the job and what was the pay?

stovak’s picture

You can criticize the design or architecture or the Drupal implmentation, but the cultural change that a website like this has made in FED decision making and the ripple effects on software development throughout FED projects can't be measured.

It's not an understatement that this project changed and is continuing to change government software forever.

http://www.nextgov.com/nextgov/ng_20090717_9839.php

So proud off my coworkers for their efforts on this project!

pkiula’s picture

The exact same things could have been done with WP, and would have been a little easier out of the box.

Probably a touchy thing to mention on a Drupal community. Sorry.

It's nice either way to see open source CMSes getting mainstream traction. Yay!

mdeltito’s picture

I know you aren't starting a "platform war", but I also know you don't have the necessary information to make this assumption.

togbonna’s picture

So, why was it not? The developers probably arrived at the decision, after considering all other options, including WP, that Drupal was the best choice. Am pretty sure a drupal-bias didn't influence that choice. :)

www.icelark.com
Think.Act.Be
@togbonna
@icelark

shamio’s picture

I am sure other CMS platforms couldn't handle this site. WP uses uses for running blogs or small websites and Joomla can be used for small top medium size of websites and it has problems too. But Drupal is a good choice for running professional and large websites. Its the reason that most government websites use Drupal as other platforms can fit their requirements.

kfogel’s picture

@rick_deputy

Personally, I'm pretty happy with the UI on the site, so I'm not sure how much to interpret your criticisms as questions of taste (I'm not a visual designer, so it might be that you're more in tune with what modern web applications should look like than I am). In any case, note that they are planning to release the code eventually -- see http://civiccommons.org/2010/09/open-sourcing-the-federal-it-dashboard/ . So you'll get a chance to send in patches for these things :-).

UI quibbles aside, I completely agree with the other commenters' points about the effect of this software on federal decision-making and accountability.

-Karl

franz’s picture

--
Franz

Adam S’s picture

Ok... Where should the Twitter / FB links be placed? And, what are the other 49 things that in your expert opinion should be different? If you can't do that then your just bitching for no reason.

Edit: Now that we got some of this puerile stuff taken care of. I have a question about http://www.itdashboard.gov/investment?buscid=622 ?

How does running the two websites, usaspending.gov and itdashboard.gov, on Drupal play into the project being 19.61% under budget.

Anonymous’s picture

"I can list 50 things overlooked in Drupal implementation that any expert would have paid attention to."

And then you end the message. I would like to know what these 50 things are. Can you please clearify so that we can all learn how to be experts. Or maybe a link to your blog with these things. ( I think an expert would have at least a link to clarify such a statement ;) cause damn that site is nice

mohammed_8960’s picture

nice job guys
big hand

Virgil’s picture

are you sure you don't need to check your eye doctor?

SemperVI’s picture

I am a total rookie deputy when it comes to developing php/drupal solutions. Just started 5 days ago. Having said that I have been developing in the web tier for over 15 years - so I know a thing or two about this space we all work in. While Drupal is really a great tool to publish content. Not everything a developer in the web tier is content driven.

Point in case - the case study clearly stated the business case

- Where are our IT dollars going?
- Are we getting value from our investments?

More times than not in the web tier, business applications are more about process management and reporting than it is about content management. I think this is a great example of how Drupal can be used out side of the CMS areana. Mad props to the development team and thanks for sharing

te-brian’s picture

Great job on the site guys!

I love the heavy use of data-visualization. Tables can be boring can't they? But charts are FTW!

I've been very impressed with the case-studies lately. We are seeing Drupal grow-up before our eyes, and really get the design and architecture treatment it has deserved all along.

cpliakas’s picture

Congratulations to the REI team! I am sure that behind the scenes there were a lot of additional political and technical barriers beyond what was illustrated in this case study, and I appreciate the effort that was made to get this launched on Drupal.

Also, thank you so much for posting the lessons learned on this project. Beyond the technical accomplishments, this information is vital to companies and organizations who are beginning to use Drupal for the first time.

Great work, and thanks for helping to drive the platform deeper into the public sector.
~Chris

Dig1’s picture

Great Job Guys! The way the data is listed, graphed and quickly available makes it very easy to absorb what is going on.

Well done.

Dig

rahulbile’s picture

Th UI is very clean and eye catching. Good job guys !!!

Cheers

vallab444’s picture

fantastic, neat & tidy look; great charting.

-Vallab

wfx’s picture

Great work - that site is slick! I can't believe this is a .gov site. I'm used to circa 1999 gif graphics and rolling seas of plain text. Really sets the bar much higher for future websites.

Ildar Samit’s picture

Department of Defence's id number is 007? Nice :)

aellison’s picture

http://radar.oreilly.com/2011/03/white-house-it-dashboard-open-source.html

And congrats to everyone involved in getting this open sourced!

Maung Maung’s picture

It is an amazing project.

Even you all release the whole application as opensource (http://sourceforge.net/projects/it-dashboard/), I have to use a lot of time to learn from the whole project. :)

Then, I see tcpdf library in the source files, but can't find the pdf download option on live site. (Or may be I can't reach some parts to find out). Can I know how this pdf library is using? I'm researching various ways that can produce nice PDF files on the fly, without using resource hungry library like "wkhtmltopdf".

Also thanks for releasing the Mobile version. I can learn a very nice Drupal mobile site, theme and implementation from your hard works.

Thanks everyone from this project and the Drupal community.

jmendez1’s picture

The public can view PDFs generated by this library by clicking the "Exhibit 300" link at the top of the Investment Dashboard page.

For example: http://it.usaspending.gov/investment?buscid=184

Maung Maung’s picture

Thanks for pointing the right place.
The generated PDFs are nice and useful for reference.

gloscon’s picture

This shows effort of US govt to attempt to bring transparency in IT Spending. I hope other govts get inspired from this and do something for their country.

I followed all the way through few links and the transparency is awesome..e.g Being in India, I cannot imagine I can see something like this : http://usaspending.gov/explore?&typeofview=complete&tab=By+Prime+Awardee...

tmckeown’s picture

Rajiv,

It's great to see this whole project documented as a case study. The whole team at REI, including myself, put blood, sweat and tears into this project to make in the tight deadline of 6 weeks. The lessons learned and challenges section is right on point. Awesome you guys tapped into view 3 external data sources! It's a really cool feature that I am very excited about. Are there any plans to create drupal projects for some of the modules you guys created? Tell everyone I say hello and that I miss the late nights and the red bulls :-).

Warm regards,

Trevor

lca7’s picture

That's great advice I think there lot of drupal users will be interested

jwalpole’s picture

Another great example of how Drupal is helping Govt and open sourcing the outcomes. Nice work everyone, including Karl and REI!

Jeff

dheeraj.dagliya’s picture

Good to see that Drupal has been used for presenting Analytical Data (Charts & Graphs)

Thanks for sharing the Case Study and CODE.

lan5’s picture

thanks for the showcase, just a question on the charts used , are these charts on the site made up dynamically by querying a database , or filled with static data ?

is there a module that does reporting with charts ?

rjivan’s picture

The charts are dynamically generated by querying the database using Fusion Charts. We are in the process of migrating the custom php code that generates the charts to a drupal module. For example the chart on http://itdashboard.gov/investment/cost-summary/836 has been generated using a custom module (views fusioncharts) that we developed. This module is a views style plugin (compatible with views 2 and views 3) that takes the data generated by a view and renders it as a chart.
You can find the source code for the module at http://it-dashboard.svn.sourceforge.net/viewvc/it-dashboard/trunk/webapp...

zeezhao’s picture

Great work. Looking forward to the module.

rgs’s picture

Any idea when the actual fusion charts views plugin will be released?
Have tried using the module from the source code but get a wsod on a basic vanilla D6, so I assume you need the full IT Dashboard installation to for that to work.
Thanks for this really great work.

rjivan’s picture

The plugin should work on a vanilla D6 without any modifications. Please file a bug report at http://sourceforge.net/tracker/?group_id=508075&atid=2065789 along with the steps to reproduce it and we will take a look.

msimanga’s picture

I managed to get the chart module working from SVN without installing the full ITDashboard application. Ensuring that the correct libraries are available, i.e. ensuring that FusionCharts files were in the correct location (in libraries folder) did the trick for me.

The chart module is wonderful from a Views usage perspective, it has many more configuration options than the current crop of Charts modules. It will be interesting to see if a D7 version is in the pipeline. Great work!

szabeszcsi’s picture

Hi!

How did you get the views_fusioncharts module working? I installed but only blank page appears...

stovak’s picture

Try our highcharts module

http://drupal.org/project/highcharts

emptyvoid’s picture

Gee, why do I not find it surprising that the websites the show what agencies, department, consulting companies are consistently over their budgets and never deliver on time suddenly want to de-fund and shut down the websites that out them?

It's pretty sad that the political (and it is political not financial) maneuvering is going to shut down clearly useful services for the American public who pay their salaries. Perhaps we could not make one tank, jet, or helicopter and fund the 9 websites that the various agencies are trying to shut down.

The fact that I can't log into a centralized city, state or federal government service to review my representatives votes, fundraising, contributions and annual salaries is ridiculous. I should be able to review all of the past and new bills, contracts, and proposals for my city, state, and federal issues. And dare I say I should be able to securely post my vote for such issues (removing the need for highly paid representatives) and cast my vote for an issue and get counted. It is a sorry state of affairs in this day and age where this hasn't happened and it is appalling and spits in the face of Democracy in the 21st century.

But then again, I should be careful of what I ask for, I'm sure the Federal government would award the contract to Diebolt or Haliburton and next thing I know identity thieves are using my personal information to buy a house in Hawaii or worse removing money from my accounts (thanks oh so secure Diebolt for those stellar voting machines a 10th grader could program better).

But I digress, .. (breath slow breaths).. it's Friday and I'm cranky.

Robert Foley Jr
Application Architect / Business Analyst / Information Designer
http://www.robertfoleyjr.com

amitsoni26’s picture

It is very user friendly. and very easy to use. Data representation is also very fast.
Th UI is very clean and eye catching.
Good job guys !!!

Regards,
Amit K Soni
amitksoni.com

mongol2012’s picture

good job

mtaka8’s picture

Perfect data analysis site!

ELS Landscape’s picture

I am relatively new to Drupal and not a programmer. I have trouble with basic web design. However, I can tell Drupal is a powerful platform and this is more proof. I can think of dozens of uses for this dash board. I almost want to go take some programming classes now.

jollyjollyjolly’s picture

No reason not to start doing your first website with Drupal right off the bat. It is more complex then say joomla, but it is a very powerful system that can handle some serious traffic. I use it when I know lots of heavy visits are coming to a website every hour. Rock Solid. Im the lead designer for San Antonio Website Design and Drupal has come a very long way as a 4th gen CMS, well open source CMS along with the other 3. It is one of the few things in IT and the internet that is RIGHT actually so start building today and stop lamenting about programming classes...none needed ;)

modstore’s picture

I implemented SOLR integration on a site with a large DB. Very happy with the results and faceted searching. I'm following the views solr integration module closely, a little more work on it and it will be a great setup.

ckvergleich’s picture

very good usability and clean design. Two thumbs up.

Here we can see what`s possible with drupal cms.

monil-dupe’s picture

The site is awesome and its clear when a team of 9 professional persons are working on it including 4 Drupal developers, you see a decent site like this. Well done, your site is excellent.

rryder’s picture

I know how hard it can be to get 26 different agencies to use one XML format. That alone makes this great looking site a feat.
The visual presentation is quickly comprehensive, pleasant to read, and the data presented is clean and valuable. great job!

.. Pardon typos, I hate typing on mobiles

sidharthap’s picture

Thank you for sharing the site. i am very excited to know, How you are maintaining the environment and load on the site.
It will be very helpful if you share the knowledge of external security bridges for this site rather than Drupal security.

Thanks & Regards
Sidhartha

ppro’s picture

clean design & good usability , we can see what`s possible with drupal .

joffreylacour’s picture

Looks very clean, nice work!

mcorser’s picture

Hi!

I'm currently looking into integrating MAX.gov with our Drupal 7 project. I understand that you extended the CAS module to do so, is there any Git repository or something I can look up to figure out how you did that? Thanks!