Eureka! Science News just launched – it is a site dedicated to provide the very latest science news, but with a special twist – it is entirely automated! There is no human editor behind it - it finds relationships between news stories from all major science sites and regroups, categorizes, ranks, tags, finds related press releases and publishes them directly on the site. The result is an efficient overview of everything happening in science, right when it happens. The following details how we built the site.

History

First, a little bit of history about how I discovered Drupal; I launched Biology News Net 4 years ago using Movable Type – biology is the #1 science and I found it weird that no site was dedicated to biology news. The site quickly became popular (#1 on Google for ‘biology news’) – it was unexpected, as the site was started as a hobby project / blog and thus I hit the limitations of Movable Type really fast; adding functionality was complicated, performance was not great (even on a dedicated server), customization was not really doable. Just as an example, the forum is actually a phpBB installation that has its sessions tied to the Movable Type sessions - it's clunky even if it works, and upgrading is a nightmare. As I could not really expect more from a blogging engine – Movable Type served me well - I searched for something better - this is when I found Drupal (about 2 years ago) and fell in love with it! It has a significant learning curve, but it is so powerful that the time invested to learn it is easily worth it in the long run. While I do not have time to contribute much to the actual development of Drupal, I help when I can and maintain one module (quickstats.module, coded by chx with small improvements from me).

Goals

I wanted to build something bigger than Biology News Net, but also something different – a site that would report news as it happens, but for science as a whole. I update Biology News Net manually and being a busy guy, that means usually only once per day. I felt the need to automate the process while keeping the quality at a high level – there is lots of science news reported daily and frankly, not all of it is interesting! I found the inspiration in initiatives like Techmeme and Google News.

Building the Eureka Engine

First, I identified the principal components of an intelligent news aggregator:

• A source of news, such as an RSS aggregator
• A clustering engine, to group news together
• A classification engine, to categorize the news (Is this Biology, Physics, Medicine or Astronomy?)
• A way to assign scores to clusters, to determine in which order the news should be displayed

First, I needed a good RSS aggregator; the default one provided by Drupal was inadequate, as it did not create Drupal nodes out of RSS items. The other aggregators available at the time were Leech and Feedparser, but they were implementing lots of functionality I did not really need and were not yet mature. Fortunately, Ted Serbinski (m3avrck) came to the rescue, releasing Simplefeed just when I needed it. Simple, fast, efficient: I could not ask for more!

Next, I built three custom modules to implement the remaining functionalities. The process was easy thanks to the infinite extensibility of Drupal – the hooks system is just wonderful! The result is what I call the Eureka! Engine. Here are some details about these three modules:

Clusterer.module

The first step is to regroup similar items in clusters. Two things are needed to cluster items together: a similarity metric and a clustering algorithm. In the case of text, similarity metrics can be based on the occurrence of words in two texts – a document can be represented as a vector (see vector space model).

Fortunately, MySQL Fulltext engine does exactly this – by using a whole article as a fulltext query against all other articles, it is possible to calculate a similarity score between every pair of articles. I use a sliding window technique to limit the number of items clusterer.module needs to look at (highly related news items are often published in a relatively small timeframe so looking at a ‘window’ of a few days worth of news at a time is adequate). Hierarchical clustering algorithms have a complexity of O(n^2), sometimes worse, so it gets exponentially more expensive to compute clusters as you add more items. To regroup items, I used a Perl API to a C library implementing hierarchical clustering; it is very fast, clustering a thousand items in a few seconds. I initially tried to implement my own clustering algorithm in PHP but it was slow and memory inefficient. Do not reinvent the wheel if you do not have to! Lots of tweaking was necessary to find the ideal clustering parameters that would allow great precision and accuracy. Be too permissive and you end up with mega clusters of unrelated stories - be too restrictive and related stories do not cluster together anymore. In the end, I got something very satisfying.

Categorizer.module

There are many classification algorithms out there; I needed an accurate one, but most importantly, a very fast one (many
items to classify from RSS feeds). I chose a naïve Bayesian filter – your email software probably uses a similar approach to determine whether incoming mail is spam or not. In the case of Eureka Science News, the algorithm needs to classify incoming news items in eight categories – Astronomy, Biology, Climate, Health, Math, Palaeontology, Physics and Psychology. For this purpose, I ported / reworked a naïve Bayesian php library to Drupal. It is reasonably fast and categorizes a new item in about a tenth of a second. The system is surprisingly accurate once trained properly - of course, it makes errors every now and then (especially when it encounters a post made up of words it did not see before) but I am pleased with its performance so far. In the future, the system could be improved by using latent semantic analysis.

Publisher.module

Finally, I built a module to rank clusters of news and to find and parse related press releases. The module ranks clusters based on the number of items they contain, the timeliness of each of those items and a few other factors such as popularity – the score is time-decayed using a formula based on radioactive half-life decay, keeping only fresh or popular news at the top of the front page.

The system as a whole outperformed my expectations; it even outsmarted me on a few occasions where it found links between stories I did not think were related (why did it regroup those four articles, they are not highly related! Oh, wait… they are slightly related but were presented at the same international conference! Cool!).

You should be able to do the same on your sites pretty soon - thanks to the memetracker.module from Kyle Mathews, a Google Summer of Code 2008 student! He plans to build a generic framework which will allow the automatic clustering and classification of Drupal nodes. Cannot wait to see what kind of implementation he will come up with – a generic framework is more complicated to build than a site-specific one such as Eureka Science News. If you read this Kyle, just send me a message should you want to talk about it!

Design

Having more time than money, I decided to design the site myself - professional designers are expensive! I used a CSS framework: Blueprint CSS. It is ideal for a grid design (which in turn is great for news sites). It’s really easy to build any grid design you might think of using Blueprint – but it only allow for fixed width designs (at least for now). In my opinion its better to use a framework than to fiddle with a custom made semi-liquid CSS design that will break in just about every browser, but in different, obscure ways. The results obtained with Blueprint are cross-browser compliant which is a huge timesaver. The framework comes bundled with a CSS reset (so that the site will look the same in every browser) and a nice basic typography which keeps a vertical rhythm of 18px (this means that every line of text will fall on the same virtual ‘line’, giving a virtual rhythm which is nice to the eye).

Coupled with Panels.module, it allowed us to create the design of most major pages on the site in record time – it is also very simple to test different layouts since it is so easy to use. Even when using a framework, CSS still has pitfalls – I was hit by somewhat obscure bugs in different browsers (guillotine bug, some menu completely disappearing in IE6, just to name a few). I used Browsershots.org to a great extent – screenshots of your site in just about every browser, free!

Drupal - Module list

The site is built using Drupal 5.x as some critical components / modules were not yet available for 6.x. It is sometimes frustrating o see the shiny new Drupal version that is much better than what you’re developing on, but that’s the way it is – I plan to upgrade soon, but in the meantime, I patched 5.x with a few features that were more interesting for this project (such as JS aggregation).

The power of Drupal relies in the strength of the community; someone somewhere has probably contributed something you need for a particular feature! Here is a short list of the contributed modules I used in no particular order:

• Simplefeed
• Views
• CCK
• Pathauto
• Global Redirect
• Imagecache
• Forward
• Panels (1.x as 2.x came a bit late for us to use, sadly)
• jLightbox
• Taxonomy access control
• Quickstats
• CAPTCHA
• Service Links

Notice that the list is very short. Drupal offers easy access to contributed modules, which is great but can be the source of a disease that I call 'modulitis' - too many modules - which leads to bad performance. Most contributed modules use a broad ‘catch every scenario’ approach (‘kitchen sink approach’), which leads to feature bloat and sometimes bad performance; don’t forget that a module has to be loaded completely in memory, so every feature you don’t need is an extra cost on every page load. For example, the Adsense.module, while very useful in some cases, can most of the time be replace by simply pasting the Adsense code in node.tpl.php (or where appropriate). Same thing with Google Analytics; there is a module for that, but just incorporating the code in the footer of every page does the job most of the time. I would also like to see more ‘modular’ modules in the future; it is often hard for maintainers to resist feature creep. Module ratings on D.O. would also help choose between the five image module solutions, the three different lightbox implementations, etc.

As a site developer, it is also easy to fall in the same kind of trap; adding tons of features users do not want / need and that lead to bad performance in general. I could have added voting, comments, a forum, a blog for every user, instant messaging, etc. I started with what users want: the news. I can add other things later on if there is significant demand for it.

Search: Meet the Sphinx

At first I used Google Site Search as the search solution, as I felt that Drupal search.module could not keep up with a very large number of nodes (the site produced about 50 000 nodes in a month and a half during testing – what about in 5 years, with more sources?). With Google Site search, the updates were not instant, (there is a significant delay between when an item is published and when Google will crawl it), the interface was also very restrictive - even if I could integrate the results directly in the site, I could not alter the layout or results in any way. I discovered Sphinx Search thanks to a post made by chx on his blog. It is easy to configure and extremely fast - it indexes hundreds of thousands of nodes in mere seconds and searches are all returned in one second or less. Using a “main + delta” scheme of indexing, I can index news as soon as they are published.

In addition to the search form, I used one of sphinx built-in function to generate our stopwords list (for clustering / classification).

Performance

Performance is always the #1 concern when building a site with a potentially huge number of nodes. Eureka Science News is quite complex, so I generated custom SQL for almost everything instead of using the very convenient Views.module - just about every query is optimized and thus very fast. Views is used exclusively for the Archive.

MySQL optimization is important and the Drupal page caching mechanism and the MySQL query cache are not an excuse to have badly optimized queries. What if I want users to register in the future? What about the one user hitting an uncached page that is forced to wait seconds while the page loads?

Servint – I installed APC cache, tweaked Apache and Mysql but did not yet install more advanced performance improvements like Memcache. Right now AB (Apache benchmark) gives us upward of 500 requests per second on a cached page, which is very satisfying.

I applied the YSlow principles thoroughly – a special thanks to Wimleers for his detailed post about YSlow vs Drupal – while I left things out like using a Content delivery network for now as I feel it is overkill, I keep a very interested eye on Wimleers CDN.module. Some of the things which made a small but noticeable difference on the page loading times:

• Minified CSS and JS using Yahoo YUI compressor
• Aggregated JS and CSS
• Used CSS sprites for small icons
• Used PNG crush and PNGslim to reduce the size of all the images on the site

I also used Drupal cache_set and cache_get functions for SQL intensive blocks that are on every page - like the ‘popular’ block.

Be wary of implementing tons of javascript plugins, as that can affect load time performance. At one point the ‘recent images’ block on the front page were embedded into a javascript carousel, but it impacted the page load times significantly (users had to download additional images that they wouldn’t see most of the time, plus the actual javascript parsing and execution time was significantly high - 500 milliseconds for that carousel alone). Sure it looked cool, but cool is not our #1 priority, performance is!

Drupal page cache is great for anonymous users, but can also create problems; I wanted to display the amount of time elapsed since the publication of news items in some place (‘published 2 hours ago’) but with minute precision. I did not want users to see for example ‘published 1 minute ago’ for 10 minutes. I resolved the problem by porting Drupal format_interval function to JavaScript.

The optimization paid off as the site was hit by Reddit (two different stories), YC News and Stumbleupon the day of the launch and the following days: the relatively modest VPS did not even break a sweat; page loads were still instant, even with about 3000 page views within a few hours, about 15 000 in a few days. Since then we have been hit by Slashdot multiple times and were featured on the front page of Mashable with no performance problems whatsoever. I feel confident that a single VPS will be able to handle the growth of the site for quite some time.

Miscellaneous

Developing on Windows

Doing web dev work on Windows is possible! I built everything using Windows exclusively – using Wamp and Notepad++ mainly. I also could not live without the Firebug and the ‘It’s all text!’ extensions for Firefox.

Of course developing on Windows have disadvantages; I was bitten by a strange bug where my text editor would sometimes insert an invisible character or line at the beginning or end of a module file, which would cause errors (white screens of death mainly); once I figured things out, I had to reedit those within Linux to get rid of the problem. Speaking of Linux, it is actually much easier that I previously thought; I learned using it a month ago to tweak the server; it is easy if you have DOS experience (but I hate VI – anything better to edit files?).

Moreover, my dev box got somehow infected by a nasty piece of adware that made IE and Firefox crash and opened tons of pop-ups. Fortunately, it happened post-launch and I had plenty of backups (on the server, on an external HDD and on a laptop). It still took me 3 days straight to clean completely.

Lessons learned / random thoughts

Here are a few lessons I learned in no particular order

• Finding a good domain name is hard and takes time (and / or money) – start early and never stop searching, even if you got one or 2 good ones! You might find something even better.
• Backup everything often! Especially on Windows.
• Think simple; more is often less – do only what your users actually need / want as a starting point
• Don’t be afraid to redo something from scratch if it’s not working right the first time around - I rebuilt critical components of the clustering system days before launch
• Drupal is a market disrupting tool – it allows a single guy part time to build something great while learning it; imagine what a whole team of professionals can do!
• I wish I knew about simpletest a year ago; I chased bugs for a long time (and sometimes the same bug that was reoccurring). Things like clustering and regex-based parsing could have been a whole lot easier with appropriate tests.
• Release early; don’t be afraid to put off minor features to later
• Keep a todo list through the process, and try to remove items from it as fast as they appear! (this is not as easy as it sounds!)
• You know you built something great when you visit your own site and find it interesting :)

The future

On a closing note, the platform I built is highly adaptable to other news sites (sports, tech, general news source). The only problem is that press releases are not available in a centralized location as is the case for science; I might seek venture capital to license Associated Press news to build a news site similar to Newsvine (any VC reading this?). I am also toying with the idea of implementing anonymous comments on the site using Mollom. But of course, I have new unrelated projects to start, and a PhD to finish! I hope you enjoyed the write-up and that you will enjoy Eureka Science News! If you have any questions feel free to ask ;) As a closing note, special thanks to funkyhat who helped me tremendously with this write-up.