Add support for the ExternalFileField field type

Attached is the patch that adds the ExternalFileField field to the schema.xml file. The dynamic field is prefixed with "eff_". A simple example of how this works is illustrated below.

Let's say we wanted to add additional boosts to a few nodes. We could add a flat file to Solr's data directory named external_eff_ranking, which follows the external_{field_name} nomenclature Solr expects. It would contain the following data, replacing {site_id} with the site's apachesolr_site_hash():

{site_id}/node/1=3
{site_id}/node/2=10
{site_id}/node/3=5

In a custom module, we could boost the respective nodes by these values by implementing the following code:

/**
 * Implements hook_apachesolr_query_alter().
 */
function mymodule_apachesolr_query_alter(DrupalSolrQueryInterface $query) {
  $query->addParam('bf', 'eff_ranking');
}

The results will then rank node 2 higher than node 3, and node 3 higher than node 1. This is a very simple example, but in the case of the Facebook-style "like" counts mentioned in the original post, we could either do the normalizing math when generating the flat file or leverage solr's function queries to convert the raw counts to a boost that makes sense when calculating the relevancy.

Changing component and status accordingly.

Updated issue summary.

Updated issue summary.

I would say this is an edge case right now for the Drupal community, however it is far from an edge case in the Solr community and is the accepted method of boosting by user generated ranking data (i.e. voting, thumbs up, this is helpful, etc.) for larger datasets. This assertion is based on presentations delivered by Lucid Imagination and others at Lucene / Solr conferences. Below are a few worth highlighting:

• http://www.slideshare.net/lucenerevolution/potter-timothy-boosting-docum...
• http://www.slideshare.net/giokincade/solr-etsy-apache-lucene-eurocon-991...
• http://www.slideshare.net/LucidImagination/bialecki-andrzej-clickthrough...

Sure you could do this on the client side, but let's say you are a news site has 50k articles and you want to boost based on Radioactivity and a "thumbs up" count provided by the Flag module. Every time someone clicks on a flag or does something that effects radioactivity like view a page, it would trigger the effected node being re-queued for indexing. This adds a ton of processing on the Drupal side during cron runs and produces a lot of unnecessary updates. Using the ExternalFileField technique, all you would have to do is regenerate the file every so often, i.e. 5 or 10 minutes, so that the ranking data could be up to date without having to reindex a ton of documents. It would be great to have the option to do the client side thing for smaller sites, but at least allow for using this benefit of Solr for larger sites.

To me, it would be an arbitrary limitation to prevent this just because the solution isn't 100% contained to Drupal. There is no drawback of exposing this field, and most people won't even know that this change has been made. However, at least opening up the door for the possibility of leveraging techniques that others in the Solr community are implementing could facilitate some interesting innovations for Drupal + Solr.

As an example, there are a lot of companies that offer hosted Solr solutions targeted specifically towards Drupal such as Acquia Search, Pantheon, A12 Search, Midwestern Mac, DrupalCONCEPT, and others (sorry for leaving anyone off the list). One of the value-added benefits they could provide beyond just hosting a Solr instance would be to offer these rank boosting as a service based on this technique. How cool would it be for people using these services to say "Boost documents based on relevancy X, Y, and Z refreshing every X minutes" without having to tax the Drupal site and Solr server with excessive updates? There could be a common module that collects the data generically, and then has a pluggable connection piece to make an API call to the service or take some action that reflects the change. Anyways, just some ideas here, but they are only possible if the EmbeddedFileField door is open.

I can even imagine implementing a click-boosting approach that for small sites does mark the docs for re-index at some interval (e.g. hourly) and puts the boost directly in a field, or even easier (perhaps) applies a document-level boost so the query doesn't need to be altered at all.

Totally agree. I think that the logic could be reused, but for larger sites using the external file filed is at least an option so that your side can leverage the same code but provide more real-time boosting.

Looks like you could use tfloat without adding that?

Hm, maybe. Let me try it out. My experience is that yes, Solr barfs hard when valType isn't defined. I can validate this further though. Marking as needs work based I these two items.

Nice. I effectively played around with different configurations until I found the "magic" combination. I found the <fieldType name="pfloat" class="solr.FloatField" omitNorms="true"/> technique in a support forum somewhere.

Hm, accidental change of status to "needs work" in #10. Ignore that.

Woo-hoo! Thanks, Nick. I foresee some really cool things coming from this. Can't wait to start playing around with solutions.