How to handle URL redirects in SemanticProxy?

Also very interested in this question.

Human vs machine input
Pheedo (and other 3rd party companies that deliver content services and advertising for large content producers) represents a difficult challenge for converting NY Times RSS feeds into nodes.

Perhaps there is a way to supply an article headline into a field through manual entry, or automatically in set of fields or cells, like a spreadsheet? The idea would be to further automate the *search* for the original article and content.

This approach would have to find a creative way of extracting the actual headline from the results in order to retrieve the correct link. But because online content changes constantly, this represents a moving target and would therefore be hit or miss without human participation. Maybe one could filter for word hit accuracy by percentage in order to ensure that the results are very close to the original entry, or exact?

Filtering out advertising... or is that content?
Related to this problem with Pheedo is that advertising inserted in the IMG tag from the RSS, which makes it difficult to distinguish from content. So a related challenge is how to filter ads out to both determine which is the ad and which is the image related to the article, and refine the spacing for the headline and summary for the aggregator's layout.

Moving target for FeedAPI results
After the original links are created by the content producer and/or 3rd party company who supplies the redirect, what happens when the headline and/or article changes? Would there be duplicates in the aggregator's nodes or, as in my case, would one see nodes evaporate because the original link is no longer valid. (I have yet to post about this issue.)

* * *

EXAMPLE ARTICLE
Here is an interesting case of several links to the same content, for example:

A print article in NY Times, dated 7 Dec 09, entitled:
Data Mother Lode (newspaper print version)
(The article relates to data mining and software development, but references to DC Bikes do not mention Drupal.)

But searching for the print-based headline produces few useful results. The real headline online is:
Local Governments Offer Data to Miners (full online version)
http://www.nytimes.com/2009/12/07/technology/internet/07cities.html?scp=...

But THEN there is this text-based article for printing (which I find works well with OpenCalais), but it has yet a third headline:
Local Governments Offer Data to Software Tinkerers (online version for user printing)
http://www.nytimes.com/2009/12/07/technology/internet/07cities.html?sq=d...

Lastly, this is the RSS version of the article, handled by Pheedo:
Local Governments Offer Data to Software Tinkerers (RSS feed version)
http://feeds.nytimes.com/click.phdo?i=80fd2de0ef08f017d23ef99918b17b6a

This is what appears in the RSS feed:

<div class="apple-rss-article apple-rss-read" onclick="handleArticleClick(this)" showSeparator="true"
        articlesortdate="0281898316.000000" articlesorttitle="local governments offer data to software tinkerers" articlesortsource="" sourceindex="0" articlesortid="00000000000000000055" articlelocaldate="0281901732.628975" articleid="91c0ebf28ae496abf675eb77b5598251751bf571">
    <div class="apple-rss-article-footer"></div>

    <div class="apple-rss-article-head" >
        <div class="apple-rss-unread-dot"><img src="file://localhost/System/Library/Frameworks/PubSub.framework/Versions/A/Resources/PubSubAgent.app/Contents/Resources/unread.tif" width="9" height="9" /></div>
        <div class="apple-rss-subject" title="Local Governments Offer Data to Software Tinkerers"><a href="http://feeds.nytimes.com/click.phdo?i=80fd2de0ef08f017d23ef99918b17b6a">Local Governments Offer Data to Software Tinkerers</a></div>
        <div class="apple-rss-author" title="By CLAIRE CAIN MILLER">By CLAIRE CAIN MILLER</div>
        
        <div class="apple-rss-summary" >Many local governments are turning over big chunks of data to programmers to create useful Web sites and apps.</div>
        <div class="apple-rss-date" title="Today, 12:05 PM">Today, 12:05 PM</div>
    </div>

    <div class="apple-rss-article-body-container">
    <div class="apple-rss-article-body">
    Many local governments are turning over big chunks of data to programmers to create useful Web sites and apps.<br clear="both" />
<br clear="both" />
<a href="http://ads.pheedo.com/click.phdo?s=80fd2de0ef08f017d23ef99918b17b6a&amp;p=1"><img alt="" border="0" src="http://ads.pheedo.com/img.phdo?s=80fd2de0ef08f017d23ef99918b17b6a&amp;p=1" /></a> <img alt="" height="0" width="0" border="0" src="http://a.rfihub.com/eus.gif?eui=2218" />
    


&nbsp;<a class="apple-rss-article-link" href="http://feeds.nytimes.com/click.phdo?i=80fd2de0ef08f017d23ef99918b17b6a">Read more&hellip;</a>
    <!-- end articlebody --></div></div>
<!-- end article --></div>

Ideally this should be handled on SemanticProxy side...

Frank, if they can't do it, there's already code in Tattler that does that kindof thing that can be "borrowed/ported/used/whatever".

We did actually think of this and do it currently for news.google.com. If you look at the bottom of calais.module (around line 780) you will see an implementation of hook_semanticproxy_url_alter(&$url). If you have a few domains that you know it does a redirect, you can handle the redirect first, then provide calais/semanticproxy the real URL.

Give is a shot and let me know how it turns out.