Duplicates: use URL OR GUID for deduping, accept URLs > 255

feedapi_item_node module

On some feeds GUID changes wildly for the same article (= same URL). In a majority of cases GUID == URL .

In a set of 28663 feed items I ve got a total of 1378 duplicate sets. In a duplicate set there are at least 2 articles with the same GUID. This does not count duplicates across feeds - currently, feedapi only checks if a feed item already exists for the given feed. The feed items are from dozens of keyword searches on search engines like Google News or Flickr.

Of those 1378 duplicate sets, 927 have the same GUID, but have different URLs, 450 have the same GUID and the same URL.

(If you think deduping with GUID is faster, because it is mostly a hash: On 16823 of all 28663 feed items GUID = URL. )

Using URL instead of GUID for determining wether an article already exists, could be a lot more accurate with little cost.

This assumption holds only true of course, if you consider a URL as more secure indicator of uniqueness. One could argue about that, but especially for news feeds, by and large there is a URL per article.

In regards to the 450 duplicate sets that have the same GUID AND the same URL - their URLs are all longer than 255 chars :) - the length of the URL and the GUID field in feedapi_item_node reason why the dupe-check fails.