Currently FeedAPI does the following in order to find out that the feed item is unique or not.
1) If the original url of the items are the same, it's the same item.
2) If the guid of the items are the same, it's the same item.
3) Cross-feed deduping magic

But unfortunately this is a false behavior. Check out this feed:
http://www.snellspace.com/wp/wp-atom1.php

The guid is the ONLY option to decide. But unfortunately in the real world, we need to rely on original url too.
I suggest the following change:
If there is guid, let's trust on guid only and not check for other things.

Please share your ideas on this topic!

Comments

gsnedders’s picture

gsnedders commented

Just using the ID doesn't work in the real world either. See This for an example. Also see James Holderness's post on detecting duplicate items. I'd just ignore the ID if a feed contains a single ID more than once (at a certain time).

aron novak’s picture

aron novak
Primary language Hungarian
Location Hungary, Budapest
commented
Title: Not proper duplicate item checking » The perfect way for duplicate item checking
Category: bug » task

I seek for the perfect way for duplicate item checking.
The current method is not bad, but far from perfect.

Current method:
Feed item is duplicate, if URL or GUID are duplicate or if they are both missing.

Possible improvement:
Feed items have the same title, are from the same domain (url) and were published within chronoligical proximity of a day or so

Any thoughts on this?

eyecon-1’s picture

eyecon-1 commented

Aaron generally resists filtering (which you are basically suggesting) and rightfully so in order to keep the code relatively simple.

That said, this could be done with a temporary table and a unique index on node_revisions.title. As the records are copied, MySql ignores the duplicates if thus instructed. Then the unpublished records in the original table can be cleared and the new table copied over, changing the status from unpublished to published. Obviously, this requires setting the workflow to unpublished.

What I would NOT do is to create a routine in php to do this (other than the records manipulation above). I suspect that would consume considerably more cycles, particularly if you tried to intervene prior to saving the item.

Anonymous’s picture

Anonymous (not verified) commented

In my experience I have yet to meet a fool proof filter. So far sooner or later a article creeps through no matter what filter I have implemented (based on experience with different cms's).

So we need a better way for feed conversions to articles.

Thank you for starting this topic Aron.

Maybe the question we should ask is it possible to completely automate this? Theoritically yes. It should be.

On the other hand I have been toying with the following idea:
Take x words in the title. Say for example 3 main words.
Alternatively automatically search through the title and the article text at the same time. Say take x words from the article title and text.
Alternatively autotagging technology could also do the trick.
Filter out all common words (such as 'the, a, and, or, etc.)

Whenever you have a x word match send the article to a seperate category for review. Automatically look up and display any articles that share those x words in this "review" category in alphabetical order.
Mark newly imported feed items with "review needed". Using "review needed" will make it easier to differentiate from existing articles.
There probably will be a lot of false positives, but theoretically the selection would be narrowed down to at least ballpark relative content. This could already save a significant amount of time from a administrative point of view.

Problems in general in my experience usually occur when:
a duplicate article has a different feed url, but the same (or different) date and a different url.
a duplicate article has the same feed url, but a different date.
a duplicate article has the same feed url, but the same date and a different url.
a duplicate article has the same feed url, but a different date and a different url.

Also taking in consideration that a article might be of the same subject, but not different enough to publish. An autotagging preview categorization for manual review would allow a editor to review and choose which article to publish.

In any case I have found it to be best practice to review articles before publishing instead of publishing straight away.

This is just some brainstorming, please feel free to find and couple back any benefits and/ or drawbacks.

Hope it helps.

Anonymous’s picture

Anonymous (not verified) commented

Update:

Possible resource for duplicate item checking

Topic: deny duplicate node titles
URL: http://drupal.org/node/89599

Edit:

MySQL Query

This MySQL query checks for any duplicate titles in the regular Drupal aggregator title field, and displays all duplicate titles. Change the prefix to relevant table prefix: 

SELECT title, 
 COUNT(title) AS NumOccurrences
FROM prefix_aggregator_item
GROUP BY title
HAVING ( COUNT(title) > 1 )

Source:
http://www.petefreitag.com/item/169.cfm

Note:

Suggestion to open a book page to keep everything centralized.