Duplicate items

i would check the actual item that is duplicated. i have seen the same problem with ap feeds. the problem is syndication. ap provides stories to affiliates that also provide them with content. ap publishes an article with a title but the actual link will be from one of their syndicates so the id changes over time. i think this is to give their affiliate exposure. if you actually click on the different duplicates i think you will find that it actually changes where it goes. i've not actually looked at the google feed but you might check

when i was looking at the feedapi code i thought i saw that it was also checking on the title, but i've not checked back. something on my todo list.

the only solution for now i see is to actually take the feed from one of the affiliates that pumps out the entire syndicated feed and tag it as coming from the syndicate, if you are tagging them. watch out, some affiliates seem to only take a subset.

i would prefer that the items never get created in the first place rather than creating and then removing, cron time should not be given out to added work. when i get a chance in a few weeks i will look into the title duplicate check and contribute what i come up with (unless the maintainers pipe in and say it's done or do it themselves), been awhile since i've coded anything fresh and never php so it should be interesting.

This would be a very useful addition.

Well, you need overrideable deduping logic. Currently this is FeedAPI's own job to decide what is duplicate and what's not.
To achieve a clean solution now you need to patch feedapi. This is not a good way to do, but the only one.

Aron I think you are onto something.

What if hook_feedapi_after_parse would assign/override a unique identification hash (md5) for every feed item?

This id hash can default to feed item guid, url and feed url, but can be custom overwritten to be just the feed item title, etc..

This hash can be used more robustly for all deduping and updating operations.

Thanks for these ideas, I just found this thread. I guess my feature request is somewhat related to this post so I post the link here, I hope it is ok.
http://drupal.org/node/349082

I marked that post as a duplicate of this one.

Is the function we need to play with the following?
hook_feedapi_after_parse

In my case, my feed has a field with a unique id, which I am storing as a CCK text field. I would be checking against this one for uniqueness. This is a simple SELECT query and php if else, but I wonder where is the best place to make this check.

Closing after an extended period of inactivity.