The feed I try to leech is from the OnlineGame Guildwars: http://www.guildwars.com/support/gameupdates/rss/updates-current.xml

Leech (and all other modules I tried so far) only reads one item an keeps telling me all the other items are duplicates. That is because every item in the feed is linked with the same url: http://www.guildwars.com/support/gameupdates/

Is there any way to get over this obstacle and leech every item in a feed like that?

Thanks in advance.
XPectIT

Comments

alex_b’s picture

alex_b commented
Version: 5.x-1.8 » 5.x-1.x-dev

Huh, this is a good one. You could add an option to turn off checking for duplicates by link to the item, but then you would have to figure out another way of duplicate checking: By title? Or are there other good parameters?

Say, you find a good alternative way how to determine a duplicate, you will want to set the duplicate checking mode on a per feed basis.

This is a development version issue, I set the Version to 5.x-1.x-dev therefore.

Patches are appreciated.

alex_b’s picture

alex_b commented
Status: Active » Postponed (maintainer needs more info)

any news?

XPectIT’s picture

XPectIT commented

Sorry, I've been kind of busy lately.

I'm close to give it up.
The main problem I see is that there is no way to tell the items apart.

All items have the same URL. Some Items have also the same title. So I don't see any unique attribute to tell between duplicates and new items. I don't think it's a good idea to create an md5 out of the first 20 (or so) characters of the body but something like that seems the only solution.

alex_b’s picture

alex_b commented

So, do they only differ by their body?

If so, I would create an optional title+body-based duplicate checking on a per feed basis.

* store an md5 calculated from title+part of body
* if md5 equals, do full text comparison to be 100% sure about duplication