set a feed to a node, refresh and wait for the cron.
If the cron.php e run manually nothing special happens maybe because the feed was updated a short time ago.

Comments

alex_b’s picture

alex_b commented
Status: Active » Postponed (maintainer needs more info)

Could you post the feed that is creating duplicates here? What is your feedapi configuration?

Ian Ward’s picture

Ian Ward commented

I also experience this. It seems to happen on cron, as well as when I hit the "Refresh" tab on the feed. It happens for me w/ many feeds (seems like all feeds) so maybe something in the module as opposed to a feed issue. However, here's one feed for example http://blogsearch.google.com/blogsearch_feeds?hl=en&q=Rwanda&ie=utf-8&nu... It does not actually create duplicates, but rather updates or resaves all the items even though nothing on them has changed.

xpete’s picture

xpete commented

@Ian Ward
Are you still having this problem, I think it's fixed but i will do more tests.

xpete’s picture

xpete commented

NOT fixed.
"Update existing feed items" is disabled

alex_b’s picture

alex_b commented

I just committed another duplicate-check fix ( http://drupal.org/node/192589 ). Probably this does it for you. If you are in need of a de-dupe script, let me know. How many dupes do you have?

xpete’s picture

xpete commented

Still NOT fixed.
"Update existing feed items" is enabled

xpete’s picture

xpete commented

I'm using feedapi_node and simplepie parser.
The nodes aren't repeated in feedapi_node_item table but they are repeated in the node table.

alex_b’s picture

alex_b commented

So there exist nodes in the node table, that are feed items, but don't have a corresponding entry in feedapi_node_item table?

Could you run the following query and check wether there are feed items that don't have an entry in feedapi_node_item? (If you haven't created any nodes of the feed item content type by hand, this query shouldn't yield any results).

# replace "story" by your feed item content type. 
SELECT n.nid, fi.nid, n.type FROM node n LEFT JOIN feedapi_node_item fi ON fi.nid = n.nid WHERE fi.nid IS NULL AND type = "story"

Then, could you run the next query, to figure out wether there are feedapi entries that dont have a corresponding node entry?

SELECT n.nid, f.nid, n.type FROM feedapi f LEFT JOIN node n ON n.nid = f.nid WHERE n.nid IS NULL

What I would like to find out with those queries is wether there are feed records in the feedapi table without nodes, that keep creating the same feed items as your good feed records.

xpete’s picture

xpete commented

>So there exist nodes in the node table, that are feed items, but don't have a corresponding entry in >feedapi_node_item table?
yes

Per each feed item in the feedapi_node_item table I have more than one node and it increases in each cron run.

>Could you run the following query and check wether there are feed items that don't have an entry in >feedapi_node_item? (If you haven't created any nodes of the feed item content type by hand, this query >shouldn't yield any results).

I have lots of nodes created by hand of the feed item content type and all returns NULL like it should be.

>SELECT n.nid, f.nid, n.type FROM feedapi f LEFT JOIN node n ON n.nid = f.nid WHERE n.nid IS NULL

returns nothing.
I had some feedapi table lines that didn't have a feedurl but i don't think that the problem is related to that

alex_b’s picture

alex_b commented

This is strange. We are running here the 5.x 1.x dev version on 6 systems downloading full throttle and from various sources (blog feeds, but also a lot of subscription feeds to search engines) but I dont get this problem.

Does this problem occur for _all_ of your feeds or only for some of them? If it occurs on a feed, does it occur on all feed items of the feed or only on some of them? Can you find a pattern in the URL or GUIDs of feeds or feed items which yield duplicates? There might be URLs or GUIDs that don't yield a result when you search for them in the DB.

BTW, is your feedapi_node_item table updated? The scheme changed in revision 1.1.2.3 http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/feedapi/fee...

If not - run the update scripts.

Alex

xpete’s picture

xpete commented

i only have one for feed:
http://sourceforge.net/export/rss2_projnews.php?group_id=90363&rss_fullt...
I have the problem on all feed items.
I have the database updated.

alex_b’s picture

alex_b commented

I just tried your feed and it seems to be working fine - no duplication on cron or on manual refreshing. Using Apache2, PHP5.2.2, MySQL5.0.41. What is your server environment?

xpete’s picture

xpete commented

Apache 1.3.37
MySQL 4.1.22
PHP 4.4.7

On each cron the nodes are deleted in the feeapi_node_item table but are'nt in the node table.
And the nodes are created again in both tables.
The nodes should not even be deleted.

alex_b’s picture

alex_b commented

To what expiry period ("Delete news items older than") have you set the feed?

xpete’s picture

xpete commented

I was set to 1 hour, now i've changed that to "never".

alex_b’s picture

alex_b commented

Does the error still occur? My hunch is that as the expiry date is set to 1 hour, the feedapi_node_item table entry is being created and deleted in the same cycle, but the node is still being created.

xpete’s picture

xpete commented

The error does not occur anymore. I think you are wrong, the problem is that the nodes are not deleted in the node table.

aron novak’s picture

aron novak
Primary language Hungarian
Location Hungary, Budapest
commented

I committed various bugfixes in the topic of feed item expiring. This bug should be fixed too, please give me feedback about this.

xpete’s picture

xpete commented
Status: Postponed (maintainer needs more info) » Fixed

It's fixed

xpete’s picture

xpete commented
Status: Fixed » Closed (fixed)