7.x-2.x-dev will begin automatically deleting nodes on cron run, despite having GUID mapped and periodic updates disabled.

CommentFileSizeAuthor
#15 1446516-feed-item-deletion.patch4.23 KBdgtlmoon
#13 feeds-deleting-nodes-7.x-2.0-alpha6.diff5.16 KBpdcarto
#10 feeds-deleting-nodes-1446516-10.patch5.01 KBtwistor

Comments

killtheliterate’s picture

killtheliterate commented

With a completely fresh install, this is still occuring. Using the stable release of Feeds.

Using csv parser, with periodic updates set to false. I'm using "process in background", with file upload. The number of nodes being created during import number about 50k, so I've adjusted the number of nodes processed per cron run by adjusting FeedsProcessor.inc.

This is completely crucial, as what use can feeds be if it auto deletes the content it builds?

Could this possibly be related to an error that I've seen during import, "Integrity Constraint Violation"

Straight up, Feeds is not respecting it's own setting of expiring nodes never.

killtheliterate’s picture

killtheliterate commented
Priority: Normal » Major

Looks like the expiry is definitely not getting set in the UI. If I look at the configuration for this feed in the Feeds UI, then I can see that nodes are set to expire "Never". Yet, in the database, I can see that the 'expire_period' is set to '3600'.

whaddupwitdat?

Here's a pastebin of the entire array, unserialized ---> http://pastebin.com/0c5U8LWA

killtheliterate’s picture

killtheliterate commented

I notice, also, that there are 2 jobs for each of my feeds in the job scheduler...

Dupe of this

http://drupal.org/node/1103606

Deleting the records from job_schedule will hopefully fix this, though these should never have been added in the first place. Any ideas?

emackn’s picture

emackn commented

You should probably use a module dedicated to importing a large number of nodes.

killtheliterate’s picture

killtheliterate commented

@emackn

I totally agree, I should have used migrate for this. But, I'm already pretty deep, and have to just kind of finish it now.

Deleting the records added to job_scheduler did fix this.

matt v.’s picture

matt v. commented

I was seeing the same issue. For what it's worth, clearing out the job_schedule table seems to have fixed the issue for me. Thanks!

Magic03’s picture

Magic03 commented

feeds 7.x-2.0-alpha5 - still not fixed

levakon’s picture

levakon commented
Version: 7.x-2.x-dev » 7.x-2.0-alpha5

same here...
in the middle of the import (right about 1h after starting to call cron job every 5 minutes) it starts deleting nodes,
is there any way to change this 'expire_period' to "never"?

EDIT:
it seems to work if I change "expire_period" to -1 in table feeds_importer, hope that is not also used for something else
oh, and i also cleared "clear" job from job_schedule table
I'm not sure which one did the trick, but hope it stays that way and someone else solves it the right way before i'm gonna learn enough to be able to do it (cause it might take a while for me to get there, still learning basic stuff...)

killtheliterate’s picture

killtheliterate commented

A really sloppy solution would be to conditionally delete records from the job_schedule table on cron, as that's when that entry is added, from what I can tell. For large amounts of stuff, I've begun using Migrate, as it seems more up to the task.

twistor’s picture

twistor commented
Status: Active » Needs review
Issue tags: +D7 stable release blocker
StatusFileSize
new feeds-deleting-nodes-1446516-10.patch5.01 KB

I haven't been able to reproduce this, but after going through the scheduling code, it's a mess.

furamag’s picture

furamag commented

I was able to reproduce this issue. I also had issue with hook_cron. Code which should run importer in hook_cron executed once and never executed again because we set drupal variable feeds_reschedule to FALSE after first run. Patch in #10 fixed both issues for me.

pdcarto’s picture

pdcarto commented

I had this problem and the patch in #10 solved it for me and is working in a production site. I'm tempted to change this bug status to "reviewed and tested by community", but I guess only two anecdotal reports of success may not warrant that.

Thank you twistor!!

pdcarto’s picture

pdcarto commented
StatusFileSize
new feeds-deleting-nodes-7.x-2.0-alpha6.diff5.16 KB

My attempt of a re-roll of that patch for alpha6.

dgtlmoon’s picture

dgtlmoon commented
Status: Needs review » Needs work

Rerolled your patch to current dev, looks like the tests are now failling with this patch..

"Other feeds_scheduler.test Line 182"

this fails

    $this->assertEqual(1, db_query("SELECT COUNT(*) FROM {job_schedule} WHERE type = 'syndication' AND name = 'feeds_importer_expire' AND period = 3600 AND last = :last", array(':last' => $new_min_last))->fetchField());
dgtlmoon’s picture

dgtlmoon commented
StatusFileSize
new 1446516-feed-item-deletion.patch4.23 KB

oops, attached

dgtlmoon’s picture

dgtlmoon commented

Also, @twistor - can you explain your fix a bit? seems you're removing the job schedule for deleting/expiring items?

twistor’s picture

twistor commented
Status: Needs work » Closed (duplicate)

It appears that what fixed this got into another issue. #1690434: Notice: Undefined variable: job in FeedsSource->scheduleClear() (line 319 of sites/all/modules/feeds/includes/FeedsSource.inc).

Closing this as a dupe.

At this time, there should always be one expiry entry, regardless. The expiry period is 3600, which means it runs every hour, but then it looks up the expiry time from the importer and won't run it expiration is disabled.

For anyone that thinks this is dumb, see #930652: Expiry batching broken. I could really use a review on that. thanks.