FeedAPI GC: handle items what are not in the feed anymore

Subscribing. This would be extremely useful, as we want to use RSS to mirror content from another site. I haven't yet had time to see if the patch applies against 1.4...

This would be great. I'd love a way to set a max number of feed items for a given feed. Or even just a way to set a max number to be fetched at cron or on refresh.

Possible already???

This is not possible at the moment. Also unfortunately I don't plan to add new features to the 5.x branch. If someone step up with a patch, i happily review it.

Are maxes and or throttling planned for V6?

I'm about to start on a feed aggregation project, that might enable me to spend time on this! :)

Subscribing

I've created a patch for the 6.x branch that I think provides the functionality that the original poster mentioned (thanks to therzog for some of the code).

Basically, there is a new option under the FeedApi Processor settings to delete old nodes that drop off the feed (see attachment for screenshot). If that box is checked then when a feed is refreshed (manually or through cron) any items that have fallen off the feed will be deleted.

It's working well for a couple of feeds I'm using at the moment, but definitely needs some testing and review.

Noticed that the changes to feedapi.module in the last patch only apply to the node processor, which might not be used in all cases. I don't like the idea of storing feed items as nodes and am working on a lighter weight processor, and am having the same "dropped" feeds issue.

This is the code I ended up writing within my own processor to make this work. It's called during the expire operation. Works well in preliminary testing.

   if ($settings['processors']['feedapi_toolbox']['delete_missing']) {
    $guids_to_keep = array();
    foreach ($feed->parsers as $parser) {
      $result = module_invoke($parser, 'feedapi_feed', 'parse', $feed);
      foreach($result->items as $item){
        if(!empty($item->options->guid)){
          $guids_to_keep[] = "'" . $item->options->guid . "'";
        }
      }
    }  
    if (!empty($guids_to_keep)) {
      $result = db_query("SELECT * FROM {feedapi_toolbox_item} WHERE nid = %d AND guid NOT IN (".implode(", ", $guids_to_keep).")", $feed->nid);
      while ($item = db_fetch_object($result)) {
        // We callback feedapi for deleting
        feedapi_expire_item($feed, $item);
      }
    }  
  }

NOTE: Code snipped updated since original post.

Has any of this been committed to the dev branch? It seems like mirroring the RSS feed should be a relatively trivial operation...

@chrism2671 - I do not believe so

Subscribe

We've recently had a similar requirement. I'm way to paranoid to actually delete items when they are not found on the feed (what happens if the feed blips?) - so I opted for unpublishing them. I wrote it as separate processor and called it FeedAPI garbage collector :)

Here's the module. If somebody wants to run with it ad maintain it on d.o. as separate project, I'm all for it.

I have another problem i that when I hit refresh on the feed to start creating nodes from a google calendar. I get a blank white screen.

#12 works for me. Simple and clean.

Sweet! it works. and if i change line 45 in "feedapi_gc.module" from

db_query('UPDATE {node} SET status = 0 WHERE nid = %d', $feed_item->nid);

to

node_delete($feed_item->nid);

It deletes those offending nodes. Thanks for this module.

#12 not working for me.

The 'FeedAPI Garbage Collector' is appearing in the content type as a Processor correctly, but it fails to appear in the actual feed item content. Only the 'FeedAPI Node' processor is showing and no items are ever unpublished (or deleted with #15's code)

subscribing

If you manually refresh the feed will it unpublish items?

If you can manually refresh the feeds and they work OK, but they don't update with a cron run you may want to check out this discussion: http://drupal.org/node/580508

I can now understand why there is not feedapi GC processor shown when editing the feed item (not content type) as there is no function feedapi_gc_feedapi_settings_form. This seems ok in that there are no settings required.

Strangely however, the function feedapi_gc_feedapi_after_refresh never runs on a manual or cron refresh for me (I'm using Firebug for Drupal to check this).

...

I have now found out why. The module_feedapi_after_refresh only gets to run if there has been a change in the actual feed. In my testing, my test machine had run cron after some items were removed from the feed but before adding the gc module.

Making a small change in the feed kicked it into gear.

This extra module approach seems the most elegant and bug free approach to solve this issue. Maybe it can be added to the feedapi standard add ons?

I suggest the following changes:
Change feedapi_gc.info from:

name = FeedAPI Garbage Collector
description = This FeedAPI processor unpublishes feed item nodes that are not present on the feed.
core = 6.x

To:

name = FeedAPI Garbage Collector
description = This FeedAPI processor unpublishes feed item nodes that are not present on the feed.
dependencies[] = feedapi
package = FeedAPI Add On
core = 6.x

AND add the following to the end of the README.txt

After installation, the feed must actually change before the garbage collector 
will run.

Attaching update version of alex_b's module for review

Whoops, the above attachment says 'items _deleted_' rather than 'items unpublished'

For flexibility, also inserted this code at line 45 of feedapi_gc.module

      // TO DO: change the following into an admin preference
      // Uncomment the next line to delete rather than unpublish
      //node_delete($feed_item->nid);
      // Comment the next line to delete rather than unpublish

Another issue has arisen with the message feedback to the user.

    drupal_set_message(t('!checked items checked, !removed items removed.', array('!checked' => $checked, '!removed' => $removed)));

is good and helpful unless it is a cron run, in which case the message is shown to the next Anonym visitor, confusing them. Therefore something like this code is needed:

  if (!$cron) {
    drupal_set_message(t('!checked items checked, !removed items removed.', array('!checked' => $checked, '!removed' => $removed)));
  }

However, there is no access to the $cron variable in the calling function from feedapi.module _feedapi_invoke_refresh so what to do?? As I see it there needs to be an API change to pass this variable in the hook_feedapi_after_refresh.

Namely, feedapi.module line ~1278:

  foreach (module_implements('feedapi_after_refresh') as $module) {
    $func = $module .'_feedapi_after_refresh';
    $func($feed);
  }

Change to:

  foreach (module_implements('feedapi_after_refresh') as $module) {
    $func = $module .'_feedapi_after_refresh';
    $func($feed, $cron);
  }

Or can someone think of a less disruptive method. No feedback message is one way...

Subscribing.

Subscribing

#21 johnmunro, I agree with the code comment about it being an admin setting, perhaps on a per-feed basis.

How to use this module?
After the module was enabled all my nodes come as "unpublished",but I need it could be in "published" state by default and change to "unpublished" after feed's items will removed.

If the module is working correctly it would only unpublish all nodes if the feed was inaccessible. (Which is something that probably needs work on as this is not a good behaviour)

sub

Subscribing.

There is an actual setting for this, FeedAPI is suppose to refresh to match the existing feed. Actually I think it used to when I used to use it.

Not sure why it doesn't now. But this is a very needed feature.

You could always run cron to delete feed items then refresh them.

subscribing

subscribing

I did couple of changes to make this tiny module more robust.
Changes:

• Use watchdog instead of set_message() and only log when we needed to delete or unpublish, also include the feed URL into the watchdog msg
• Now it's possible to configure the processor on a per-feed basis so you can decide if the items should be deleted or just unpublished.
• For delete, instead of node_delete(), use _feedapi_node_node_delete() to avoid perm check, at cron, we're anonymous...
• Some coding standard improvements
• Added feedapi_node as a requrement to the .info file

Please review, test and so on.
Known issue: the module totally messes up feed item statistics what is shown after feed refresh (new / updated), it's not trivial to fix.

If some folks review it and confirm that it works, i'm ready to include it in the next release of FeedAPI.

This Solution sounds nice.
I'll test it for a project i'm working on and will give feedback later on.

+1 subscribe

Anyone is eager to review #33 before commit?

#33 is just what we needed. It seems to work perfectly for our Google/iCal feed. I hope it will be integrated into the main branch of the module.