FeedsImporter/FeedsScheduler should log better stats when refreshing. This will be very helpful when fixing performance problems later.

Comments

Version:» 6.x-1.x-dev

Also would be great to have an admin page showing each feed, last time updated, etc... nice quick browse page for admins.

From #870540-6: Option to suppress watchdog message: relate log entries to feed nodes so that they can be browsed by (together with the) feed node.

+1 Subscribe

Version:6.x-1.x-dev» 7.x-2.x-dev
Status:Active» Needs work
StatusFileSize
new7.85 KB

This is a first untested shot at moving all watchdog logged activity to a feeds log. The vision here is to log related to importer / source and higher detail and eventually provide a log tab on sources.

http://skitch.com/alexbarth/dhw6h/country-relief-web

This will also allow for much more verbose logging as we won't clog the watchdog log anymore.

- Add log messages when items are created/updated #948936: Add watchdog entry on node create/update

Title:Log statsMore detailed log
StatusFileSize
new32.81 KB

- Added update hook.
- Added views integration.
- Added default view.

See default view on a feed node:

http://skitch.com/alexbarth/dhigw/feeds-test-site

Open:

- log tab on standalone form is not working (you'll see I've reworked hook_menu() but log tab is still not showing).
- add a global log view on admin/reports/feeds.

Status:Needs work» Needs review
StatusFileSize
new36.36 KB

This patch adds:

- log tab on standalone form is not working (you'll see I've reworked hook_menu() but log tab is still not showing).
- Severity filter on both feed nodes and standalone form.

Missing:

An all subscriptions log on admin/report/feeds

StatusFileSize
new39.17 KB

This is complete now. Will commit shortly.

StatusFileSize
new46.86 KB

More tests revealed... that I forgot to export the improved view.

- Adds all subscriptions log on admin/reports/feeds, including necessary views integration.

Version:7.x-2.x-dev» 6.x-1.x-dev
Status:Needs review» Patch (to be ported)

This is committed now.

http://drupal.org/cvs?commit=441580

It would be great if someone wanted to backport this patch to 6.x 1.x, shouldn't be too hard and I think a lot of people could benefit from this improvement.

Status:Patch (to be ported)» Needs review
StatusFileSize
new45 KB

This is a quick port. It's still using views 3. There should probably be logging added to processors besides the node processor or we could move it into FeedsProcessor. Tests pass.

Looks like a good improvement; maybe I'll test tomorrow, but I would have to upgrade to Views 3.

Would it be out of scope to add logging of feed import times to this patch? I'm trying to figure out how long it takes for my feeds to import, to assess the scalability of the system. See http://drupalbin.com/18789 for how I'm trying to do it.

It looks like if I could get access to the currently active $importer, I could at least log the importer title using $importer->config['name'];. But could I also get the number of nodes imported?

If I'm not mistaken, it looks like the issue marked duplicate at #948936: Add watchdog entry on node create/update to show how many created/updated is not actually in the committed patch, or the one for 6.x. Am I correct?

Here's some code that adds more detailed logging for nodes created/updated. The only thing I haven't figured out yet is how to get the name of the feed from which the nodes are being imported:

<?php
 
public function process(FeedsImportBatch $batch, FeedsSource $source) {
   
timer_start('feeds_import');
   
// Keep track of processed items in this pass, set total number of items.
   
$processed = 0;
    if (!
$batch->getTotal(FEEDS_PROCESSING)) {
     
$batch->setTotal(FEEDS_PROCESSING, count($batch->items));
    }
    while (
$item = $batch->shiftItem()) {
   
// snip the loop code...
   
}
   
$timer_value = timer_stop('feeds_import');
   
$import_time = $timer_value['time'] / 1000;
   
// Set messages. (Evan: added logging to watchdog.)
    // TODO: The @feedname information is not here - find out how to get access to the $importer->config.
   
if ($batch->created) {
     
drupal_set_message(format_plural($batch->created, 'Created @number @type node.', 'Created @number @type nodes.', array('@number' => $batch->created, '@type' => node_get_types('name', $this->config['content_type']))));
     
watchdog('feeds', 'Feeds module created @number @type nodes from @feedname feed in @time seconds', array('@time' => $import_time, '@number' => $batch->created, '@type' => $this->config['content_type'], '@feedname' => $this->config['name']), WATCHDOG_DEBUG);
    }
    elseif (
$batch->updated) {
     
drupal_set_message(format_plural($batch->updated, 'Updated @number @type node.', 'Updated @number @type nodes.', array('@number' => $batch->updated, '@type' => node_get_types('name', $this->config['content_type']))));
     
watchdog('feeds', 'Feeds module updated @number @type nodes from @feedname feed in @time seconds', array('@time' => $import_time, '@number' => $batch->created, '@type' => $this->config['content_type'], '@feedname' => $this->config['name']), WATCHDOG_DEBUG);
    }
    else {
     
drupal_set_message(t('There is no new content.'));
     
watchdog('feeds', 'There was no new content in @feedname', array('@feedname' => $this->config['name']));
    }
   
$batch->setProgress(FEEDS_PROCESSING, FEEDS_BATCH_COMPLETE);
  }
?>

StatusFileSize
new35.27 KB

This is the patch from #12 updated to latest head with Views 2 support.

Status:Needs review» Postponed (maintainer needs more info)

looks like some of this got in already, so I'm marking this postponed until it's sorted out.