Posted by alex_b on October 16, 2009 at 5:18pm
6 followers
| Project: | Feeds |
| Version: | 6.x-1.x-dev |
| Component: | Code |
| Category: | task |
| Priority: | normal |
| Assigned: | Unassigned |
| Status: | postponed (maintainer needs more info) |
Issue Summary
FeedsImporter/FeedsScheduler should log better stats when refreshing. This will be very helpful when fixing performance problems later.
Comments
#1
Also would be great to have an admin page showing each feed, last time updated, etc... nice quick browse page for admins.
#2
Related: #870540: Option to suppress watchdog message
#3
From #870540-6: Option to suppress watchdog message: relate log entries to feed nodes so that they can be browsed by (together with the) feed node.
#4
+1 Subscribe
#5
This is a first untested shot at moving all watchdog logged activity to a feeds log. The vision here is to log related to importer / source and higher detail and eventually provide a log tab on sources.
http://skitch.com/alexbarth/dhw6h/country-relief-web
#6
This will also allow for much more verbose logging as we won't clog the watchdog log anymore.
- Add log messages when items are created/updated #948936: Add watchdog entry on node create/update
#7
- Added update hook.
- Added views integration.
- Added default view.
See default view on a feed node:
http://skitch.com/alexbarth/dhigw/feeds-test-site
Open:
- log tab on standalone form is not working (you'll see I've reworked hook_menu() but log tab is still not showing).
- add a global log view on admin/reports/feeds.
#8
This patch adds:
- log tab on standalone form is not working (you'll see I've reworked hook_menu() but log tab is still not showing).
- Severity filter on both feed nodes and standalone form.
Missing:
An all subscriptions log on admin/report/feeds
#9
This is complete now. Will commit shortly.
#10
More tests revealed... that I forgot to export the improved view.
- Adds all subscriptions log on admin/reports/feeds, including necessary views integration.
#11
This is committed now.
http://drupal.org/cvs?commit=441580
It would be great if someone wanted to backport this patch to 6.x 1.x, shouldn't be too hard and I think a lot of people could benefit from this improvement.
#12
This is a quick port. It's still using views 3. There should probably be logging added to processors besides the node processor or we could move it into FeedsProcessor. Tests pass.
#13
Looks like a good improvement; maybe I'll test tomorrow, but I would have to upgrade to Views 3.
Would it be out of scope to add logging of feed import times to this patch? I'm trying to figure out how long it takes for my feeds to import, to assess the scalability of the system. See http://drupalbin.com/18789 for how I'm trying to do it.
It looks like if I could get access to the currently active $importer, I could at least log the importer title using
$importer->config['name'];. But could I also get the number of nodes imported?#14
If I'm not mistaken, it looks like the issue marked duplicate at #948936: Add watchdog entry on node create/update to show how many created/updated is not actually in the committed patch, or the one for 6.x. Am I correct?
Here's some code that adds more detailed logging for nodes created/updated. The only thing I haven't figured out yet is how to get the name of the feed from which the nodes are being imported:
<?php
public function process(FeedsImportBatch $batch, FeedsSource $source) {
timer_start('feeds_import');
// Keep track of processed items in this pass, set total number of items.
$processed = 0;
if (!$batch->getTotal(FEEDS_PROCESSING)) {
$batch->setTotal(FEEDS_PROCESSING, count($batch->items));
}
while ($item = $batch->shiftItem()) {
// snip the loop code...
}
$timer_value = timer_stop('feeds_import');
$import_time = $timer_value['time'] / 1000;
// Set messages. (Evan: added logging to watchdog.)
// TODO: The @feedname information is not here - find out how to get access to the $importer->config.
if ($batch->created) {
drupal_set_message(format_plural($batch->created, 'Created @number @type node.', 'Created @number @type nodes.', array('@number' => $batch->created, '@type' => node_get_types('name', $this->config['content_type']))));
watchdog('feeds', 'Feeds module created @number @type nodes from @feedname feed in @time seconds', array('@time' => $import_time, '@number' => $batch->created, '@type' => $this->config['content_type'], '@feedname' => $this->config['name']), WATCHDOG_DEBUG);
}
elseif ($batch->updated) {
drupal_set_message(format_plural($batch->updated, 'Updated @number @type node.', 'Updated @number @type nodes.', array('@number' => $batch->updated, '@type' => node_get_types('name', $this->config['content_type']))));
watchdog('feeds', 'Feeds module updated @number @type nodes from @feedname feed in @time seconds', array('@time' => $import_time, '@number' => $batch->created, '@type' => $this->config['content_type'], '@feedname' => $this->config['name']), WATCHDOG_DEBUG);
}
else {
drupal_set_message(t('There is no new content.'));
watchdog('feeds', 'There was no new content in @feedname', array('@feedname' => $this->config['name']));
}
$batch->setProgress(FEEDS_PROCESSING, FEEDS_BATCH_COMPLETE);
}
?>
#15
This is the patch from #12 updated to latest head with Views 2 support.
#16
looks like some of this got in already, so I'm marking this postponed until it's sorted out.