Posted by vasan.85 on April 29, 2012 at 4:21pm
19 followers
| Project: | Feeds |
| Version: | 7.x-2.x-dev |
| Component: | Feeds Import |
| Category: | bug report |
| Priority: | major |
| Assigned: | Unassigned |
| Status: | needs work |
Issue Summary
Hi,
Feeds not import during cron runs but while i trying to import manually means data successfully imported.I using drupal 7.10. job scheduler and cron ran successfully without any errors.any one guide me to sort it issue.
thanks.
Comments
#1
any one pls.......
#2
(answered my own question)
#3
Ok so from what i can gather, it's a lie that this module will import things at each cron run, what it really means is - it will schedule something to happen when ever someone submits a form of a particular content type, and this scheduling happens at each cron interval.
Supplied patch cleans up this ambiguity in the interface (fixes some labels) and handles imports inside hook_cron
#4
#5
Doh just noticed this module implements http://api.drupal.org/api/drupal/modules!system!system.api.php/function/hook_cron_queue_info/7
#6
So, it seems the data structure returned by feeds_cron_job_scheduler_info() is not quite correct, as the job_schedule module is ignoring it (wrong structure)
#7
Patch supplied, makes hook_cron schedule jobs with out a node-type binding too
#8
Hello! I had the same problem and your patch solved it! Thank you!
#9
#10
#11
+++ b/feeds.moduleundefined@@ -42,12 +42,19 @@ function feeds_hook_info() {
+ if (!strlen($feeds[$id]->config['content_type'])) {
The above *should* be catching this without an extra check. The feed_nid should be 0 for standalone feeds.
+++ b/includes/FeedsSource.incundefined@@ -516,6 +516,12 @@ class FeedsSource extends FeedsConfigurable {
+
+ if ($this->feed_nid == 0) {
+ // some exports are not bound to a node, so we assume EXPORT_IN_CODE otherwise they will never execute
+ $this->export_type = EXPORT_IN_CODE;
+ }
Is this really the case?? This is a bug, and not the correct workaround.
Overall, this patch is taking the wrong approach. If this is really the behavior you're seeing then something is seriously wrong. Standalone feeds get their feed_nid set to 0, that is all.
#12
Yeah, well then you have two seperate issues, because i stepped thru the code with a breakpoint debugger and this is where it was failling, after i fixed the issues with the cron not scheduling
#13
My current project has major problems with feeds (2.0-alpha7) set to run in cron (as often as possible). The feeds cron hook is executed every few minutes, but the import page may state any amount of time for last import (usually between 20 min - 1 hour). It has to be imported often and reliably.
So, is the patch at #7 safely useable (@twister & @dgltmoon)? From what I understood it works, but the solution is considered partially wrong.
#14
I have issues as well with 50 feeds or so... in different importers using different parsers, but the might be clearing up. I think so, I'll report back tomorrow with my findings.
#15
bibo: I run the latest dev together with alysia cron.
I have run both of the patches in this page with patch -p1 < blablabla and they apply OK.
I run a crontab on my server every 5 minutes. This reflects OK in elysia cron.
After this, it gets a bit fuzzy because if I run Cron, the log will tell you that cron will have nothing to do, so you have to test this by fire off each cron task from the link on each task row. Even then , it might tell you that it has nothing to do. This is determined by the setting on each feed if it should import every 15 minutes, every 30 minutes etc.
It turns out that in my case it depends wheter it will respect this setting or not when running the cron task manually. Most of the times it will, and that is the best assumption to have, and what you have to do is simply to wait. Don't try to run the task a lot as its not going to help.
You can see in the feeds log (it has its own link in the reports menu) if something was imported or not. In the cron log you won't see a lot of detail but it is useful only if you enable the debug feature in elysia cron.
Basically, if you see cron running in elysia cron and if you can import manually and you run the latest dev and have applied both patches, it really should import, but it might take a while until it has imported everything.
I altered the worker time to 25 seconds from 15 seconds in feeds.module. If you have a lot of feeds it might spend the first half hour to just detemine how many nodes to place in the queue. After that it will start to import this queue. pay attention to the items number on the feed import cron task row in elysia cron.
this is not affected by what ever parser you use etc... i have a combination of parsers and execution times..
good luck!
#16
I'm not real clear which way forwards from here, twistor?
#17
Here's some hopefully related information after much troubleshooting. My importer (single CSV file, 70,000+ rows) was getting consistently stuck and only processing 1% at a time.
The fix for me was to override
$conf['feeds_process_limit']in settings.php to a much higher number. The default is to import 50 nodes/entities at a time. This setting is documented in README.txt.See also #1551246: How can I speed up import process?.
#18
OK i just created a new feed, and it doesnt seem to fire at cron time, nothing is being imported at run-time, my breakpoint debugger shows the fetcher is not being called either
my job_schedule table is empty
mysql> select * from job_schedule;Empty set (0.00 sec)
- Periodic Imports are set to 'as often as possible'
#19
I still dont think this feeds module is built to import data during cron runs, other than jobs that start in UI and run in the background after that. Can someone confirm this and provide some evidence?
#20
@dgtlmoon - We have at least two sites in production that are reliably running Feeds imports every day. They both use the File upload fetcher on a CSV file that is uploaded at regular increments. Not sure what evidence I can provide.
#21
There is definitely something wrong in Feeds, it's not working as it should. A simple solution is to import your feed in custom module by calling hook_cron
<?phpfunction MODULE_NAME_cron() {
$name = 'FEED_NAME';
$source = feeds_source($name);
$source->import();
}
?>
#22
@mErilainen does this work for you?
I've not written a module before, but it seems like this may be the best approach - please would you be willing to post the steps to to do this, for this example?
I can't seem to get Cron to Import (or update) Feeds using a HTTP Fetcher...
Thanks
UPDATE My issue appears to be with Feeds Self Node Processor sub-module - importing and updating a Feed attached to a Node. I am able to import a standard feed on Cron ok (using the module suggestion above).
It would be useful to extend this though - e.g. for batch importing etc. based on Fields values (or other criteria)
UPDATE 2 I've solved my Cron update issue for Feeds Self Node Processor - just in case it helps anyone else.
#23
I have a feed importer that imports product entities from a CSV file. In my CSV file there are approximately 2500 products.
I tried the solution provided in #21, but when I run my cron job it only imports 8%. When I manually run the cron job the second time it processes a next 2%, third time again 2%....
I use Elysia Cron to manage my cron jobs.
Anyone knows how to fix this?
#24
@sandravdv - It sounds like you're having the same issue I was. See my comment in #17.
#25
@Cottser increasing
$conf['feeds_process_limit']isn't a solution I think => if I set it to 2000 it still processes 2000 from the 2500 (or more) products at a time => feed still gets locked until next cron run...#26
@sandravdv - as far as I know,
$conf['feeds_process_limit']determines how many nodes/users/etc. are processed each cron run. Either increase that limit or schedule cron to run more frequently if you want the import to run faster.Edit: The setting also affects the UI/batch method of importing which could be problematic. Maybe we need to create a separate setting to set a limit for cron imports?
#27
@Cottser - I think it would be great to extend the Feeds module to give greater control over how Feeds are imported / updated on each Cron run. For example, allowing the user to configure:
- How to order the batches (e.g. by Node title A-Z, or by field values)
- How to prioritise the batches (e.g. by Node title A-Z, or by field values)
- Number in each batch (although this can be set as you suggest in #26 above)
This may be most relevant to users who are updating existing Nodes, rather than importing new ones. So may need to extend to include Feeds Self Node Processor as well (as per my comment above).
Unfortunately I'm still learning how to build modules...so this is currently beyond me....
#28
yeah, I think it's just that feeds does not schedule with job_scheduler when you submit the form, this is because Feed's for some bizzare idea is triggering the import by submitting a form of some type, which is totally bizzare behaviour but I guess they must have had a reason for it
#29
I also have the problem on http://www.nid-de-poule.net/ for importing the 'git commit log'.
#30
@Pol : That doesn't really help us...
#31
It wasn't meant to help, if I had time I would look into it, but right now, I'm quite busy with other modules... sorry.
#32
huh?