Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.
Hi,
Feeds not import during cron runs but while i trying to import manually means data successfully imported.I using drupal 7.10. job scheduler and cron ran successfully without any errors.any one guide me to sort it issue.
thanks.
Comment | File | Size | Author |
---|---|---|---|
#7 | 1553190-no-import-at-cron-fix-scheduling.patch | 1.69 KB | dgtlmoon |
#3 | 1553190-real-cron-processing.patch | 1.94 KB | dgtlmoon |
Comments
Comment #1
vasan.85 CreditAttribution: vasan.85 commentedany one pls.......
Comment #2
dgtlmoon CreditAttribution: dgtlmoon commented(answered my own question)
Comment #3
dgtlmoon CreditAttribution: dgtlmoon commentedOk so from what i can gather, it's a lie that this module will import things at each cron run, what it really means is - it will schedule something to happen when ever someone submits a form of a particular content type, and this scheduling happens at each cron interval.
Supplied patch cleans up this ambiguity in the interface (fixes some labels) and handles imports inside hook_cron
Comment #4
dgtlmoon CreditAttribution: dgtlmoon commentedComment #5
dgtlmoon CreditAttribution: dgtlmoon commentedDoh just noticed this module implements http://api.drupal.org/api/drupal/modules!system!system.api.php/function/...
Comment #6
dgtlmoon CreditAttribution: dgtlmoon commentedSo, it seems the data structure returned by feeds_cron_job_scheduler_info() is not quite correct, as the job_schedule module is ignoring it (wrong structure)
Comment #7
dgtlmoon CreditAttribution: dgtlmoon commentedPatch supplied, makes hook_cron schedule jobs with out a node-type binding too
Comment #8
dimakopu CreditAttribution: dimakopu commentedHello! I had the same problem and your patch solved it! Thank you!
Comment #9
dgtlmoon CreditAttribution: dgtlmoon commentedComment #10
dgtlmoon CreditAttribution: dgtlmoon commentedComment #11
twistor CreditAttribution: twistor commentedThe above *should* be catching this without an extra check. The feed_nid should be 0 for standalone feeds.
Is this really the case?? This is a bug, and not the correct workaround.
Overall, this patch is taking the wrong approach. If this is really the behavior you're seeing then something is seriously wrong. Standalone feeds get their feed_nid set to 0, that is all.
Comment #12
dgtlmoon CreditAttribution: dgtlmoon commentedYeah, well then you have two seperate issues, because i stepped thru the code with a breakpoint debugger and this is where it was failling, after i fixed the issues with the cron not scheduling
Comment #13
bibo CreditAttribution: bibo commentedMy current project has major problems with feeds (2.0-alpha7) set to run in cron (as often as possible). The feeds cron hook is executed every few minutes, but the import page may state any amount of time for last import (usually between 20 min - 1 hour). It has to be imported often and reliably.
So, is the patch at #7 safely useable (@twister & @dgltmoon)? From what I understood it works, but the solution is considered partially wrong.
Comment #14
ibuildit CreditAttribution: ibuildit commentedI have issues as well with 50 feeds or so... in different importers using different parsers, but the might be clearing up. I think so, I'll report back tomorrow with my findings.
Comment #15
ibuildit CreditAttribution: ibuildit commentedbibo: I run the latest dev together with alysia cron.
I have run both of the patches in this page with patch -p1 < blablabla and they apply OK.
I run a crontab on my server every 5 minutes. This reflects OK in elysia cron.
After this, it gets a bit fuzzy because if I run Cron, the log will tell you that cron will have nothing to do, so you have to test this by fire off each cron task from the link on each task row. Even then , it might tell you that it has nothing to do. This is determined by the setting on each feed if it should import every 15 minutes, every 30 minutes etc.
It turns out that in my case it depends wheter it will respect this setting or not when running the cron task manually. Most of the times it will, and that is the best assumption to have, and what you have to do is simply to wait. Don't try to run the task a lot as its not going to help.
You can see in the feeds log (it has its own link in the reports menu) if something was imported or not. In the cron log you won't see a lot of detail but it is useful only if you enable the debug feature in elysia cron.
Basically, if you see cron running in elysia cron and if you can import manually and you run the latest dev and have applied both patches, it really should import, but it might take a while until it has imported everything.
I altered the worker time to 25 seconds from 15 seconds in feeds.module. If you have a lot of feeds it might spend the first half hour to just detemine how many nodes to place in the queue. After that it will start to import this queue. pay attention to the items number on the feed import cron task row in elysia cron.
this is not affected by what ever parser you use etc... i have a combination of parsers and execution times..
good luck!
Comment #16
dgtlmoon CreditAttribution: dgtlmoon commentedI'm not real clear which way forwards from here, twistor?
Comment #17
star-szrHere's some hopefully related information after much troubleshooting. My importer (single CSV file, 70,000+ rows) was getting consistently stuck and only processing 1% at a time.
The fix for me was to override
$conf['feeds_process_limit']
in settings.php to a much higher number. The default is to import 50 nodes/entities at a time. This setting is documented in README.txt.See also #1551246: How can I speed up import process?.
Comment #18
dgtlmoon CreditAttribution: dgtlmoon commentedOK i just created a new feed, and it doesnt seem to fire at cron time, nothing is being imported at run-time, my breakpoint debugger shows the fetcher is not being called either
my job_schedule table is empty
- Periodic Imports are set to 'as often as possible'
Comment #19
dgtlmoon CreditAttribution: dgtlmoon commentedI still dont think this feeds module is built to import data during cron runs, other than jobs that start in UI and run in the background after that. Can someone confirm this and provide some evidence?
Comment #20
star-szr@dgtlmoon - We have at least two sites in production that are reliably running Feeds imports every day. They both use the File upload fetcher on a CSV file that is uploaded at regular increments. Not sure what evidence I can provide.
Comment #21
mErilainen CreditAttribution: mErilainen commentedThere is definitely something wrong in Feeds, it's not working as it should. A simple solution is to import your feed in custom module by calling hook_cron
Comment #22
swfindlay CreditAttribution: swfindlay commented@mErilainen does this work for you?
I've not written a module before, but it seems like this may be the best approach - please would you be willing to post the steps to to do this, for this example?
I can't seem to get Cron to Import (or update) Feeds using a HTTP Fetcher...
Thanks
UPDATE My issue appears to be with Feeds Self Node Processor sub-module - importing and updating a Feed attached to a Node. I am able to import a standard feed on Cron ok (using the module suggestion above).
It would be useful to extend this though - e.g. for batch importing etc. based on Fields values (or other criteria)
UPDATE 2 I've solved my Cron update issue for Feeds Self Node Processor - just in case it helps anyone else.
Comment #23
SandraVdv CreditAttribution: SandraVdv commentedI have a feed importer that imports product entities from a CSV file. In my CSV file there are approximately 2500 products.
I tried the solution provided in #21, but when I run my cron job it only imports 8%. When I manually run the cron job the second time it processes a next 2%, third time again 2%....
I use Elysia Cron to manage my cron jobs.
Anyone knows how to fix this?
Comment #24
star-szr@sandravdv - It sounds like you're having the same issue I was. See my comment in #17.
Comment #25
SandraVdv CreditAttribution: SandraVdv commented@Cottser increasing
$conf['feeds_process_limit']
isn't a solution I think => if I set it to 2000 it still processes 2000 from the 2500 (or more) products at a time => feed still gets locked until next cron run...Comment #26
star-szr@sandravdv - as far as I know,
$conf['feeds_process_limit']
determines how many nodes/users/etc. are processed each cron run. Either increase that limit or schedule cron to run more frequently if you want the import to run faster.Edit: The setting also affects the UI/batch method of importing which could be problematic. Maybe we need to create a separate setting to set a limit for cron imports?
Comment #27
swfindlay CreditAttribution: swfindlay commented@Cottser - I think it would be great to extend the Feeds module to give greater control over how Feeds are imported / updated on each Cron run. For example, allowing the user to configure:
- How to order the batches (e.g. by Node title A-Z, or by field values)
- How to prioritise the batches (e.g. by Node title A-Z, or by field values)
- Number in each batch (although this can be set as you suggest in #26 above)
This may be most relevant to users who are updating existing Nodes, rather than importing new ones. So may need to extend to include Feeds Self Node Processor as well (as per my comment above).
Unfortunately I'm still learning how to build modules...so this is currently beyond me....
Comment #28
dgtlmoon CreditAttribution: dgtlmoon commentedyeah, I think it's just that feeds does not schedule with job_scheduler when you submit the form, this is because Feed's for some bizzare idea is triggering the import by submitting a form of some type, which is totally bizzare behaviour but I guess they must have had a reason for it
Comment #29
PolI also have the problem on http://www.nid-de-poule.net/ for importing the 'git commit log'.
Comment #30
dgtlmoon CreditAttribution: dgtlmoon commented@Pol : That doesn't really help us...
Comment #31
PolIt wasn't meant to help, if I had time I would look into it, but right now, I'm quite busy with other modules... sorry.
Comment #32
dgtlmoon CreditAttribution: dgtlmoon commentedhuh?
Comment #33
osopolarAny news/progress on this?
Comment #34
osopolarIf process_in_background is not TRUE, FeedsSource::startImport(); calls FeedsSource::startBatchAPIJob which adds a new batch by calling batch_set(). But the batch still didn't got executed.
Using the "standalone form" the batch gets progressed when the batch processing page gets rendered (_batch_page()). Not rendering the batch processing page requires a call to batch_process(); as pointed out in the BatchAPI documentation.
Actually I haven't tested using batch_process() because I use a drush script for the import and therefore I need to use drush_backend_batch_process() (see: Processing batch jobs with drush).
Comment #35
Vacilando CreditAttribution: Vacilando commentedCreating a separate cron call for a feed import task is indeed a workaround for this problem, as suggested in #21 above. An Elysia solution for one or more feed import tasks is here: http://drupal.stackexchange.com/a/82422/196
Comment #36
don01001100 CreditAttribution: don01001100 commentedIs there a recommended work-around for this? I'd like to be able to have the feeds importer run once every fifteen minutes automatically. Thanks!
Comment #37
mErilainen CreditAttribution: mErilainen commentedUsing the code from #21 will give you a custom cron task which you can time with Elysia Cron to be run every fifteen minutes. The cron task will appear as the name of your custom_module. The link in #35 describes the whole process in code without using Elysia Cron user interface for the timing.
Comment #38
twistor CreditAttribution: twistor commented#1231332: periodic import imports only one file per cron was recently committed. This will attempt to re-queue a feed during cron so that it can import as many items as possible in the given time. The time was also raised from 15 to 60 seconds.
Comment #39
rameshbalda CreditAttribution: rameshbalda commented@mErilainen #21 it works for me thank you...
Comment #40
saurabh.tripathi.cs CreditAttribution: saurabh.tripathi.cs commentedHi i tried #21
i got below error
Exception: Download of failed with code -1002. in FeedsHTTPFetcherResult->getRaw() (line 33 of /var/www/.example.com/sites/all/modules/feeds/plugins/FeedsHTTPFetcher.inc).
On searching i found that this error could be due to CURL library missing.
I have installed CURL library also.But still i get same issue.
Comment #41
mErilainen CreditAttribution: mErilainen commentedI don't think your problem has anything to do with this issue. Maybe this issue is more relevant #752862: Download of feed failed with code 0..
Comment #42
saurabh.tripathi.cs CreditAttribution: saurabh.tripathi.cs commentedThanks @mErilainen,
Anyways i got it working so if anyone else faces this issue below is what i did.
Actually my feed was not made selecting using standalone form in Attach to content type select box which is found in basic settings of feed importer, it was using a content type 'feed'. So i passed the nid of the 'feed' and it worked.
Comment #43
jamesquinton CreditAttribution: jamesquinton commentedHi,
I came across this issue yesterday on a project - I had the feed Periodic import set to every 30 minutes, however cron was not triggering the import as it should.
However, after setting the Periodic import to 'As often as possible' cron now triggers the import once per cron run.
Comment #44
raj45 CreditAttribution: raj45 commentedFeeds works for me, by setting
Periodic import
toAs often as possible
and running cron. It seems like I had to trigger Cron twice though ... I will keep an eye on it, and experiment some more.Comment #45
gokul0801 CreditAttribution: gokul0801 commentedI was having a similar issue on our site. The feeds importer was working fine when importing manually with a standalone form , but the cron job was not updating the feed automatically. (the cron job was working ok for a few months and then stopped working last week). Tried everything - different periodic import settings, reinstalled the feeds module, changing fields in the feeds_importer table as discussed in another thread,.. nothing. cron job still wont create new nodes.
Thanks to mErilainen. that solution above worked great. created a custom module to trigger the cron job and used Elysia cron to schedule this job.
Comment #46
kenorb CreditAttribution: kenorb commentedComment #47
capysara CreditAttribution: capysara commented#21 worked for me. Thanks!