I've managed to get a better handle on the Feeds import process using the Elysia Cron module to separate the 2 cron processes used by Feeds. I now run "feeds_cron" and "job_scheduler_cron" on its own channel every morning at 4am. By setting a high value (1800) for "stuck time" and "time limit", I'm able to import my 4700+ product feed completely every day.
Before using the Elysia Cron module, the feed import timing was attached to whatever my default cron schedule was, so once a day it would import about 2/3rds of my approx. 4,800 product nodes but time out before finishing them all. Since cron was only running once a day, Apache Solr would have to be manually triggered to index all the new items because the highest setting of 200 nodes per cron run wasn't enough.
Now I have a "search index" channel running every 10 minutes, a "product import" Feeds channel running once a day in the early morning when traffic is low, and the "default" channel running the rest of the cron processes once a day.
Thanks to the Elysia Cron module I feel much more in control of the Feeds import process now, it seems like it'd be a helpful thing to mention in the documentation.
Comments
Comment #1
rvdtuin CreditAttribution: rvdtuin commentedthanks for the information.
I installed ultimate cron but this doesn't give me what I want (at least I can't figure out how...)
I am using a couple of feeds importers that I would like to import at different intervals. because for some feeds (importers) it is necessary to import every hour and for some erery ten minutes etc. with ultimate cron all the feed importers run all at the same time.
Is there a way with Elysia Cron to set a run per importer?? ( I couldn't read this in your article, but maybe this is because of my lack of english...)
Comment #2
OldAccount CreditAttribution: OldAccount commentedAs I understand it, Elysia Cron just splits up all the various cron hooks, but feeds_cron is still one big process. Have you tried setting the Periodic Import time for each feed (admin/structure/feeds/feed_name/settings)? Seems like if you set one to "Every 15 min" and the other to "Every 1 hour", then use Elysia Cron to run feeds_cron and job_scheduler_cron at least every 15 minutes, that should work..?
Comment #3
rvdtuin CreditAttribution: rvdtuin commentedThanks for replaying,
Yes I have tried that and that is a good idea for now. But my problem is that I need to be able to import some feeds every minute, some every 10 min and some every 20 etc. So is there a workaround feeds_cron's one big processes and get control over the feed importers separatly (with Elysia Cron or some other module)?
Comment #4
OldAccount CreditAttribution: OldAccount commentedThe suggestion I gave in my previous comment is the best I could think of. If you need something more complex I'd suggest submitting a support request. I'm sure there are other instances where people have needed to add different time options like you need (1 min, 10 min, etc.), just try searching the issue queue.
Comment #5
rvdtuin CreditAttribution: rvdtuin commentedOke thanks.
(where should I post my request? feeds module?)
Comment #6
OldAccount CreditAttribution: OldAccount commentedYes, your request is related to the Feeds module, not Elysia Cron. But first you should check to see if someone has asked the same question already: http://drupal.org/project/issues/feeds?text=&status=All&priorities=All&c...
Comment #7
swfindlay CreditAttribution: swfindlay commented@lrobeson - Thank you for your guidance set out at the start of this post. Please can you set out the exact steps in a little more detail?
I'm struggling to replicate your configuration - I think I need to amend my cron.php - but am not sure what to do. Many thanks in advance for your help!
Comment #8
OldAccount CreditAttribution: OldAccount commented@swfindlay, which did you follow in the Elysia Cron install directions, the basic setup or the Step B section?
Also, how are you currently running cron: manually, periodically via Drupal core, Acquia Network? And how often are you wanting to run cron?
Comment #9
cdmo CreditAttribution: cdmo commentedrvdtuin, did you ever post your question to the queue? Did you find a solution? I'm wrestling with the same problem.
Comment #10
ressa CreditAttribution: ressa commentedThanks @robeson, that's a great tip and something I knew I needed, but hadn't investigated yet. Would you mind posting an image of your Elysia Cron settings page, so that we can get the full picture of how to set it up the best way?
Also, how did you set your Feed up to solve this issue?
From: https://drupal.org/node/1453930#comment-5994096
Was starting fresh every day the solution? Thank you in advance :-)
Comment #11
ressa CreditAttribution: ressa commentedI have found a solution to my second question about expiring nodes with unavailable products, not by using Feeds, but in stead un-publishing them with this custom module, posted by @rodtatham: https://drupal.org/node/628152#comment-7418446
Comment #12
sinn CreditAttribution: sinn commentedInstead of Elysia you can try to use ultimate_cron. I've just developed feeds and ultimate_cron integration https://drupal.org/sandbox/sinn/2073403
Comment #13
ressa CreditAttribution: ressa commentedThanks for the tip @sinn, but I have since managed to set it up by combining the above mentioned module, where I compare a time-stamp, and unpublish nodes older than 7 days, with Feeds Import in Elysia, by running "job_scheduler_cron".
Comment #14
mErilainen CreditAttribution: mErilainen commentedThe only solution I have found for running a specific importer is to define it in a custom module:
this way you can create as many custom modules as you need and then run them when you need to with Elysia cron.
Comment #15
autopoietic CreditAttribution: autopoietic commented@mErilainen I actually quite like your method, it avoids hacking around too much, and drupal cron now sees these as different tasks, whilst feeds still handles the batch processing etc.
Shame to have to create multiple modules, but otherwise I guess you would need to put the timing logic inside hook_cron rather than using Elysia Cron or equivalent, which is more easily configurable etc.
Anyway, this will work for my current usage, where I am only indexing a limited number of sites (3 or 4). Thanks!
Comment #16
maxplus CreditAttribution: maxplus commentedHi,
thanks @mErilainen
Just for others: In the first place I made a small mistake in the function name: modulename_site_cron
=> the function name should be modulename_cron, like this: