Hi,

Feeds not import during cron runs but while i trying to import manually means data successfully imported.I using drupal 7.10. job scheduler and cron ran successfully without any errors.any one guide me to sort it issue.

thanks.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

vasan.85’s picture

any one pls.......

dgtlmoon’s picture

Category: bug » support
Priority: Major » Normal
Status: Needs review » Active

(answered my own question)

dgtlmoon’s picture

Category: support » bug
Priority: Normal » Major
FileSize
1.94 KB

Ok so from what i can gather, it's a lie that this module will import things at each cron run, what it really means is - it will schedule something to happen when ever someone submits a form of a particular content type, and this scheduling happens at each cron interval.

Supplied patch cleans up this ambiguity in the interface (fixes some labels) and handles imports inside hook_cron

dgtlmoon’s picture

Status: Active » Needs review
dgtlmoon’s picture

dgtlmoon’s picture

Status: Needs review » Active

So, it seems the data structure returned by feeds_cron_job_scheduler_info() is not quite correct, as the job_schedule module is ignoring it (wrong structure)

dgtlmoon’s picture

Status: Active » Needs review
FileSize
1.69 KB

Patch supplied, makes hook_cron schedule jobs with out a node-type binding too

dimakopu’s picture

Category: support » bug
Priority: Normal » Major
Status: Active » Needs review

Hello! I had the same problem and your patch solved it! Thank you!

dgtlmoon’s picture

Status: Needs review » Reviewed & tested by the community
dgtlmoon’s picture

Version: 7.x-2.0-alpha4 » 7.x-2.x-dev
twistor’s picture

Status: Reviewed & tested by the community » Needs work
+++ b/feeds.moduleundefined
@@ -42,12 +42,19 @@ function feeds_hook_info() {
+      if (!strlen($feeds[$id]->config['content_type'])) {

The above *should* be catching this without an extra check. The feed_nid should be 0 for standalone feeds.

+++ b/includes/FeedsSource.incundefined
@@ -516,6 +516,12 @@ class FeedsSource extends FeedsConfigurable {
+
+    if ($this->feed_nid == 0) {
+      // some exports are not bound to a node, so we assume EXPORT_IN_CODE otherwise they will never execute
+      $this->export_type = EXPORT_IN_CODE;
+    }

Is this really the case?? This is a bug, and not the correct workaround.

Overall, this patch is taking the wrong approach. If this is really the behavior you're seeing then something is seriously wrong. Standalone feeds get their feed_nid set to 0, that is all.

dgtlmoon’s picture

Yeah, well then you have two seperate issues, because i stepped thru the code with a breakpoint debugger and this is where it was failling, after i fixed the issues with the cron not scheduling

bibo’s picture

My current project has major problems with feeds (2.0-alpha7) set to run in cron (as often as possible). The feeds cron hook is executed every few minutes, but the import page may state any amount of time for last import (usually between 20 min - 1 hour). It has to be imported often and reliably.

So, is the patch at #7 safely useable (@twister & @dgltmoon)? From what I understood it works, but the solution is considered partially wrong.

ibuildit’s picture

I have issues as well with 50 feeds or so... in different importers using different parsers, but the might be clearing up. I think so, I'll report back tomorrow with my findings.

ibuildit’s picture

bibo: I run the latest dev together with alysia cron.

I have run both of the patches in this page with patch -p1 < blablabla and they apply OK.

I run a crontab on my server every 5 minutes. This reflects OK in elysia cron.

After this, it gets a bit fuzzy because if I run Cron, the log will tell you that cron will have nothing to do, so you have to test this by fire off each cron task from the link on each task row. Even then , it might tell you that it has nothing to do. This is determined by the setting on each feed if it should import every 15 minutes, every 30 minutes etc.

It turns out that in my case it depends wheter it will respect this setting or not when running the cron task manually. Most of the times it will, and that is the best assumption to have, and what you have to do is simply to wait. Don't try to run the task a lot as its not going to help.

You can see in the feeds log (it has its own link in the reports menu) if something was imported or not. In the cron log you won't see a lot of detail but it is useful only if you enable the debug feature in elysia cron.

Basically, if you see cron running in elysia cron and if you can import manually and you run the latest dev and have applied both patches, it really should import, but it might take a while until it has imported everything.

I altered the worker time to 25 seconds from 15 seconds in feeds.module. If you have a lot of feeds it might spend the first half hour to just detemine how many nodes to place in the queue. After that it will start to import this queue. pay attention to the items number on the feed import cron task row in elysia cron.

this is not affected by what ever parser you use etc... i have a combination of parsers and execution times..

good luck!

dgtlmoon’s picture

I'm not real clear which way forwards from here, twistor?

star-szr’s picture

Here's some hopefully related information after much troubleshooting. My importer (single CSV file, 70,000+ rows) was getting consistently stuck and only processing 1% at a time.

The fix for me was to override $conf['feeds_process_limit'] in settings.php to a much higher number. The default is to import 50 nodes/entities at a time. This setting is documented in README.txt.

See also #1551246: How can I speed up import process?.

dgtlmoon’s picture

OK i just created a new feed, and it doesnt seem to fire at cron time, nothing is being imported at run-time, my breakpoint debugger shows the fetcher is not being called either

my job_schedule table is empty

mysql> select * from job_schedule;
Empty set (0.00 sec)

- Periodic Imports are set to 'as often as possible'

dgtlmoon’s picture

I still dont think this feeds module is built to import data during cron runs, other than jobs that start in UI and run in the background after that. Can someone confirm this and provide some evidence?

star-szr’s picture

@dgtlmoon - We have at least two sites in production that are reliably running Feeds imports every day. They both use the File upload fetcher on a CSV file that is uploaded at regular increments. Not sure what evidence I can provide.

mErilainen’s picture

There is definitely something wrong in Feeds, it's not working as it should. A simple solution is to import your feed in custom module by calling hook_cron

<?php
function MODULE_NAME_cron() {
  $name = 'FEED_NAME';
  $source = feeds_source($name);
  $source->import();
}
?>
swfindlay’s picture

@mErilainen does this work for you?

I've not written a module before, but it seems like this may be the best approach - please would you be willing to post the steps to to do this, for this example?

I can't seem to get Cron to Import (or update) Feeds using a HTTP Fetcher...

Thanks

UPDATE My issue appears to be with Feeds Self Node Processor sub-module - importing and updating a Feed attached to a Node. I am able to import a standard feed on Cron ok (using the module suggestion above).

It would be useful to extend this though - e.g. for batch importing etc. based on Fields values (or other criteria)

UPDATE 2 I've solved my Cron update issue for Feeds Self Node Processor - just in case it helps anyone else.

SandraVdv’s picture

I have a feed importer that imports product entities from a CSV file. In my CSV file there are approximately 2500 products.
I tried the solution provided in #21, but when I run my cron job it only imports 8%. When I manually run the cron job the second time it processes a next 2%, third time again 2%....
I use Elysia Cron to manage my cron jobs.
Anyone knows how to fix this?

star-szr’s picture

@sandravdv - It sounds like you're having the same issue I was. See my comment in #17.

SandraVdv’s picture

@Cottser increasing $conf['feeds_process_limit'] isn't a solution I think => if I set it to 2000 it still processes 2000 from the 2500 (or more) products at a time => feed still gets locked until next cron run...

star-szr’s picture

@sandravdv - as far as I know, $conf['feeds_process_limit'] determines how many nodes/users/etc. are processed each cron run. Either increase that limit or schedule cron to run more frequently if you want the import to run faster.

Edit: The setting also affects the UI/batch method of importing which could be problematic. Maybe we need to create a separate setting to set a limit for cron imports?

swfindlay’s picture

@Cottser - I think it would be great to extend the Feeds module to give greater control over how Feeds are imported / updated on each Cron run. For example, allowing the user to configure:

- How to order the batches (e.g. by Node title A-Z, or by field values)
- How to prioritise the batches (e.g. by Node title A-Z, or by field values)
- Number in each batch (although this can be set as you suggest in #26 above)

This may be most relevant to users who are updating existing Nodes, rather than importing new ones. So may need to extend to include Feeds Self Node Processor as well (as per my comment above).

Unfortunately I'm still learning how to build modules...so this is currently beyond me....

dgtlmoon’s picture

yeah, I think it's just that feeds does not schedule with job_scheduler when you submit the form, this is because Feed's for some bizzare idea is triggering the import by submitting a form of some type, which is totally bizzare behaviour but I guess they must have had a reason for it

Pol’s picture

I also have the problem on http://www.nid-de-poule.net/ for importing the 'git commit log'.

dgtlmoon’s picture

@Pol : That doesn't really help us...

Pol’s picture

It wasn't meant to help, if I had time I would look into it, but right now, I'm quite busy with other modules... sorry.

dgtlmoon’s picture

huh?

osopolar’s picture

Component: Feeds Import » Code

Any news/progress on this?

osopolar’s picture

If process_in_background is not TRUE, FeedsSource::startImport(); calls FeedsSource::startBatchAPIJob which adds a new batch by calling batch_set(). But the batch still didn't got executed.

Using the "standalone form" the batch gets progressed when the batch processing page gets rendered (_batch_page()). Not rendering the batch processing page requires a call to batch_process(); as pointed out in the BatchAPI documentation.

Actually I haven't tested using batch_process() because I use a drush script for the import and therefore I need to use drush_backend_batch_process() (see: Processing batch jobs with drush).

Vacilando’s picture

Creating a separate cron call for a feed import task is indeed a workaround for this problem, as suggested in #21 above. An Elysia solution for one or more feed import tasks is here: http://drupal.stackexchange.com/a/82422/196

don01001100’s picture

Is there a recommended work-around for this? I'd like to be able to have the feeds importer run once every fifteen minutes automatically. Thanks!

mErilainen’s picture

Issue summary: View changes

Using the code from #21 will give you a custom cron task which you can time with Elysia Cron to be run every fifteen minutes. The cron task will appear as the name of your custom_module. The link in #35 describes the whole process in code without using Elysia Cron user interface for the timing.

twistor’s picture

#1231332: periodic import imports only one file per cron was recently committed. This will attempt to re-queue a feed during cron so that it can import as many items as possible in the given time. The time was also raised from 15 to 60 seconds.

rameshbalda’s picture

@mErilainen #21 it works for me thank you...

saurabh.tripathi.cs’s picture

Hi i tried #21
i got below error

Exception: Download of failed with code -1002. in FeedsHTTPFetcherResult->getRaw() (line 33 of /var/www/.example.com/sites/all/modules/feeds/plugins/FeedsHTTPFetcher.inc).

On searching i found that this error could be due to CURL library missing.
I have installed CURL library also.But still i get same issue.

mErilainen’s picture

I don't think your problem has anything to do with this issue. Maybe this issue is more relevant #752862: Download of feed failed with code 0..

saurabh.tripathi.cs’s picture

Status: Needs work » Closed (duplicate)

Thanks @mErilainen,

Anyways i got it working so if anyone else faces this issue below is what i did.

Actually my feed was not made selecting using standalone form in Attach to content type select box which is found in basic settings of feed importer, it was using a content type 'feed'. So i passed the nid of the 'feed' and it worked.

 function MODULE_NAME_cron() {
  $nid = 408; //nid of content feed, try to get this using a query by using feed name instead of using static value.
  $name = 'feed name';
  $source = feeds_source($name,$nid);
  $source->import();
}
jamesquinton’s picture

Hi,

I came across this issue yesterday on a project - I had the feed Periodic import set to every 30 minutes, however cron was not triggering the import as it should.

However, after setting the Periodic import to 'As often as possible' cron now triggers the import once per cron run.

raj45’s picture

Feeds works for me, by setting Periodic import to As often as possible and running cron. It seems like I had to trigger Cron twice though ... I will keep an eye on it, and experiment some more.

gokul0801’s picture

I was having a similar issue on our site. The feeds importer was working fine when importing manually with a standalone form , but the cron job was not updating the feed automatically. (the cron job was working ok for a few months and then stopped working last week). Tried everything - different periodic import settings, reinstalled the feeds module, changing fields in the feeds_importer table as discussed in another thread,.. nothing. cron job still wont create new nodes.
Thanks to mErilainen. that solution above worked great. created a custom module to trigger the cron job and used Elysia cron to schedule this job.

kenorb’s picture

capysara’s picture

#21 worked for me. Thanks!