I downloaded and installed the latest version and got the node_import to run via cron. Using the node_import without the cron setup, I can successfully import all rows.

However, when using node_import via cron, after appending data to the file, only the first 25 rows have been imported.

I checked the node_import_tasks table, and the following values have been set on this task:

+---------+----------+-----------+--------+
| offset  | row_done | row_error | status |
+---------+----------+-----------+--------+
| 7749059 |       25 |         0 |      0 |
+---------+----------+-----------+--------+

I see no errors when viewing reports, or in the web server's error log. Does anyone have a solution or advice on inspecting the problem a little deeper?

Thanks.

Comments

jbova’s picture

jbova commented

It would appear that the offset is being incorrectly set as the file size, as opposed to the actual offset of the last record inserted. This only happens when the import is run via cron. It will only process 25 records, then stop.

This is not limited to the first 25 rows. It will import only 25 rows after any offset. So, I may start and then pause a task that is not currently assigned to cron via the "Import Appended Data". To do this, I simply navigate away from the Task View page. If 100 records are before checking the "Import Appended Data", then the box is checked, saved, and cron is run. It will result in a total of 125 rows being imported. At this time, the offset is set to the end of the file, and no more rows are imported.

I see that the node_import_cron() function within the node_import module reads as follows:

  module_load_include('inc', 'node_import');

  $unit = variable_get('node_import:cron:unit', 'rows');
  $count = variable_get('node_import:cron:count', 25);

  node_import_do_all_tasks($unit, $count);

I understand that the function may only process 25 rows at a time, but why is the offset being set to the end of the file, instead of where the import had left off? I'm guessing that I'll run into script execution timeouts or memory limits being exhausted of I change the value of 25 to something like 1,000,000.

I am using node_import and ni_cron to input large amounts of products into ubercart.

Alun’s picture

Alun commented

This seems to be a problem in the coding. Instead of setting the marker to the last imported item, the ni_cron module automatically assumes you have imported all the data in the file, and will only import new files.
This is done on line 34 in the code:

$filesize = filesize($filepath);

db_query("UPDATE {files} SET filesize = %d WHERE filepath = '%s'", $filesize, $filepath);
db_query("UPDATE {node_import_tasks} SET status = %d WHERE taskid = %d", NODE_IMPORT_STATUS_PENDING, $taskid);
				
if ($reset_offset)
db_query("UPDATE {node_import_tasks} SET offset = %d WHERE taskid = %d", $filesize, $taskid);

It seems the error is done on the if ($reset_offset) line.

The solution would be to change the line $filesize = filesize($filepath), but I wouldn't be able to tell you what to!!
Al

jbova’s picture

jbova commented

In the mean time, I did change the value of 25 to 1000000. This allows me to import an unlimited number of products. The script eventually times out, but the offset and record count is correct. The next time the cron job starts, it starts where it left off and continues the import. Using this method, I have imported over 30K records and counting.

I'll look into the bug further over the next few days.

dealancer’s picture

dealancer commented

I think that this couldn't be easily changed without modification of Node Import module. There should be added additional hook or modified existent which will tell correct offset.

dealancer’s picture

dealancer commented
Status: Active » Fixed

Current bug is fixed in new release (1.3).

Problem was in line #13 :)) Arguments was passed to reset offset to the end of file. Now the problem has gone, please, check it.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.