LOAD DATA LOCAL INFILE command

Hello stella,

I had the same problem and changed this line.

xc_oaiharvester_bridge.module

function xc_oaiharvester_bridge_load_csv
[Befor]
$sql = "LOAD DATA LOCAL INFILE '" . $path . "' INTO TABLE " . $table
. " CHARACTER SET utf8 FIELDS TERMINATED BY '\\0\\t' ESCAPED BY '' LINES TERMINATED BY '\\0\\n'";

[After]
$sql = "LOAD DATA INFILE '" . $path . "' INTO TABLE " . $table
. " CHARACTER SET utf8 FIELDS TERMINATED BY '\\0\\t' ESCAPED BY '' LINES TERMINATED BY '\\0\\n'";

Hope this helps.
Best wishes!

Hi,

it seems, that there are two different issues here:

1) LOAD needs special privilege in MySQL, which is a global privilege, and not a database specific one.
Stella has right, that ignored this when I found this solution. The reason was that according to benchmarks the LOAD DATA INFILE construction works much much more quicker than INSERT (see for example Kevin van Zonneveld's post at http://kevin.vanzonneveld.net/techblog/article/improve_mysql_insert_perf...). In the code there is a condition if it have to use INSERT or LOAD, but it is not driven by a user modifiable setting. Is it OK if I modify the code, that the admin is able to decide whether he wants to use LOAD DATA INFILE or INSERT command.

2) with/without the LOCAL keyword. When I introduce this in the systems I tested, it worked only rith LOCAL, and it doesn't worked without LOCAL. Maybe I should run another tests. This can be also a administer modifiable variable, because without local the CSV files have to located on the same server where MySQL operates, which is not always true.

I commited changes for solving the first issue. It needs an update (i.e. run update.php), because the database structure were changed. The admin can choose between LOAD DATA INFILE and INSERT in the second page of schedule editing form. I wrote the following text on the form, plase review it, and let me know whether it is understandable.

"Use MySQL INSERT method for creating records in batch mode instead of default LOAD DATA INFILE?:
* Use CSV file and the SQL command LOAD DATA INFILE
* Use SQL command INSERT
To import the harvested records into relational database the Toolkit can use CSV files, and LOAD DATA INFILE commands or the more traditionally INSERT command. The first is much more quicker, but it requires general (and not database) level privilege in MySQL, which is not available in all situation. If you chose this, be very careful with privilege settings. The INSERT is slower command, but it requires only database level privileges, so it is more secure. If you have millions of records, and time is an important factor, consider using LOAD DATA INFILE with carefully, if you don't have as many record, or time is not an important factor, use the INSERT command."

I need more tests, but the bulk of the solution has to work right now.