Posted by whoiskevin on October 4, 2007 at 11:59am
Jump to:
| Project: | User Import |
| Version: | 5.x-1.x-dev |
| Component: | Code |
| Category: | support request |
| Priority: | normal |
| Assigned: | Unassigned |
| Status: | closed (fixed) |
Issue Summary
I am getting a variety of errors when trying to import a large number of users (almost 700,000). My guess is that this is simply too many for this module to handle. Perhaps even for Drupal. Does anyone have any experience using this module to import large amounts of users?
Things I've tried.
1. Lowering the number to 10,000 at a time (I don't dare go any lower because it simply takes too long)
2. stopping and starting all services between imports (just trying to clear memory)
I've checked and re-checked the import file and it is fine. No problems that I can find there.
Comments
#1
is the send email option checked? the system may be trying to send an email to all your added users- which may be killing it on time. For this volume of data - I would recommend just importing the file directly into your mysql database via command line tools and not through the module. Let me know what your requirements are and I can pull together a simple mysq load command to do this.
#2
No email.
Thanks for the reply. I'm already putting together a direct mysql setting. Just wanted a sanity check before I did the work.
#3
User Import was designed to be highly scalable, by splitting imports over multiple cron runs.
I'd love to spend some time stress testing it to see where the bottlenecks are.
#4
whoiskevin, could you post, more or less, the mysql commands you've been using?? Just to get an idea... Thanks!
#5
Here is the section of code I had that writes the sql file I then imported into MySQL....
Basically it creates a user and then inserts the related profile_values using the data read from a tab separated file of user data.
This worked just fine. It was attempting to use the same file with the user import module that would fail (and nothing was turned on other than a straight import. no emails to users etc.)
// Insert the user into the database.fwrite($fh_userSQL, sprintf("INSERT INTO users (uid, name, pass, mail, created, status, init, login, theme, signature, language, picture) VALUES (%d, '%s', '%s', '%s', %d, %d, '%s', %d, '%s', '%s', '%s', '%s');\n",
$dbUserId, $dbUserName, md5($currentUser[7]), $currentUser[1],
strtotime($currentUser[10]), 1, $currentUser[1], strtotime($currentUser[8]), "", "", "", ""));
// Insert all the profile field values
fwrite($fh_userProfileSQL, sprintf("INSERT INTO profile_values (fid, uid, value) VALUES (%d, %d, '%s');\n", 1, $dbUserId, mysql_real_escape_string($currentUser[2])));
fwrite($fh_userProfileSQL, sprintf("INSERT INTO profile_values (fid, uid, value) VALUES (%d, %d, '%s');\n", 2, $dbUserId, mysql_real_escape_string($currentUser[3])));
fwrite($fh_userProfileSQL, sprintf("INSERT INTO profile_values (fid, uid, value) VALUES (%d, %d, '%s');\n", 25, $dbUserId, mysql_real_escape_string($currentUser[4])));
//fwrite($fh_userProfileSQL, sprintf("INSERT INTO profile_values (fid, uid, value) VALUES (%d, %d, '%s');\n", 5, $dbUserId, mysql_real_escape_string($currentUser[22])));
//fwrite($fh_userProfileSQL, sprintf("INSERT INTO profile_values (fid, uid, value) VALUES (%d, %d, '%s');\n", 6, $dbUserId, mysql_real_escape_string($currentUser[23])));
fwrite($fh_userProfileSQL, sprintf("INSERT INTO profile_values (fid, uid, value) VALUES (%d, %d, '%s');\n", 7, $dbUserId, mysql_real_escape_string($currentUser[24])));
fwrite($fh_userProfileSQL, sprintf("INSERT INTO profile_values (fid, uid, value) VALUES (%d, %d, '%s');\n", 23, $dbUserId, mysql_real_escape_string($currentUser[27])));
fwrite($fh_userProfileSQL, sprintf("INSERT INTO profile_values (fid, uid, value) VALUES (%d, %d, '%s');\n", 9, $dbUserId, mysql_real_escape_string($currentUser[26])));
fwrite($fh_userProfileSQL, sprintf("INSERT INTO profile_values (fid, uid, value) VALUES (%d, %d, '%s');\n", 10, $dbUserId, mysql_real_escape_string($currentUser[15])));
fwrite($fh_userProfileSQL, sprintf("INSERT INTO profile_values (fid, uid, value) VALUES (%d, %d, '%s');\n", 11, $dbUserId, mysql_real_escape_string($currentUser[16])));
fwrite($fh_userProfileSQL, sprintf("INSERT INTO profile_values (fid, uid, value) VALUES (%d, %d, '%s');\n", 13, $dbUserId, mysql_real_escape_string($currentUser[13])));
fwrite($fh_userProfileSQL, sprintf("INSERT INTO profile_values (fid, uid, value) VALUES (%d, %d, '%s');\n", 14, $dbUserId, mysql_real_escape_string($currentUser[14])));
fwrite($fh_userProfileSQL, sprintf("INSERT INTO profile_values (fid, uid, value) VALUES (%d, %d, '%s');\n", 17, $dbUserId, mysql_real_escape_string($currentUser[21])));
fwrite($fh_userProfileSQL, sprintf("INSERT INTO profile_values (fid, uid, value) VALUES (%d, %d, '%s');\n", 19, $dbUserId, mysql_real_escape_string($currentUser[17])));
fwrite($fh_userProfileSQL, sprintf("INSERT INTO profile_values (fid, uid, value) VALUES (%d, %d, '%s');\n", 20, $dbUserId, mysql_real_escape_string($currentUser[12])));
fwrite($fh_userProfileSQL, sprintf("INSERT INTO profile_values (fid, uid, value) VALUES (%d, %d, '%s');\n", 22, $dbUserId, mysql_real_escape_string($currentUser[])));
fwrite($fh_userProfileSQL, sprintf("INSERT INTO profile_values (fid, uid, value) VALUES (%d, %d, '%s');\n", 21, $dbUserId, mysql_real_escape_string($currentUser[5])));
fwrite($fh_userProfileSQL, sprintf("INSERT INTO profile_values (fid, uid, value) VALUES (%d, %d, '%s');\n", 24, $dbUserId, mysql_real_escape_string($currentUser[25])));