I needed to get this working in version 4.7 and after a lot of changes I think I have it working. I made no changes to the core functionality, but practically everything related to the forms and tables used to display the information needed massive changes. I'm not sure how to create a patch file, and there are so many changes it really is a new file, so I have just attached the changed file to this message. I'm sure it needs testing, but it seems to work for me.

The program uses a function called "is_int" to test numeric imput values and that function fails on my system. I changed all occurances of "is_int" to "is_numeric" which I believe is more reliable. That is the only thing I changed other than the form and table displays.

CommentFileSizeAuthor
user_import.module38.27 KBkarens

Comments

robert castelo’s picture

Assigned: Unassigned » robert castelo
Status: Active » Fixed

I've already updated User Import module for 4.7, and included a lot of code clean up and optomisations.

I haven't released it because the 4.7 API is still changing, so I can't be sure that what I developed for beta 3 will work in the final 4.7 release. I guess if people really want it now I can put it on the cvs, although I don't understand why anyone would be importing users into an application that hasn't even reached Release Candidate stage yet.

Anyway Karen, thanks for the effort you put into this, hope the module helps with your project.

karens’s picture

The reason I'm doing this is that I'm porting several large existing web sites over to Drupal. Since I'm just getting started, it seemed to make the most sense to go ahead and develop in 4.7 so that I don't have to do everything over again when it is officially available and everything I'm doing right now is offline so I can't really "break" anything (plus a number of modules have added some nice new features available only on 4.7 and I wanted to try them out). I'm on the latest cvs version of 4.7 and I've had very few problems, so it's quite encouraging.

I have a lot of existing users and content and I am porting things in so I can see how it all will work before I take it live, and the import users module will really help (plus I think I can tweak it to import other things I need) :-)

robert castelo’s picture

OK, just uploaded a 4.7 version of User Import module to cvs, it will take 24 hours to package it, so wait until tomorrow to start playing with it.

Update includes some new code for keeping track of filenames, which should solve some issues.

karens’s picture

Thanks for this, however I am having a problem that is not really your problem but affects the ability to use thiis module. When I try to upload a csv file, I get a message that mime magic reports it is not a valid file and I am unable to proceed. This is actually a problem that relates to the interaction of file.inc and mime magic. Mime magic is reporting that something about the file is failing its test for a valid csv file and the upload fails. What's happening behind the scenes is that the upload actually succeeds (I can see it hit my tmp directory), but file.inc has appended a .txt extension to the file name, so when you try to access or move the file it is not found. This is all related to Drupal's efforts to protect the system against the upload of dangerous or infected files, but there apparently are some problems with mime magic and some file types, at least in some php/apache configurations. There are dozens of posts about this on Drupal (see http://drupal.org/node/43220, for instance) so it is a known problem.

Anyway, in the earlier version of user upload I found a way to work around this by renaming the file so it had a .txt extension instead of a .csv extension. In this newer version of the module that no longer works. Using this latest version, if I try to upload a file with a .txt extension the upload silently fails (no error messages at all). I am guessing you added something somewhere to test that the uploaded file is a .csv file, some sort of test that wasn't present in the older module.

Anyway, if this is a problem for me it will be a problem for others. Not sure what, if anything, you want to do about it.

I will go back and use the modified I created earlier (posted on this thread), which works if I rename my file with a .txt extension.

robert castelo’s picture

Hi,
thanks for the bug report.

If you open the module and search for a block of code that starts:

    // alias the file to avoid file name clashes 

What this should do:

* Upload the file
* Rename the file to 'csv_' . timestamp . '.txt'
* Give message on rename failure: 'File failed to properly upload'

By renaming the file we avoid file name clashes, we store the original name in the database for display in forms, and store the new name for file operations accross multiple cron run imports.

I suspect this is failing at the stage where it uploads:

    $file = file_save_upload($file); 

What do you think?

karens’s picture

I think that may be complicating things but is not the real source of the problem. Since I am the one who has a file that will not upload, I will try to find a way to get it past the error somehow or other and let you know what changes needed to be made to make it work. It looks like everything is fine once the file is completely uploaded, so we just have to get it that far :-)

robert castelo’s picture

Thanks.

What happens if you manualy change the file name to .txt before selecting it for upload, does it work then? Is it the .csv extension which it chokes on?

karens’s picture

Yes, the .csv extension is the problem. Like I said, in the older version I could get past this problem by renaming the file, but for some reason that is not working now.

I am tearing my hair out -- what a goofy problem this is! The root of the problem is in file.inc in the file_check_upload function. This is where it has mime magic check the file type. My file with a .csv extension fails at that point so it renames the file with a .txt extension. This creates a file in my tmp directory that has a temp name with a .txt extension (not even the original file name). When it tries later to do anything with that file, it keeps checking for the original file name or the temp name without the .txt extension (neither of which exist in the temp directory) and sends back an error message. To make things worse, the file_check_upload gets run multiple times either in your module and in file.inc (file_save_upload has another call to file_check_upload). Each time it tries to execute it gets even more confused about the file name. Sometimes it is looking for the original file name, sometimes for the temp name, never for a name with the .txt extension.

I have tried taking out the code where you rename the file (just to keep that from muddying the water.) I turned off various checks and finally got to the place where I could get my .csv file with a csv extension uploaded and sitting in my temp directory, but by that time file.inc has gotten completely confused about the file name and still generates an error that reports that the file doesn't exist.

Still working on it, but I may just have to throw in the towel soon. I can't figure it out.

karens’s picture

I've spent most of the day on this and I'm throwing in the towel, but maybe you can do something with it from here:

1) Comment out the part of the program that renames the file and change it so $new_name is the same as $file->$filename. The renaming creates additional problems and you don't need it anyway since file.inc is already checking for existing file names and creating new ones if there is a clash. (It appends an incrementing digit to the end of the filename if a file with the same name already exists.)

2) Comment out everything in user_import_upload_validate. That one last pass through file_check_upload ends up wrecking things no matter what else I do. I tried every which way I could to leave that in, but it always ended up leaving me with a mis-named temporary file in my temp directory. The program already executed file_check_upload as a part of file_save_upload, so I think it is redundant anyway.

3) With those two changes I can get the file uploaded and sitting in the temp directory, but it has a .txt extension appended on to it. (i.e. myfile.csv has become myfile.csv.txt).

4) At that point, the module returns me back to the file upload screen, probably because it is looking for a file with the wrong name. I can't figure out where in the program to tell it to look for a renamed file as an alternative to the original name.

Hope you can make some sense out of this. I vote for making a change in file.inc (which is really where the problem lies) but I see there has already been lots of discussion, and lots of disagreement, about that.

robert castelo’s picture

Status: Fixed » Active

Many thanks for the time and effort you've put into working on this, much appreciated.

If it's any consolation, I also spent a week getting this updated for 4.7, a lot of the time bumping up against file API. Time well spent though as it also gave me a good understanding of forms API.

I'll put more time into updating user import, and fixing bugs, once Drupal 4.7 hits the release candidate stage.

nikle’s picture

Hey just curious if this is working reliably in 4.7.
Thanks!

robert castelo’s picture

If anyone is curious...

There is now a 4.7 (cvs) version available.

Uploading the csv user data file through the browser is still not working, but a new feature has been added which allows the use of a csv files which have been uploaded with FTP - this is actually much more useful for large imports were the file is too big for browser upload anyway.

The code is still in development, so use with caution, there may be bugs.

There will be a full (stable) 4.7 release in the next two weeks.

pkej’s picture

I tried browsing the module page (http://drupal.org/node/31940) and there the latest CVS version seems to be from last year. Could you provide a link to the cvs?

This is a very valuable module, I would need it in the very near future. Hint ;)

webchick’s picture

Assigned: robert castelo » Unassigned

Woah, that was weird... this showed up in the issue queue with all the information (Project, Version, etc.) cleared. Hrm.

robert castelo’s picture

Status: Active » Fixed

Fixed in User Import 4.7.x-1.0 version.

Anonymous’s picture

Status: Fixed » Closed (fixed)