I have recently added several hundred users to a new site, using your fantastic module. However, I have noticed that those users with french characters in a field have that field truncated just before that character. The CSV files are saved from Excel, and if I open them with a text editor, the characters are still displayed correctly. Any ideas? I could edit each incorrect entry, but at almost 600 users, that might take a while :)

CommentFileSizeAuthor
#16 user_import.module.txt55.23 KBavior

Comments

Zack Rosen’s picture

Zack Rosen commented

This is because User Import relies on the PHP function fgetcsv which is not Unicode compliant. I've hacked user import with a custom CSV import function and it seems to work. Will clean up and post in a couple weeks.

davemybes’s picture

davemybes commented

Great news. Thanks.

Fidelis’s picture

Fidelis commented

Have the same problem - look forward to see the issue resolved :-) Many thks for this great module!

s_oneill’s picture

s_oneill commented

I suspect my problem may be related to the same issue - when I import fields which containt a danish character like æ ø or å ... it gets replaced by a space.

The cvs file is utf8 and so is MySQL.

Looking forward to a solution as this module is an absolute LIFE SAVER !!!

liquidcms’s picture

liquidcms commented

using the node import module to import french data from a excel/csv and i get the same issue.. was wondering if there was a solution here yet??

tenshu’s picture

tenshu commented

do you plan to commit this?

alippai’s picture

alippai commented
Priority: Normal » Critical

It breaks its functionality so it's critical.

alippai’s picture

alippai commented
Title: fields with accented french characters get truncated » UTF-8 character set support
syngi’s picture

syngi commented

Issue seems to be a duplicate of http://drupal.org/node/190437

nickvidal’s picture

nickvidal commented

To solve this issue:

1) Edit user_import.module

a) Find this function call in the code:

fgetcsv

b) Add the following line before the function call:

setlocale(LC_ALL, 'en_US.utf-8');

c) Save user_import.module

2) Be sure that your import file is saved in UTF-8

a) In gvim type this command:

:setlocal fenc=utf-8

Best of luck,
Nick Vidal

Anonymous’s picture

Anonymous (not verified) commented

I don't know if this is of any value, but, just for a test, I hacked user_import by simply setting the value of $data[1] (the First Name in my case) like that :

$data[1] = "Étiène";

and it still doesn't work : when we list the users, we see " ti ne" instead of "Étiène". The string "Étiène" which fails to display correctly has nothing to do with fgetcsv(). So this problem is not with fgetcsv(). Of course, the problem is perhaos created because of the way I assign the value to $data[1], and it may have nothing to do with the original problem discussed here. However, I also tried to change the encoding before assigning the string to $data[1], and nothing seems to work.

Anonymous’s picture

Anonymous (not verified) commented

Well, I don't know what I did, but I cannot reproduce the above behavior anymore.

If I set $data[1] with

$data[1] = utf8_encode ( "Étiène" ) ;

it works as expected. It was a false alert.

However, I still had the problem with the user name, which I think is the original problem discussed here, but this is only because of the line

$username = preg_replace('/[^a-zA-Z0-9@ ]/', ' ', $username);

in user_import.module, which replaces any thing that is not a latin-1 letters or numbers (including all accentued characters) with a space. I solved this problem by replacing this line with

mb_ereg_replace ( '/[^\pL\pN@ ]/' , ' ' , $username );

which does the samething, except that it accepts the UTF-8 (upper and lower case) letters and numbers not only the Latin-1 letters and numbers.

robert castelo’s picture

robert castelo commented
Status: Active » Fixed

Fixed in 5.x-2.0-beta1.

I've improved the way that usernames are sanitised.

Not sure if that solves all the UTF-8 problems, feel free to open this issue again if you're still experiencing problems with character set support.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for two weeks with no activity.

avior’s picture

avior commented

this issue is still open in the 6.2.4 version
I have used the code above and this solved my problem
attached the patched module version

sorry, I can not attach the module , i get validation error

avior’s picture

avior commented
StatusFileSize
new user_import.module.txt55.23 KB

2nd try

robert castelo’s picture

robert castelo commented
Status: Closed (fixed) » Needs review
perandre’s picture

perandre commented
Status: Needs review » Needs work

I tested the attached module. It creates a user and a profile properly, but it cuts wherever the (in my case) Norwegian character show up. E.g: John Tromsø -> John Troms

perandre’s picture

perandre commented
Status: Needs work » Needs review

My bad; I accidently used a csv with Western char set. After converting the file to UTF-8, it worked without the patch, actually.

robert castelo’s picture

robert castelo commented
Status: Needs review » Closed (works as designed)