Special characters encoded incorrectly

I've encountered a similar problem with greek pdf files, which were incorrectly assumed to be in the default system encoding (ISO-something) and were erroneously converted to utf-8 in function search_files_convert_to_utf8 of file search_files.module. The input data to the function were otained by a previous call to pdftotext via shell_exec. The problem was solved by replacing the line

$encoding = iconv_get_encoding("output_encoding");

by

$encoding = mb_detect_encoding($text);

How can I solved this issue?

I'm building a website that is not in english. The characters changed to "?" anytime I save/update. It is really frustrating. Please help me. If the above patch can solve the issue, could you please explain to me on how I can solve the issue. Thank you.

Voting for #3 patch, it solved my issue with wrong encoding, without breaking anything for the files that already were parsed correctly.
The reason I like this change is module detecting correct encoding even if you change helper's output with command-line switches.

Tested and working, please commit

Hi,

Doesn't work for me (french), with 6.x-2.x-dev and patch applied

"systèmes" appears as "systymes"
"activité" appears as "activity

Please, try to run corresponding helper in command line and check if the helper understands the encoding of documents. Can be helper problem as well.
Which document format do you use and what helper for parsing? What command line do you use for that helper in Search Files helpers settings?

Thanks for your answer

paths are :

/usr/bin/catppt
/usr/bin/xls2csv
/usr/bin/catdoc
/bin/cat

command line gives error on special caracters

in file1.doc : "Rédaction" displayed as "Rйdaction"
in file2.ppt : "Modèle" displayed as "Modèle"
in file3.xls : "Congés" displayed as "Congйs"
file4.txt is OK : système, activité

I've installed the helpers with yum, from RPMforge for CentOS 5, and search files settings autodetect

should I add charset options in the path ? I tried in command line but without success. I don't know exactly how to write it.

should I check something about charset in the /bin directory ?

So, the first thing we can conclude is - the problem is in helpers incorrectly passing data to Drupal. TXT helper (cat) - I wonder what format it uses to output the data? Check the default regional settings on the server running Drupal, as they can influence the output of those helpers.
There are several command line switches that can alert helpers about special encoding of the input file or particular output encoding required, running helper with -? or -help to get possible options is a good idea. For example, catdoc understands options like -d utf8 and I am using this option myself to keep Cyrillic characters correctly displaying.

Thank you. I tried -d utf8, -d UTF-8 etc... without success (but no error message) . I can't see any charset file on the server. Where could they be ?

catdoc.exe -w -d utf-8 %file% is what works for me, but I use Windows server to host Drupal site.

Thanks. Does'nt work either.

Hi,
it's ok with /usr/bin/catdoc -d cp1251 it displays accents with command line : "système" "expérience"

Then I changed helper path (admin/search files/setting) : /usr/bin/catdoc -d cp1251

1. I checked with a new file : the word "équipe" is found by the search "équipe", but is displayed without accent : "equipe". This is not perfect but will do. (core search displays "équipe", but not within file of course)

2. How to delete index for previous indexed files, witch remain with wrong characters ?

admin/settings/search - reindex site.

But... fact that you don't know that (and that cp1251 is for cyrillic alphabet) suggests me that your setup is still far from perfect. That means you have to open new support issue and ask new question instead of misleading maintainers that there is possibly something wrong with this particular patch.

Anyways, you need to change you encoding to latin1 (or it is even possible you dont have to change encoding at all, just apply patch and clear search index because latin1 is default anyway). Also you may check if you have latest versions of helper apps.

oh oh, sorry if I've misused this issue system and misleaded maintainers.
I see that I cannot delete my previous posts. If someone can, feel free to delete.

Just a notice to everyone - when you use this patch or just install the new version with the patch included - you have to reindex your site to make the patch work!

solved for me (french) using catdoc -s 8859-1 -d 8859-1 %file%

Just to remind you - you have to re-index (not just simply index, but reindex from the beginning) your site again after changing command lines, because the results of previous (incorrectly encoded) indexing stay in DB until next re-indexing occurs.

i am in the same situation.
very strange, if i do by console in my computer
pdftotext -enc UTF8 110-111.pdf 110-111.txt -nopgbrk
it is perfect
if i do in my local server, in the same computer, with drupal with utf8 and the database utf8, firefox computer, it gets wrong
/usr/bin/pdftotext -nopgbrk %file% -
like "direcciã n"<-dirección
and if i do it
/usr/bin/pdftotext -nopgbrk -enc Latin1 %file% -
then it get right... 0.0
lets see what the server wants...

to know which encodings you have, in your console
pdftotext -listenc

i didnt try the patch... maybe i have to do it when the server... today i am tooooo tired

anyway, thanks a lot for this module :)

now i am with the doc and docx documents.
docx are not eaten by catdoc and i am looking for a helper.
does anybody know another one? :)

also, i am worried because the different users can use different charsets...
could it be possible to integrate with this module...?
http://drupal.org/project/mimedetect

sorry that i dont know enough php to do it... :(
but if i found something interesting about this, i tell here.

edit i found unoconv and installed, but when i run autodetect, i doesnt take it.
by console its very good an takes the charsets perfect.
any body has tried? :)

@patrinos you are great! last night i was too tired to try your patch. now that i have tried, works!!!! :)
thanks!

applied the patch in #3 to 6.x-2.x and 7.x-2.x branches

thanks very much @jrglasgow for your generous work :)

just in case someone has an apache2 server and catdoc.
apache by default has charset 8859-1
drupal utf-8

in console it will work but in the server (local o real) not.
it was clear that it was something in the server that was changing the charset in the middle.

i try with catdoc tons of comands, change httpd.conf with "AddDefaultCharset utf-8", at .htaccess, and anything i found in internet... of course i restarted all the time the apache server...

until i put the comand /usr/bin/catdoc -d utf-8 -s 8859-1 -w -f tex %file% as apache been the source of the document... and at the end it works :) i have accents in the database!!!

@candelas: You are very welcome. I'm glad I was able to help.