I tried uploading a picture named "testæøåëéè.png" while having the imagemagick toolkit enabled. This fails:

    * ImageMagick reported an error: convert: unable to open image `/tmp/test.png': No such file or directory. convert: unable to open file `/tmp/test.png'. convert: missing an image filename `files/images/temp/test.thumbnail.png'.
    * Unable to create scaled Miniatyrbilete image
    * The selected file /home/www/vhosts/foetus could not be copied.

This does not happen when using GD2.

This is on Debian Linux / PHP 5.2.6 / MySQL 5.0.51a / Apache 2.2.9 / Drupal 6.8

Comments

sun’s picture

sun
Primary language German
Location Karlsruhe
commented
Status: Active » Closed (won't fix)

You might want to use http://drupal.org/project/transliteration to circumvent this issue.

hba’s picture

hba commented
Status: Closed (won't fix) » Active

But why would I want to use transliteration? The file system is UTF-8, apache runs with a UTF-8 locale, the file has a UTF-8 file name, and the drupal site is displayed in UTF-8. Also, this works great when selecting the GD2 toolkit instead.

sun’s picture

sun
Primary language German
Location Karlsruhe
commented
Status: Active » Closed (won't fix)

Hm. If GD2 works, this can only mean that your version (or configuration) of ImageMagick does not support UTF-8.

hba’s picture

hba commented
Status: Closed (won't fix) » Active

You are extremely quick to close this issue. This UTF-8 encoded script runs without problems on my server:

<?php
print system("/usr/bin/convert -verbose testæøåëéè.png testæøåëéè.jpg");   
?>
sun’s picture

sun
Primary language German
Location Karlsruhe
commented

Please have a look at _image_imagemagick_convert() and _image_imagemagick_convert_exec() in image.imagemagick.inc.

The source and target filenames are not altered, only certain chars are escaped for the shell:

    escapeshellarg($source),
    escapeshellarg($dest),

Afterwards, convert is invoked without further modifications:

  if ($h = proc_open($convert_path .' '. $command_args, $descriptors, $pipes)) {

Personally, I did not have any issues with UTF-8 characters and ImageMagick yet. I.e., we need you to debug this a bit on your own. If you find out what is causing the removal of UTF-8 characters, we can probably fix it.

hba’s picture

hba commented

I've discovered that escapeshellarg() is influenced by the current locale. In Debian, LANG is hard coded in /etc/init.d/apache2:

ENV="env -i LANG=C PATH=/usr/local/bin:/usr/bin:/bin"
...
APACHE2CTL="$ENV /usr/sbin/apache2ctl"

This makes escapeshellarg() strip away non-ascii characters. The characters aren't stripped away if I use a valid locale instead. The locale doesn't have to be Unicode, I can use both en_US.ISO-8859-1 and en_US.UTF-8.

Using setlocale before escapeshellarg() in image.imagemagick.inc also works fine. It's not sufficient to put it in my settings.php.

setlocale(LC_CTYPE, "en_US.ISO-8859-1");

putenv does not solve this, by the way:

putenv("LC_ALL=en_US.UTF-8"); //does not work


Maybe there should be a test somewhere to detect this? Something like:

if (escapeshellarg("☀") == "") {
  print "Something is rotten in the state of Denmark";
}
sun’s picture

sun
Primary language German
Location Karlsruhe
commented

So, it seems that the locale must be different to "C" and "POSIX", like with any other application that uses UTF-8.

However, we cannot force a certain locale. All we could do is to check the default locale on the image toolkit configuration page and spit out a warning if the locale is one of the aforementioned ones (using setlocale() without second argument). But then again, we can only warn the user, because the actual steps to fix this are different on platforms. Also, I do not know of a handbook page that explains this, and I could not find documentation on the system locale on imagemagick.org.

hba’s picture

hba commented

It's not an imagemagick problem, it's an escapeshellarg() problem, I guess that's why you didn't find any info on imagemagick.org. It would be great if we could show a warning to the admin when the toolkit is enabled, and steps on how to avoid the problem.

By the way, I've checked several Debian installations, the locale issue is a problem on all of them. The following gives me a "☀" on MacOSX and FreeBSD, but not on Ubuntu and Debian:

LC_ALL=C LANG=C php -r "setlocale(LC_ALL, 'C'); echo escapeshellarg(\"☀\") . \"\n\";"

So, the problem with PHP & escapeshellarg() summed up:

1) MacOSX/FreeBSD returns UTF-8 characters regardless of the locale
2) Linux/Ubuntu strips UTF-8 characters when the locale is C
3) Linux/Ubuntu starts Apache with a hard coded LANG=C, which triggers 2)

sun’s picture

sun
Primary language German
Location Karlsruhe
commented

Well, I would say it is a generic server configuration issue. I experienced the same issue with C/POSIX system locales when using Subversion.

So here is the deal:

If you create a handbook page on drupal.org that explains this issue and how to fix it (on Debian as example), and create a patch for Image that implements an error message on the image toolkit configuration page when the locale is C/POSIX, which links to that handbook page, I would commit that patch.

If you feel it is not worth the effort, then I am going to close this issue, since it is - overall - outside of the scope of Image module.

Either way, please let me know how you want to proceed (first).

hba’s picture

hba commented

I'll try to create a page and a patch.

sun’s picture

sun
Primary language German
Location Karlsruhe
commented

#287082: Imagemagick returns error code -1 on Debian has been marked as duplicate of this issue, might contain further information.

smitty’s picture

smitty commented

Hi hba!

Did you succeed in creating a page and a patch? I couldn't find them yet.

I'm not sure how you got this issue fixed. I tried to insert

setlocale(LC_CTYPE, "en_US.ISO-8859-1");

in line 126 of the image.imagemagick.inc. But that did not fix my problem (described in: http://drupal.org/node/287082).

So: What do I have to change?

lialimar’s picture

lialimar commented

Actually it's better to put

setlocale(LC_CTYPE, "en_US.UTF-8");

That works fine.
...
I had the similar problem: I could not use Russian names for additional image sizes.
Yes, the problem is escapeshellarg function. It strips out any UTF-8 symbols from source or destination filename.
By the way, according to PHP buglist that behavior is a bug in some versions.

May be the better way is to place
setlocale(LC_CTYPE, null);
if we can be sure that Apache has correct locale setting: en_US.UTF-8. But we cannot be sure. Imagine the situation where russian company uses hosting somewhere in US.

Another solution is to instruct user to use only ASCII chars in image size names and uploaded filenames. But it is ugly solution.

And the last possible solution is to avoid using escapeshellarg function. Yes, there is a need for some escape-function. May be another.

sun’s picture

sun
Primary language German
Location Karlsruhe
commented
Category: bug » support
Status: Active » Fixed

Marking this as fixed now.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

OnkelTem’s picture

OnkelTem commented

So how/when it was fixed?