drush eval and php -r do not proceed from consistent encoding settings

Searching the issue queue seems to be broken on drupal.org right now (?), but I would recommend searching for iconv in the Drush queue. There may be an older issue that could offer some insight as to what Drush is doing vis-a-vis character encodings, and why.

Searching working again -- no issues with 'iconv', but I think there was an old issue that might be interesting, if you can find it.

The issue I was thinking of was #797700: Convert drush output to terminal console code page, but not sure that explains the output shown in #0 above.

Trying to clarify issues from database side below....

My apologies if I am missing the point.

Note that there are several layers of character set and collation encoding in mysql.

server default
schema default
table definition
connection
client settings

Try this and note that each can be configured statically and dynamically, as per version of mysqld:

mysql> show variables like '%char%';
+--------------------------+--------------------------------------------------+
| Variable_name | Value |
+--------------------------+--------------------------------------------------+
| character_set_client | latin1 |
| character_set_connection | latin1 |
| character_set_database | latin1 |
| character_set_filesystem | binary |
| character_set_results | latin1 |
| character_set_server | latin1 |
| character_set_system | utf8 |
| character_sets_dir | /Applications/MAMP/Library/share/mysql/charsets/ |
+--------------------------+--------------------------------------------------+
8 rows in set (0.00 sec)

mysql> show variables like '%collat%';
+----------------------+-------------------+
| Variable_name | Value |
+----------------------+-------------------+
| collation_connection | latin1_swedish_ci |
| collation_database | latin1_swedish_ci |
| collation_server | latin1_swedish_ci |
+----------------------+-------------------+
3 rows in set (0.00 sec)

If you do it outside Drupal (hence without Drupal bootstrap) you get correct output:

~ $ php -r 'echo iconv("utf8", "ASCII//TRANSLIT", " héllô") . "\n";'
 hello
~ $ drush ev 'echo iconv("utf8", "ASCII//TRANSLIT", " héllô") . "\n";'
 hello
~ $ cd /var/www/drupal7
/var/www/drupal7 $ php -r 'echo iconv("utf8", "ASCII//TRANSLIT", " héllô") . "\n";'
 hello
/var/www/drupal7 $ drush ev 'echo iconv("utf8", "ASCII//TRANSLIT", " héllô") . "\n";'
 h?ll?

Will check what in boostrap makes it behave this way.

I predict that whatever it is that is affecting encoding will be found in the Drupal bootstrap code. Best of luck in your isolation efforts.

Drupal should not respect the locale settings of the server. Especially, the very important part is LC_NUMERIC=C because otherwise PHP formats floats in a locale-dependent fashion.

Respecting the locale settings of the shell might be in the scope of Drush, but it's not in the scope of Drupal.

If Drupal does a setlocale(LC_ALL, 'C');, it seems that it would be undesirable for Drush to undo that, unless, perhaps it could do a save/set/restore locale setting around actual output in drush_print & c.

This returns us to #0, which shows the symptoms but not the use case. The data that is being "incorrectly" transliterated was provided from the cli in the encoding expected for the system. This is different than data coming from the Drupal database being incorrectly interpreted. Perhaps you need to iconv to UTF8 your data before you pass it to Drush. Alternately, perhaps Drush needs to iconv the cli parameters prior to bootstrapping Drupal. And what about drushrc.php option values; do they need to be transliterated as well?

Edit: Input data is indeed UTF8; the problem here is not in the encoding of the input data, but the way that setting the locale to 'C' affects the output transliteration of iconv.

Yes, you may very well be right, but there is nothing Drush can do about the locale setting after the Drupal bootstrap. If you have code in your drush module that needs to operate under a particular locale, you could call setlocale yourself immediately prior to the i18n calls; however, you'd need to set it back again when you were done, or you'd risk breaking Drupal's expected behavior.

I think we have no choice but to close this issue as 'works as designed'.

Drush will attempt to find a Drupal site at the cwd, so whether or not Drupal is bootstrapped can indeed depend on where drush is executed from.

style