By robertdouglass on

When mailhandler receives certiain mail messages during cron, certain queries (INSERT) against the PostgreSql database (version 8) go wrong due to UNICODE incompatibilities. I'll be honest, I'm not so swift when it comes to understanding encoding issues. Has anyone experienced this problem? Any suggestions for going about debugging?

Here are the error messages:

pg_query(): Query failed: ERROR: invalid byte sequence for encoding "UNICODE": 0xe96561 in /usr/local/apache/htdocs/includes/database.pgsql.inc on line 45.



warning: pg_query(): Query failed: ERROR:  Unicode characters greater than or equal to 0x10000 are not supported in /usr/local/apache/htdocs/includes/database.pgsql.inc on line 45.



warning: pg_query(): Query failed: ERROR:  invalid byte sequence for encoding "UNICODE": 0xe92048 in /usr/local/apache/htdocs/includes/database.pgsql.inc on line 45.

Here are the headers from a typical mail that makes it crash:

From angelathomas@ispamalot.br  Tue Apr  5 14:53:17 2005
Return-Path: <angelathomas@ispamalot.com.br>
Received: from www.ispamalot.com.br (smtp.ispamalot.com.br [200.221.11.147])
	by mail.mywebsite.com (8.12.11/8.12.11) with ESMTP id j35IrEVW013688
	for <news@mywebsite.com>; Tue, 5 Apr 2005 14:53:14 -0400
Received: from [64.110.93.76] by www.ispamalot.com.br with HTTP; Tue, 5 Apr 2005 15:34:26 -0300
Message-ID: <4252458600002D27@www.ispamalot.com.br>
Date: Tue, 5 Apr 2005 19:34:26 +0100
From: angelathomas@ispamalot.com.br
Subject: {Spam?} =?iso-8859-1?Q?I=20=20NEED=20=20YOUR=20=20URGENT=20=20ASSISTANCE=20=20PLS=2E?=
To: angelathomas@ispamalot.com.br
MIME-Version: 1.0
Content-Type: text/plain; charset="iso-8859-1"
X-yoursite-MailScanner-Information: Please contact the ISP for more information
X-yoursite-MailScanner: Found to be clean
X-yoursite-MailScanner-SpamCheck: spam, SBL+XBL
X-MailScanner-From: angelathomas@ispamalot.com.br
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by mail.mywebsite.com id j35IrEVW013688
Status: O
X-UID: 305
Content-Length: 2986
X-Keywords:                                                                                                    

Dearest One,

REQUEST TO BE MY GUARDIAN AND THEN HELP ME TO COME OVER TO YOUR 
COUNTRY:
Good day and how are you today? I hope fine? I  got your contact from internet,permit
me to inform you of my desire to ask you to be a guardian or foster parent
to me and then help me out in what I'm about to tell you.I'm Angela  Thomas(19)
i,m the only daughter of Late Mr.& Mrs.Joseph Thomas. My father 
was a very wealthy Oil Marchant here in Abuja, the capital of Nigeria. He
was poisoned to death by his business associates on one of their outings
on a business trip. My mother died when i was a baby.

Comments

robertdouglass’s picture

robertdouglass commented

Somehow the following mail snippet:

Dr. André Houphouët
Président du Comité d'
Attribution des Marchés et Contrats.
Corporation Nationale Pétrolière de CI.
C.N.P.C.I 
Téléphone. +22 507 159 235.

is turning into this by the time it gets to the query:

query: INSERT INTO watchdog (uid, type, message, link, location, hostname, timestamp, server_name) VALUES (13, 'error', ' query: INSERT INTO watchdog (uid, type, message, link, location, hostname, timestamp, server_name) VALUES (13, ''user_otf'', ''In moblog_mailhandler(); message body = Dr. Andr須ouphou봍 Pr鳩dent du Comit頤'''' Attribution des March鳠et Contrats. Corporation Nationale P鴲oli貥 de CI. C.N.P.C.I T鬩phone. +22 507 159 235. Bonjour. Je suis averti que ma lettre vous apparaa comme une grande surprise, parce que nous ne nous sommes pas rencontr鳠avant, mais surtout ne pensez pas que mon courrier ࠶ous est l''''un de ces courriers publicitaires non sollicites que vous aviez d?evoir ࠶otre adresse email, votre poste ou machine de fax, comme j''''en re篩s 駡lement. Mon nom est Andr須ouphou봬 de la R鰵blique de C?d''''Ivoire et aussi le Pr鳩dent du Comit頤''''Attribution des March鳠et Contrats ?la Corporation Nationale P鴲oli貥 de C?d''''Ivoire. J''''ai 鴩 dans cette position durant les dix derni貥s ann饳, mais je n''''ai rien qui puisse montrer cela. Il serait d''''un 鮯 in /usr/local/apache/htdocs/includes/database.pgsql.inc on line 64.', '', '/runcron.php?include=mailhandler', '80.133.98.103', 1112904523, 'devupload.nowpublic.com')
error:ERROR: invalid byte sequence for encoding "UNICODE": 0xe92048

- Robert Douglass

-----
www.robshouse.net
www.webs4.com

killes@www.drop.org’s picture

killes@www.drop.org commented

I guess part of the problem is that mailhandler does not convert the iso-8859-1 encoding that the mail has to utf-8.
--
If you have troubles with a particular contrib project, please consider filing a support request. Thanks. And, by the way, Drupal 4.6 will support PHP 5.

--
Drupal services
My Drupal services

robertdouglass’s picture

robertdouglass commented

mb_convert_encoding($str, "UTF-8", "iso-8859-1");

?

- Robert Douglass

-----
www.robshouse.net
www.webs4.com

Steven’s picture

Steven commented

We now have a function for this in core (drupal_convert_to_utf8).

--
If you have a problem, please search before posting a question.

cyberchucktx’s picture

cyberchucktx commented

All:

A caveat about drupal_convert_to_utf8: it calls/requires the "iconv" library with PHP4 (iconv is built in to PHP5, but "iconv" is a compile option with PHP4).

My ISP hasn't got iconv ... and won't, they say, until CPANEL doesn't break with it (still don't QUITE understand why they say this is related).

I ended up ripping (commenting) out all calls to "drupal_convert_to_utf8" in the mailhandler module.

I saw the bit about "iconv" in the logs ... thank goodness the watchdog and log stuff is working pretty well.

Charlie

killes@www.drop.org’s picture

killes@www.drop.org commented

You could also have recode or the mb_string extension.
--
Drupal services
My Drupal services

--
Drupal services
My Drupal services

killes@www.drop.org’s picture

killes@www.drop.org commented

You could try that. You shoul dof course check that the encoding is actually iso-8859-1 (maybe that fucntion does that). For Drupal 4.6 there will be the drupal_convert_to_utf8() function and a patch to mailhandler would be welcome..
--
If you have troubles with a particular contrib project, please consider filing a support request. Thanks. And, by the way, Drupal 4.6 will support PHP 5.

--
Drupal services
My Drupal services