Hello!
I am again with encoding problem. As i write in http://drupal.org/node/117505, there are feeds with another coding page then utf8.

I use your patch, but... one feed work with it, another - no.

I think that need to do "use encoding" as parameter for leech-feed.

I make a few step on this direction, but i am a novice in PHP.
This steps:
1. Add field "useencoding" (tinyint -0) in table "leech"
2. Add form-field (checkbox) in function leech_form_alter() (node->leech->useencoding)
3. Dont know how - in leech_prepare() add element from form to $connection = array('useEncoding' => $node->leech->useencoding); ???
4. Store parameter $node->leech->useencoding in leech_insert(), leech_update()
5. use $connection=>useencoding in leech_connection() for pass it in leech_http_request(,,,,$connection->useEncoding)
6. And at the end, in leech_http_request() use as additional condition in

    if ($useencoding and function_exists("mb_check_encoding")) {
      preg_match("/encoding=['\"]([a-z0-9-\/]*)['\"]/", $result->data, $matches);
      if (!empty($enc_from) && mb_check_encoding($result->data, $enc_from)) {
       $result->data = mb_convert_encoding($result->data, "UTF-8", $enc_from);
      }
   }

In this case any exotic (mixed) feed will be reworked.

Please help me to solve step 3.

Comments

marafa’s picture

marafa commented

hi
i dont know if my bug is related to this or not but it is an encoding problem:
i get this output:

warning: mb_check_encoding() [function.mb-check-encoding]: Invalid encoding "windows-1256" in /home/marafa/public_html/drupal/modules/leech/leech.module on line 2134.

when using:

Source URL http://newsrss.bbc.co.uk/rss/arabic/news/rss.xml
although it does create the nodes with no problems

aron novak’s picture

aron novak
Primary language Hungarian
Location Hungary, Budapest
commented

marafa: It's pretty strange that PHP generates a warning when the module tries to check if the string is a valid encoding name or not. Workaround: http://php.net/manual/en/ref.errorfunc.php#ini.error-reporting
Set the error_reporting level to E_ERROR.
toologic: I don't think it's a good idea to add per-feed encoding setting.
Can you give me specific feeds where leech don't work again? It would be better to find an ultimate solution for all encoding problems rather than per-feed.

toologic’s picture

toologic commented

Aron!

please test these two feeds

_http://votpusk.ru/news.xml and _http://www.destinations.ru/news/rss/

Both feeds have declare encoding as windows-1251, but first convert with
mb_convert_encoding($result->data, "UTF-8", $enc_from); - correct.
Second - or return unreadable text, or do nothing.

alex_b’s picture

alex_b commented

I am with Aron here: we shouldn't add options for encoding. There must be a way how to detect and adjust encoding automatically.

aron novak’s picture

aron novak
Primary language Hungarian
Location Hungary, Budapest
commented
Assigned: Unassigned » aron novak

I tested this feed: _http://www.destinations.ru/news/rss/
I can reproduce the problem.
I find out that the additional charset detection works as i want. I added a print_r($data); statement after the charset conversion and the output is really in UTF-8.
I don't know exactly where it became wrong.
I tried out the feed at a separate leech environment w/ PHP5 and MySQL 5 and it works perfectly. I'm afraid of that this is a PHP-related issue.

arsart’s picture

arsart commented

Hi, I have a same issue with encodings too but on official Leech version 1.6, leech doesn't load non-utf feed items at all and post no errors on that. Also I'm using PHP5.

arsart’s picture

arsart commented

I've updated Leech to the last CVS version and mentioned that well-formed feeds in win1251 leeched but the worse thing happens - feed items automatically have been converted from cp1251 to utf8 and nodes became fully unreadable and broken even with manual change of browser encodings.
For example a double converted feed item text: Острые новинки,

So let's say that well-formed win1251 feeds a working but broke encoding on-a-fly and bad-formed feeds are unable to leech data at once.

arsart’s picture

arsart commented

Aron is absolutely right, so we have an input in cp1251 and output in utf-8, items became decoded and broke.
Btw, seems core module - aggregator does the job well with win1251...

arsart’s picture

arsart commented
Category: feature » bug
Priority: Normal » Critical

Hello all,

Let's say that's a humble reminder that encoding issue is still here. And please don't rid off this from the bug list cos it's a real issue having Leech running without a dozen of non-utf8 feeds aggregation.

alex_b’s picture

alex_b commented

arsart,

please understand that everybody involved in developing leech has limited bandwidth. there are a couple of issues on the tracker that we can't give enough attention simply because the day has only 24 hours.

leech is open source. especially with particular issues like this one i would love to see personal initiative - patches or specific suggestions on how to address a bug.

neither aron nor I won't have the time to look into this issue at least in the next week - if you need to get this fixed earlier, the best way would be to figure out how to bring on another developer that could provide a patch.

alex

arsart’s picture

arsart commented

Sorry Alex_b if I confuse you. Yes, of course I realize that here is not only a place to ask but the place to suggest also. I'll try to get some hands on that cos as for me I'm non-programmer in mind but keen to solve this issue.
The question is can we use any other engines to get feeds parsed?

arsart’s picture

arsart commented

Just looked at SimplePie engine that used by Simplefeed module
http://cvs.drupal.org/viewcvs/drupal/contributions/modules/simplefeed/si...
found they encode all famous non-utf encodings on-a-fly near strings "function change_encoding" and "function encoding" with Iconv or mb_string php modules.

arsart’s picture

arsart commented

Chatted to russian drupal developers and they mentioned a piece of code that should be removed to pass non-utf feed be parsed.
So, at last CVS version you have removed that code and all non-utf feeds parse well now!

Muchisamas gracias amigos!

http://designcentre.ru

aron novak’s picture

aron novak
Primary language Hungarian
Location Hungary, Budapest
commented
Status: Active » Fixed

Hi arsart!

Thanks for your response that now everything is okay w/ the non-utf feeds!

Aron

Anonymous’s picture

(not verified) commented
Status: Fixed » Closed (fixed)