Hi,

I opened a support issue a few months back and had to step aside from my Drupal project for a while. I'm back working on it and I'm still having trouble with   encodings on my pages.

I've used Import_HTML to import my files (excellent module btw) and most of it works fine. However, I have   code in various places on my site and each non-breaking space is converted to a question mark.

Here's what my config file has:

modules/import_html/coders_php_library/xhtml_tidy.conf

tidy-mark: no
tab-size: 2

indent: no

wrap: 0
markup: yes
; Actually, we are only using this for syntax validation. Please leave the incid
ental newlines alone.
output-xml: no
output-xhtml: yes
doctype: omit
; doctype confuses (breaks) IE6 in subtle ways AKA 'quirks mode'
input-xml: no
show-warnings: no
numeric-entities: yes
quote-marks: no
quote-nbsp: yes 
quote-ampersand: no
break-before-br: no
; break-before-br causes conflict with Drupal newline filter, giving us 3 lines 
where I want 1
uppercase-tags: no
uppercase-attributes: no 

; strip font definitions and crap
clean: yes
drop-font-tags: yes

; beware, for XHTML, we usually want UTF8, not the us-ascii default
char-encoding: utf8

enclose-text: no

quiet: yes
show-warnings: no

Any ideas why this would be happening? I've tried changing some configuration options like quote-nbsp and numeric-entities just to see what they would do and haven't noticed a difference. This is the proper config file to change right?

Comments

rgraves’s picture

rgraves commented

I think I fixed it. In the xhtml_tidy.conf file, I changed char-encoding from utf8 to ascii. I don't know much about character encodings but I think it will be alright since we don't use many special characters.

Does anyone see a problem with this setup? I know there is a beware message in the conf file about doing it this way, but I've tried a dozen or so pages and it seems to be working. This site is quite large though so I want to ensure this isn't going to bite me later.

Rob

dman’s picture

dman commented

Thanks for the follow-up.
I am aware that there can be some inconsistancies with character encodings, but most of my previous efforts have solved one problem only to open up another. I still have no true answer, as inputs come in all flavours and quirks.

- Certain versions of the XML-expat library behaved differently
- Certain versions of the tidy library either did or didn't convert named entities into numeric entities
- I have no idea what the correct behaviour should be when converting from internationalized character sets (UTF16 etc) into UTF8 if the source file wasn't correctly tagged
- Spaces, nbsp, ampersands and quotes sometimes need to be encoded, but sometimes are already encoded. Sometimes mixed in the same document! Sometimes I found my import process or the XSL transform encoded them twice. I've tried to patch a few known cases with regexp replacements just before parsing, but that work-around is brittle.

I don't see any actual problem with UTF8 vs ASCII - That will make sense to you. Glad it works, I think the difference between them is only a few non-printing characters, if that. IIRC, it was some edge cases with other languages I couldn't trust with that. Plus warnings from the XML parser. Newer XML libraries may cope better.

.dan.

dman’s picture

dman commented
Status: Active » Closed (fixed)

Cleaning up issue queue by closing stuff from the Drupal-5 branch and over a year old.