Internationalization support for PDF

I'm not sure what fonts PDFView needs, you might find some useful things on this page.

You might also find Gentium interesting, which is a very comprehensive Latin-alphabet font (including extensions).

By the way, don't confuse UTF-8 (an encoding) with Unicode (the character set). Searching for UTF-8 fonts won't get you much, searching for Unicode fonts will.

I have just installed the pdfview plugin, patched - uses FPDF.

Basically, it produces a too simple PDF.

I have tried making the PDF out of a Drupal node in Croatian language, and all accent characters are displayed as garbage.

FPDF by default supports several encodings, but not UTF-8. Let's say it can be done fiddling with FPDF's makefont functions, but surely - for a system that uses UTF-8 like Drupal, UTF-8 encoding for FPDF should be included with the module install archive.

Additionally, when fonts are included into PDFs they need to have a proper embedding policy. So font can be embedded in subset (only characters in use, good for online PDFs) and embedded full (for imagesetting and press printing - not necessary here), but should have embedding info present. So PDFs generated thus from Drupal should have Embedded Subset policy set, which actually embeds used characters of a font into PDF. Otherwise, if we use Helvetica with Unicode support in PDF but without embedding, when printing the font will be substituted with the printing system's Helvetica and special chars may again be garbaged.

The testing node itself included a few HTML tags: -br- and -b- and -i- tags, and an occasional -img src=...- tag. All -br- tags are OK during PDF generation, but if the author typed ENTER key for new line there are no spaces between paragraphs as were in node. Only -b- tags were interpreted. -ul- tag doesn't indent a text, it produces a space between paragraphs in PDF.

Instead of just a page number in the PDF's footer, it would be nice to have at least the originating site's name and URL.

However, the best solution by far would be the ability of choosing a template for a PDF file - so we could make a template for PDF docs which would be filled on-the-fly.

Now what happens if authors use bbCode tags? They are echoed as they were input in page's source - no formatting.

FPDF is very powerful, and it can be used to produce really great stuff.

I am involved in development of French PDF-related site http://www.abracadabrapdf.net, authored by Adobe Certified Expert Jean-Renaud Boulay, and we are planning on putting Drupal behind it and to expand French language content to French, English and at later stages Croatian. We could REALLY use this module to present users with articles and tutorials in PDF format, generated on-the-fly.

But, for that we need fully rich HTML code support, and template abilities as a must. When the image abilities are inserted, it would be good for all GIF and PNG images to be resampled to JPEG in PDF document.

We could solve template needs fiddling with FPDF, but I am sure many Drupal users interested in PDF output would like to have the ability of templating the PDF in advance.

In general, I believe this is a great idea, and a great thing you initiated here, but needs a LOT of work before it gets actually usable.

Oh yes, we would use it - extensively, and then some ;)

This would serve as a great extension to abracadabraPDF as a brand, and as a great extension to Drupal. When I imagine all our articles generated as a PDF on the fly... yummie ;)

Concerning templating - I believe it would be easier to implement a pure HTML template in for ex. subfolder pdfview/template than a FPDF's template because template editing needs to be available and easy for all Drupal's users.

Colours - it may be a good idea to implement a colour-conversion function that would replace all HEX colour values from the HTML output into RGB, so PDF thus created can have a proper, standardized colourspace: RGB tag. That's also one of the reasons all GIFs and PNGs would be better off converted to JPEGs with appropriate compression settings.

Font issue - FPDF supports TrueType and Type1 fonts, and there is even an AddFont script that makes makefont.php manipulation easier? I believe a cross-paltform solution driven through Unicode support can be implemented here, enabling all those accent characters in various languages to be displayed and printed properly.

Concerning php PDF libraries, information I have tell me FPDF is the best of the free libs. Have you checked contributed scripts on their site?

Timeframe - I am struggling with my first Drupal site at the moment which I also plan to enrich with PDF output at later stages (Drupal is a hard nut, I must say but very good), and after that we are going to start developing the new abracadabraPDF under Drupal. Since it is going to be a big chunk of work required, the old site will keep functioning but the new site will be online on another testing location. Our timeframe is when done properly, it is not strict, and we can implement PDFview features when they are finished ;) Mr. Boulay was delighted on the possibility of PDF creation, especially through FPDF.

Furthermore, I believe I can speak on my own and on Mr. Boulay's behalf to say - whatever you need, we can help. Additionally, I think Mr. Boulay had contacts the guys from FPDF.org so, if needed, we can work potential problems out together or get advices from them if necessary. Especially we can help from the PDF side of the story.

I say again - BIG THUMB UP for PDFView! :)

I have a major problem with FPDF: I posted a request on their forum for recommended Unicode fonts. The reply that came back: "you know - that unicode is not supported by FPDF?". Well no, I didn't know. And since Drupal is wholly based on Unicode (quite right too!), it seems to me impossible to make FPDF work properly for any language other than plain US English - unless the coding in pdfview is able to determine what the language is and convert from Unicode to a single-byte coding. But can it do this?

If you only use one character set, you can convert from Unicode/UTF-8 into it before sending it off to PDFVIEW. For the Latin-1 character set, you use PHP's utf8decode(); to convert from UTF-8 to ISO-8859-1. Otherwise you'll need to use iconv, librecode or mbstring extensions for PHP.

I'd actally prefer to do no conversion and do utf-8 PDFs right out of the database. i have spend some time searching but din't find an obvious solution. Maybe we could modify the fpdf sources.

Well, according to FPDF, PDF doesn't support that encoding.

See

http://www.fpdf.org/phorum/read.php?f=1&i=16111&t=16111

However ;) a little hack could be performed, where special characters would be remapped. I am really not sure how it would be done on the PHP side, but basically the following would have to be performed:

Me, for example. I am using UTF-8 to display Croatian accented characters in Drupal. For example č, ć etc. Half of these characters lie in the same positions within the font map as in 8859-2 ISO codepage. But, the rest lies within the so-called extended Unicode range. So, for Croatian language, characters coming from Drupal's UTF-8 would have to be remapped to ISO-8859-2 codepage positions, prior to FPDF handling, and then the FPDF would actually use ISO-8859-2 prepared font to be able to display characters in PDF.

So when FPDF runs, it would need to set the font where the desired characters would already be remapped.

However, I believe this could be altered for such a 'hack'. Please see makefont.php and font map files ie. iso-8859-2.map etc. in FPDF distribution. Drupal's output would have to be adjusted prior to FPDF handling ie. character positions remapped to respond to a proper *.map file of FPDF. So, a PDFView module would have to have in it's module settings page an option like: PDF Remapping CodePage where the user would select the apropriate ISO codepage and the module would remap character positions according to this. Perhaps I could try and provide assistance with this, and with some PDF testings since I have done a lot of it in the prepress.

So basically, the following codepages are available, and I believe this would cover almost the complete UTF-8 character space: cp874, 1250, 1251, 1252, 1253, 1254, 1255, 1257, 1258, 8859-1, 8859-2, 8859-4, 8859-5, 8859-7, 8859-9, 8859-11, 8859-15, 8859-16, koi8-r, koi8-u. UTF-8 output has to be remapped to one of those inputs prior to FPDF. Most of the 'usual' characters share the same positions within a group (125*), only certain accents etc. have specific positions.

A bit off topic, but:
"So basically, the following codepages are available, and I believe this would cover almost the complete UTF-8 character space: cp874, 1250, 1251, 1252, 1253, 1254, 1255, 1257, 1258, 8859-1, 8859-2, 8859-4, 8859-5, 8859-7, 8859-9, 8859-11, 8859-15, 8859-16, koi8-r, koi8-u. UTF-8 output has to be remapped to one of those inputs prior to FPDF. Most of the 'usual' characters share the same positions within a group (125*), only certain accents etc. have specific positions."

You have got be kidding me ;). Each of those codepages has 256 positions, but 128 of them are the same between all of them (ASCII). Even if you assume all positions are used (they're not) and that the upper 128 characters of each codepage do not overlap with others (they do), then you still get only 128 * 20 = 2560 codepoints. Unicode 4.0 has 96382 assigned codepoints (of which 'only' 70207 are Chinese Ideographs).

LOL :) Yep, lots of characters. But as you correctly said, maybe iconv, librecode or mbstring extensions for PHP could be used to do the actual remapping. Should be sufficient for ex. Croatian accents (there are not many of them actually - 10 only to remap or UTF-8 decode, out of which at least 4-5 lie in the standard range. That should also cover Serbian, Slovenian). Don't know about Chinese though :). I mean I could use 8859-2 in Drupal and the same encoding in FPDF, but UTF-8 is sooooo great displaying all those weird Š Ž chars worldwide ;)

A friend of mine and I have encountered the Unicode problem when (I was) translating and (he was) developing a few small JS based Acrobat plugins where no Croatian accents were displayed in Acrobat's menu item that plugin generated and to this day we haven't figured a solution out.

I've been doing some PDF reading the last days (for something else, but it was about Unicode too).

Here's what I've managed to gather from the specs:

PDF supports 2 forms of font mapping: simple and composite. Simple is where every byte in a string maps to a glyph in the font, using glyph names. Composite fonts are a lot more complex. Of interest are the CID-keyed composite fonts. These allow you to specify a way to map an arbitrary byte stream to characters (for example from UTF-8 or UTF-16 to Unicode codepoints). This is good for TrueType fonts, because they contain a mapping from Unicode codepoints to glyphs. You can embed TrueType fonts as 'CIDFont Type 2'.

The easiest approach would be to use UTF-16BE and restrict it to the Basic Multilingual Plane (codepoints below U+FFFF, no surrogates [1], which is often called UCS-2). It is especially interesting, because for these characters, every 16-bit word maps to the identical Unicode codepoint, and PDF easily allows you to use that simple mapping with the 'Identity' CIDtoGID map [2].

Now there's the issue of fonts. There are a lot of good fonts out there with a large coverage of Unicode, but the problem is that you probably do not want to embed a 23MB universal font in your PDF. So ideally, you would have a set of smaller fonts (Gentium, Bitstream Cyberbit, ...), check which Unicode ranges they support, check which Unicode codepoints are used by the text, and include only the required fonts.

To get FPDF to do all this is very difficult however. As I said, it's easiest to use UTF-16BE for PDF. PHP strings are 8-bit only, but they are binary-safe so you can put UTF-16 data in them and process them by byte pairs. Conversion from UTF-8 to UTF-16 is very straightforward if you disallow UTF-16 surrogates.

FPDF only supports so called simple fonts. So you would need to:
- Alter the font handling code to embed the file as a composite 'CIDFont Type 2' font with the Identity mapping.
- Change the glyph metrics code so it uses Unicode codepoints rather than WinANSI mappings [3].
- Alter the string handling so it processes the string in byte-pairs and treats each pair as one character (UTF-16/UCS-2).
- Add a smart font selector and switcher, based on the unicode codepoints used in the text.
- Change the output code so that every string is written out as UTF-16BE, properly escaped for the PDF (FPDF's current escape mechanism assumes the string only contains printable ASCII/ANSI characters).

Instead of UTF-16BE/UCS-2, UTF-8 could be used, either only internally in PHP (although it would not make the PHP string processing easier) or also in the PDF itself. In the latter case, a CIDtoGID CMap mapping would need to be created which maps UTF-8 byte sequences to Unicode codepoints.

Notes:
[1] The PDF specs say that the range of a CID is 0-65535. This means that you could support more than just the BMP, but only through multiple fonts which each cover at most one plane of Unicode. Restricting ourselves to the BMP is not that big of a problem.

[2] I haven't managed to figure out if, for TrueType fonts, the map maps to Unicode codepoints, or glyph indices. If it maps to glyph indices, then a custom map needs to be built per font, because a font's internal glyph indices are often assigned arbitrarily. This custom map would be based on the font's own Unicode-to-glyph mapping. In the case of a custom map, the UTF-16 Identity-mapping advantage is lost.

[3] FPDF now seems to rely on .afm (Adobe Font Metrics) files for font metric information, even for TrueType fonts (even though they contain their own metric information). I'm not sure if .afm files even work for Truetype fonts with lots of Unicode characters. In any case, the use of glyph names rather than glyph indices would be very slow for larger fonts.

And for your reading pleasure, here's the link to the PDF specs that I used:
http://partners.adobe.com/asn/tech/pdf/specifications.jsp
(a whopping 1172 pages).

Quote:

Now there's the issue of fonts. There are a lot of good fonts out there with a large coverage of Unicode, but the problem is that you probably do not want to embed a 23MB universal font in your PDF. So ideally, you would have a set of smaller fonts (Gentium, Bitstream Cyberbit, ...), check which Unicode ranges they support, check which Unicode codepoints are used by the text, and include only the required fonts.

Or embed only a subset or a font. PDF can be easily instructed to embed only the subset. And any of Adobe's OpenType Pro (Pro versions are only 'real' OpenType fonts - OpenType Non-pro fonts are simply Type 1 fonts reencoded to OpenType format) fonts could be used as universal font. Not that it helps in this ever growing complication. :(

I've gotten a first version of a Unicode-compatiple FPDF up at http://www.acko.net/node/56 .
But it only embeds TTF fonts entirely. Splitting them up would be a lot more work.

Steven, I see we have posted in the same UTF-8 related forum topic on FPDF.org. :) I belive you are onto something really good. I have downloaded UFPDF and will implement it into one site that is almost finished to see what happens and how it goes. Will keep you posted on anything I notice, to provide as much feedback as possible.

Since I am terribly booked at the moment, please give me several days until the first testing.

This could be GREAT ;)

Steven, would it be OK with you if we posted a small news on abracadabraPDF concerning UFPDF? A link to both your site and Drupal would of course be displayed?

I don't mind, but UFPDF is a hack done over a night. It's not usable for the things that Unicode is most useful for because of the inflexible font embedding.

Ok, will wait and see if it's developed further.

What about switching to TCPDF? (Downside: PHP5 only)

I correct myself: looks like there's a PHP4 version here http://sourceforge.net/projects/tcpdf/

Sounds like a good idea.