Multibyte (e.g. Chinese, Cyrillic, etc.) character support

Thanks for the report. I'm not that experienced with Chinese characters, so I'm not sure where to start on getting this in. I also don't know how much the encoding of the PDF that is produced would play a part.

i18n is certainly important, but if anyone else watching the queues wants to pitch in and offer ideas (or patches), that'd be great.

I'll start thinking about this one and see what can be done.

This definitely needs to get done, but no additional interest has been expressed, so postponing.

It is possible that the problem is just that the XFDF generation code doesn't correctly handle multibyte characters. PHP has poor multibyte support. The use of eval() in that code might introduce issues. I suggest making an XFDF with Chinese characters and trying that with pdftk. If that works, see if correct XFDF files are being generated.

Anyone have an XFDF file with Chinese characters? :)

This should be tested/solved in Drupal 7 first and then backported.

Tagging as one I want to work on during DrupalCamp LA 2012.

Yep, this fails pretty hard. pdftk actually comes the closest...but it just fills in characters that look like missing image icons in Internet Explorer.

I have to dig deeper on this. It's probably going to get hairy. Stay tuned, and any input from people who've done PDFs and UTF-8 programmatically would be valuable.

I have try to use fillpdf-service and use an pdf with chinese characters. When I output pdf, I have the following error and it seems with Font missing issue?

[[o:Exception]:"java.lang.Exception: Invoke failed: [[o:FillpdfService]]->text((o:String)[o:String], (o:String)[o:String]). Cause: ExceptionConverter: com.itextpdf.text.DocumentException: Font 'KozMinPro-Regular' with 'UniJIS-UCS2-H' is not recognized. VM: 1.6.0_20@http://java.sun.com/" at: #0 /var/lib/tomcat6/webapps/JavaBridge/java/Java.inc(232): java_ThrowExceptionProxyFactory->getProxy(2, NULL, 'T', true) #1 /var/lib/tomcat6/webapps/JavaBridge/java/Java.inc(358): jav[...]279): Java->text('fill_1', '12345') #9 [internal function]: fillpdf_support_merge_pdf('JVBERi0xLjYNJeL...', Array, 'EH6spgwPbamw2yq...', true, Array) #10 /home/fillpdf-service/public_html/includes/xmlrpcs.inc(211): call_user_func_array('fillpdf_support...', Array) #11 /home/fillpdf-service/public_html/includes/xmlrpcs.inc(70): xmlrpc_server_call(Object(stdClass), 'merge_pdf_v3', Array) #12 /home/fillpdf-service/public_html/xmlrpc.php(13): xmlrpc_server(Array) #13 {main}]

Hmm. Maybe all of these issues are font-related. I will aim to check into this Sunday.

#1826458: Extended ascii characters are dropped was marked as a duplicate of this. It may turn out to be two separate issues, but I want to look into them together for now and get a general understanding of how non-English languages tie into PDFs, the filling process, pdftk, and JavaBridge, and figure out the limitations and (hopefully) fixes to get it all working regardless of language, or at least know the inherent limitations and be able to tell people about them and have issues open to solve them when a solution becomes possible.

I had an email conversation with csunny about this some time ago. Here is a sanitized/generalized version:

csunny: Hi! I am a programmer in China and very interested in getting fillpdf to support Chinese. I'm new to fillpdf(but used drupal for one year) and please tell me if you need any kinds of help from me.

---

wizonesolutions: The code for the JavaBridge-based fill method is on http://github.com/wizonesolutions/fillpdf-service. If you can figure out how to make that work locally and let me know that would be helpful. Then I can implement the fix on the service. It seems like I might just need more fonts, but I'm not sure if that's the only issue.

---

csunny: Yes, form the itext doc:
If you are using CJK (Chinese, Japanese, Korean) font technology (fonts that aren't embedded in the PDF, but that use font packs you can install with Adobe Reader), you need an extra iTextAsian.jar.

And I will try this jar to see if it works.

...

csunny: Update,
I have placed iTextAsian.jar(a little namespace correct from "com.lowagie.text.* to com.itextpdf.text.*) in PHP/JavaBridge servlet lib directory. And now the origin pdf with chinese characters displayed well in the result pdf and error "Font 'KozMinPro-Regular' with 'UniJIS-UCS2-H' is not recognized. " disappear. But the chinese character from webform(input) displayed as white spaces.

...

csunny: I quickly get it working. But the solution is specific for chinese fonts.
I add the following before setField:
BaseFont bf=BaseFont.createFont( "STSong-Light", "UniGB-UCS2-H", BaseFont.NOT_EMBEDDED);
this.form.addSubstitutionFont(bf);

Haven't figure out more general way to deal with this.

---

wizonesolutions: Ok, that's something at least. Thanks for the info.

Really do want to solve this in the near term.

buh? Re-de-tag.

Bumping this from the Greek guy who really wants this fixed :)

And really with pdftk surrpot if possible as I don't have access to javabridge on the shared host, but then any actual solution would suffice and I'd figure out the rest.

Rest assured that this is high on my list. If you are able to sponsor anything, that would help too. Hit me up on my contact form if so.

Same problem with cyrillic characters. They disappear in downloaded pdf. I use fillpdf-service API Key.

Yes, I'm aware of this issue. A pull request to https://github.com/wizonesolutions/fillpdf-service is welcome. Otherwise, I will fix it when I have time or when I get sponsored/hired to do it.

It is not necessary to raise the same issue on multiple forums and via email. It will not result in a faster solution (unfortunately, it tends to have the opposite effect).

I am resetting this issue back to how it was. It is considered improper in the Drupal community to change the title and version of issues in the way you did here.

Sorry if I made something wrong. I'm new in Drupal community. I asked question about cyrillic support to filpdf-service 10 days ago but didn't get any answer. Btw sorry and thank you for your work.

Yeah, I was on vacation last week. Normally I reply more quickly :)

I am facing the same problem when using fields with greek characters to fill a pdf form. I am using pdftk (ver. 2.02) as a backend. What seems strange is that when I use flatten=0 as a GET parameter in order to make the fields editable, while I can't see the greek characters at first, they suddenly appear when I click on the form's fields.

Moreover, I have tried to use the [need_appearances] argument that according to the pdftk manual "Sets a flag that cues Reader/Acrobat to generate new field appearances based on the form field values. Use this when filling a form with non-ASCII text to ensure the best presentation in Adobe Reader or Acrobat. It won't work when combined with the flatten option." However, this doesn't seem to help.

What I haven't understood so far is if this is an issue specific to pdftk or related to the PDF format or there is something to be done regarding the code of this module.

Well, it seems I was mistaken in my previous post! When I use the [need_appearances] parameter the greek characters appear correctly! However, one cannot use this parameter with flatten, so flatten needs to be explicitly set to 0 as a GET param in the URL. Moreover, I don't know if this will work in previous versions of pdftk, I think this option was recently added.

I have changed the following line:

passthru($path_to_pdftk . ' ' . escapeshellarg(drupal_realpath($filename)) . ' fill_form ' . escapeshellarg(drupal_realpath($xfdffile->uri)) . ' output - ' . ($flatten ? 'flatten ' : '') . 'drop_xfa');

to:

passthru($path_to_pdftk . ' ' . escapeshellarg(drupal_realpath($filename)) . ' fill_form ' . escapeshellarg(drupal_realpath($xfdffile->uri)) . ' output - ' . ($flatten ? 'flatten ' : '') . 'need_appearances drop_xfa');

Thanks for the tips; you can add it in the documentation pages if you want. I want to get UTF-8 characters working properly with flatten, at least in https://github.com/fillpdfservice/fillpdf-servlet. I'm planning to rewrite that eventually and will probably tackle it there. It's good you found a workaround for now. You're welcome to open a separate issue in PDF Forms API and submit a patch to enable need_appearances. You'd have to make PDFtk work properly as well, though, for it to be of any use, and you might have better things to do :)

I confirm #23 also working for Chinese, simplified characters. I successfully open the generated PDFs in Adobe Reader and Chrome in Windows 7.

The man page for pdftk says that need_appearances doesn't work with flatten.

Hmm, I wonder if need_appearances works if one takes two passes at the PDF? One pass, and then run pdftk again with flatten. Would be interested to know, but my focus isn't currently on this issue.

I think if you flatten after running need_appearances, it will make need_appearances do nothing because by flattening, the final appearance has already been determined.

Yes, if I set flatten=0, the generated PDF can show Chinese characters normally, but the PDF can be still editable. If set flatten=1, Chinese characters can not show up. So that's still not perfect.

If you need more help, please re-open, provide details, and move this issue a supported branch: 7.x-1.x or 8.x-4.x