Multilanguage: new i18n implementation in the works

By Jose Reyero on 23 Nov 2004 at 11:50 UTC

I have done a proof of concept of the new i18n implementation, as I outlined in this previous forum. I've sent this to the dev list one week ago, but so far I haven't got any reply.

It is a module (i18n) and some small patches. Provides multilingual nodes, taxonomy and variables, for the moment, but it is enough to make a site look 'multilingual'. It doesn't require major database patches and it is not backwards compatible with old i18n, but only if you are using duplicated tables for languages.

You can get a fully functional implementation from my sandbox: . It is intended only as a developer's preview but I think it already is quite usable.

Although there's still a lot of work to do, I think this works fine, seamlessly integrates with the rest of Drupal, the overhead when disabled is negligible and the patching required is minimal . Also you can enable/disable i18n module with zero impact in the rest of the content. - I hope so ;-)

While I feel possitive about the node language implementation, I still have some doubts about the taxonomy. For the moment, languages have been added at the term level but maybe it would be better to have them also -or maybe instead of- for vocabularies. Any idea?

Whatever, I hope to have it working in some live site in a few weeks. Only I would like to get some feedback and fresh ideas from you and test chances of getting it -only hooks and db patches- into the core -someday :-) Update:looks like lack of interest from other developers, :-((

For the moment I'm taking care of nodes and taxonomy only, but other modules should be also made 'multi language aware' -though I think i18n system won't interfere at all with them, at least with core modules; they just work as expected, only ignoring languages.

Update:I got it already running -this is the version in the sandbox- in my site: freelance.reyero.net, which I am redesigning at the moment.

Here you are some details,

Functional Overview:
---------------------
- Added languages to nodes and taxonomy terms
- Clear separation of interface and content. Allows mixed interface/content languages.
- You can have language dependent variables
- Language can be added selectively to some node types
- Language field is not mandatory, so you can always have language-less nodes and taxonomy terms. These will show up for all languages.
- Basic interface for nodes and taxonomy -terms- translation
- When the module is disabled, you have all the content and terms, only without separation for languages.
- Path aliases can be defined with and without language preffix. They will work someway ;-)
- Initial language is taken from: url, user, browser, drupal default -in this order-
- The language code is added *always* to the url, thus incoming links should work smoothly
- Integration with locale's defined languages.
- About term translations, well, I just implemented them before thinking too much :-( . Hope we will find some use for them. :-)

Some Implementation Detail:
-----------------------
- Added 'lang' field to node table and 'term_data' --- I think it is just some tiny little harmless field which won't hurt anybody, whether you use it or not ;-)
- Translation relations are kept with a 'translation id' -'trid' field- which identifies different language translations for the same object. This field is kept in a separated table for nodes 'i18n_node' -to which some info about translation status can be added- and in the 'term_data' table itself for taxonomy terms
- Language initialization and initial path rewriting is done in the init hook, so if any other module uses arg() or $_GET['q'] in the init hook it can cause some trouble --thinking of moving it to the common.inc.
- Added a hook to module.inc::url to rewrite all outgoing urls: 'i18n_url_rewrite'
- The selection of current language nodes and terms is done with a 'node_i18n_where_sql' and 'taxonomy_i18n_where_sql' functions which I think are quite straightforward and introduce minimal overhead when i18n is disabled.
- As the language is in the node/term table itself, only a simple where clause is required -no need for complex join conditions-. This should be added -I just did some- to all the queries retrieving nodes and terms.
- Permissions are not defined yet, but will be something like: 'administer translations', 'create translations'...
- Language dependent tables -having some tables duplicated for each language- are not required anymore. But the functionality will be kept -it's just 10 lines of code- for some future use. It will remain as a low-level configurable option.

Comments

Flexinode?

chx commented 23 November 2004 at 14:27

One of the biggest problems -- for me -- in the old i18n was the lack of flexinode support. I know it's a tough one... but it is very widely used. Or am I wrong and the new one supports flexinode?

--
Drupal development: making the world better, one patch at a time. | A bedroom without a teddy is like a face without a smile.

Yes, it does support flexinode

Jose Reyero commented 23 November 2004 at 19:48

This is the only not core module I tested with and it seems to work fine.

In fact, with the new approach, most of other node types will be supported, as it doesn't use -dirty- synchronization anymore. It just doesn't need to.

https://reyero.net

clarification

oc commented 24 November 2004 at 04:10

your post states:
"it is not backwards compatible with old i18n, but only if you are using duplicated tables for languages"

I think this meant to say:
"it is not backwards compatible with old i18n unless you are using duplicated tables for languages"

please confirm... thanks!

Nop

Jose Reyero commented 25 November 2004 at 08:42

It means

If you are using duplicated tables then it is NOT backwards compatible.

The multiple table approach and the synchronization are not used -nor needed- anymore.

https://reyero.net

Trying to Get this to work

oc commented 24 November 2004 at 04:45

hi Jose -

I really wanted to try this so I downloaded the files in the sandbox:
http://cvs.drupal.org/viewcvs/contributions/sandbox/jareyero/multilangua...

1) The first problem I found was a function at the end of taxonomy.module that is never called:

function taxonomy_i18n_where_sql($table='t',$type='',$param=''){
if(function_exists('i18n_taxonomy_where_sql')){
return i18n_taxonomy_where_sql($table,$type,$param);
} else {
return '';
}
}

Apparently in 2 places you were calling i18n_taxonomy_where_sql instead.
So please update your CVS to change the 2 references to i18n_taxonomy_where_sql to taxonomy_i18n_where_sql .

2) I am not sure of this because I don't exactly know how drupal works, but in i18n.module, should the includes at the beginning point to the current directory, as opposed to:
modules/i18n/file.php

3) After installing this I wasn't able to do very much... maybe I don't have it all working yet. I set my default language to Spanish with the module disabled. I saw everything in the Drupal interface in Spanish. Then I enabled the new i18n module. It reverted back to English, even though I was still set up for Spanish. Then I went to the page for My Account to change my default back to English. It took me to a blank page with an extra slash in the URL:
http://organizersdb.org//user/6?PHPSESSID=0e8db15c325c4a8fba6866e1eb0543fe

if I get rid of the extra slash and reload everything is fine. I think the extra slash had something to do with the fact that the i18n module is not picking up the list of languages that are installed with locale.module or something like that.

4) Install.txt issues

a) not clear if we need to modify conf.php... it says in part 3.1:

"If you don't want to rely on locale module for supported languages, you have to add them in the config file:
$i18n_languages = array("es" => "spanish", "en" => "english"); "

Does this mean I can just omit this line of code and it will still work?

b) not clear if what it says in part 4 is mandatory or not:

"You need to modify $db_prefix and add $db_prefix_i18n in .conf file"
I see that this is there for backward compatibility... if I am not managing language dependent tables can I omit this step?

c) the png image referenced in part 5 does not exist:

http://lists.drupal.org/pipermail/drupal-support/attachments/20030530/7a...

Overall it looks like a nice piece of work once I get all of it running. Thanks Jose and I am happy to help with documenting this as it progresses.

-rc

1) The first problem I found

Jose Reyero commented 25 November 2004 at 09:31

1) The first problem I found was a function at the end of taxonomy.module that is never called
You are right. Fixed.

2) should the includes at the beginning point to the current directory, as opposed to...
No, all includes are made from Drupal root.

3) After installing this I wasn't able to do very much...
You can choose in i18n settings between languages defined in config file and locale module languages. If it doesn't work with locale, define languages in the config file.
There may be some issues with the locale module - and with the menu system, which does too much caching...

4) Install.txt issues
Does this mean I can just omit this line of code and it will still work?
Yes, but you have to change i18n settings to use locale's languages.

b) not clear if what it says in part 4 is mandatory or not:

"You need to modify $db_prefix and add $db_prefix_i18n in .conf file"

Yes, you are right, not mandatory . It says

4. [Kept for backwards compatibility and experimentation]

c) the png image referenced in part 5 does not exist:
You are right, it's been removed and there's a broken link in Drupal documentation.... Anyway we don't need it

d) part 5.1 about language dependent variables, the doc says you can have as many variables as you want in that PHP statement, but does not offer guidance as to whether I want a lot or a little. Can this be a bit more helpful in this regard?
The ones proposed are just one example. There are a lot of variables in Drupal that keep texts -which are 'content' in my oppinion- and currently are translated with locale module.
I just don't like them and this is my proposed fix, but you dont need to use them, only if you dont want them translated as locale's strings.

https://reyero.net

Which version?

blackjam commented 24 November 2004 at 15:01

I tried provided patches against CVS HEAD, but didn't succeed. Which Drupal version do you use with this module? Can you provide more detailed instructions on patching? I am a rookie patcher...

I am eager to test it, because I plan to create a multilanguage site soon.

Jakub Cerny

Patches are against Drupal 4.5.0

Jose Reyero commented 25 November 2004 at 08:34

Sorry, I forgot to mention

Anyway, it seemas my patches have some trouble, so this is a quick fix: I commited the patched files to the sandbox, a folder named 'patched_files'.

https://reyero.net

4.5.0 -- no surprise

chx commented 25 November 2004 at 10:15

It would be very hard to keep up with HEAD these days. Since 4.5.0 there are vast changes, new search, new block config, check_query renamed, multisite patch... It'd be a pain to redo the patches every time a new huge patch is applied and almost every module is changed.

--
Drupal development: making the world better, one patch at a time. | A bedroom without a teddy is like a face without a smile.

It would be very hard to

Jose Reyero commented 28 November 2004 at 16:06

It would be very hard to keep up with HEAD these days
Yes, that's a problem.
For the moment I will be using 4.5.0 for development. And also because I plan to deploy it for some sites running 4.5.0
And unless most of the patches were accepted for the core, I have no intention of doing and redoing patches every week, so I will be developing for latest stable version

https://reyero.net

thanks for fix, I am a few steps further

blackjam commented 25 November 2004 at 15:45

I used provided 'patched files'. I was able to enable i18n module. After that, Drupal started to prepend 'en/' at the beginning of the URL. This leads me to 'page not found'. When I manually remove language code from URL, it works.

I selected to use languages from 'localization' module and I do use clean URLs. I checked my base URL in conf.php and it is set properly.

Looking forward for further hints/updates.

Page not found?

Jose Reyero commented 25 November 2004 at 16:32

I think these 'page not found' errors may be the result of having 'Front page: Language dependent' in i18n config,, or not having the languages properly configured.

If you set 'Front page: language dependent', then you have to create an url alias for every language, like en/node, es/node... It is the only case when incoming urls are not searched for aliases with and then without language.

https://reyero.net

Got it working

blackjam commented 26 November 2004 at 11:22

Finally I made the i18n module work. I didn't realize that it is necessary to use the tgz distribution version 4.5. I used DRUPAL-4-5 branch from CVS and it didn't work...

So now I start testing the module.

DRUPAL-4-5 works, too

chx commented 26 November 2004 at 11:54

I've already emailed the necessary, simple patch to Jose. Contact me if you need it.

--
Drupal development: making the world better, one patch at a time. | A bedroom without a teddy is like a face without a smile.

1 patch still not working

oc commented 28 November 2004 at 15:43

I noticed that the patches were fixed, however one of them, bootstrap.inc still gives an error:
# patch < i18n_bootstrap_inc_01.patch
patching file bootstrap.inc
Hunk #1 FAILED at 73.
Hunk #2 FAILED at 89.
2 out of 2 hunks FAILED -- saving rejects to file bootstrap.inc.rej

I am just going to use the already patched file but I figure the person who fixed the patches could check and fix this one too. thanks!

Organizers' Collaborative -- Free Software for Activist Groups
http://organizenow.net and http://organizersdb.org

Problem's/Issues/ToDo's

aam commented 28 November 2004 at 12:57

I am running a multilingual site (en/de/ru) since 1996 and from 2003 with drupal (http://aam.ugpl.de/). For me there are three main issues, worth to work on:

Synchronized language versions of single node's
Only if content translation would be optional but an every time possible choice (just like adding a comment) and all language versions will be synchronized (node's) you will solve all problem's with manually managing/synchronizing/linking your multilanguage content.
Clean URL's and/or language parameters
Only if Clean URL's (with a language part/indicator) and/or language selector's in URL parameters are ALWAYS used, drupal's cache system will always work fine. I do not mean language dependand cache table's, that would be a bad choise.
Current implementation of text and menu translation in locale module.
If you are using only one single language and have the cache system activated, you have no performance problem's with multiple query's on locale table... , but only when you are not logged in. It is better for general performance not to make database query's, but to simply load a given translation into the module's. Okay, when the cache is really good and checks language dependant content and is always working in the background, even you are logged in, you will probably solve the performance problem too.

AAM: Building the Bridge to Real Content

P.S. Due to the issues above, I am using only the language selector from the language block now, all other features are disabled... But if you need a new tester or want me to test some patches (on postgresql), just drop a message.

Got it Working - few more questions

oc commented 28 November 2004 at 16:20

hi Jose -

It appears that everything is now installed... I noticed that when you disable the new i18n module you naturally get "pagenotfound" errors because the current page was rendered with the longer urls (/es/...). This is not a bug, it is to be expected.

I have a couple of additional questions.

1) Does the new module alter the way language defaults work? I went into:

http://organizersdb.org/admin/locale

And set the default language to spanish. However I set my personal default, under:
http://organizersdb.org/es/user/666/edit

to english. with the new i18n module enabled, all of Drupal's interface is in Spanish. It appears that the new module is ignoring the settings of the user who is logged in... which is probably useful for testing. It occurred to me that the options for choosing a language under the user settings option 'Interface language settings' should probably include an option like 'use site default.' If we had that option we could enable that for testing purposes and your module could then consider the user's settings.

2) Can you say anything about whether this module will support shorter URLS for the 'default' language? I mean, it would be nice, in a french site, if:

http://sitename.com/fr/node/34

would be equivalent to:
http://sitename.com/node/34

3) A few details about how you would go about creating a new english page and spanish page would be useful for testers... i.e.
- be sure the es and en languages are defined
- go into i18n settings and make sure page (pagina) is checked of
- create 2 different Drupal pages
- etc. etc.

I can confirm, that the

aam commented 28 November 2004 at 16:27

I can confirm, that the language setting from i18n can superseed user's locale setting.
Depending on different user locale settings, language detection and keep language setting you can get part of a menue in one language and other parts in other language. In my case the admin menu was in english (my user setting) and the rest of the menu's got their languiage from i18n.

AAM: Building the Bridge to Real Content

From the initial

Jose Reyero commented 29 November 2004 at 09:22

From the initial post:

'Functional Overview':
---------------------
.......
- Initial language is taken from: url, user, browser, drupal default -in this order-

So, if you have 'browser language detection' disabled and no user language, it should default to Drupal's default language.

But of course you cannot have both: either you use the browser language or the systems's default

About question 2) -shorter urls-, I think it could be easily added as an option, but it is not in my top prioriries list. And again, it is not compatible with browser language detection.

And yes, the documentation must be improved.... I will add most of your suggestions for the next release...

https://reyero.net

Confirmed: Drupal Default Ignored

oc commented 30 November 2004 at 17:35

I think that there is a bug then, the drupal default is not considered. The Drupal default is overridden by your i18n setting in conf.php.

I just tested it: when I18n is enabled and browser language is disabled, the default language is determined by whe $i18n_languages variable, which is one of the things you define when installing the new i18n. Whatever is first in the array determines the default: I was getting spanish until I changed the line so that English was the first element in the array. Then my site defaulted to English. Here is the line I am talking about.

$i18n_languages = array("en" => "english", "es" => "spanish");

-rich

Organizers' Collaborative -- Free Software for Activist Groups
http://organizenow.net and http://organizersdb.org

Important Update, next steps

Jose Reyero commented 30 November 2004 at 15:17

Most of the feedback I've got is possitive and the module seems already quite stable -this is only a side effect of a clean implementation :-) . It seems Drupal 4.5.1 will be here soon, and looks like some important changes to the init system have been introduced, and this 'i18n' can take advantage of them.

I have already upgraded my development version to latest DRUPAL-4-5, and started testing. And we are moving ahead. So this is the plan:

- I will be committing the new patches and updates under the cvs -HEAD- version of i18n module, so this will be ready for 4.5.1 when it comes.

- The next patches will be against DRUPAL-4-5 branch, and then against DRUPAL-4-5-1

https://reyero.net

Bug found, please verify...

oc commented 30 November 2004 at 17:17

There is a problem with the i18n implementation as provided: you can't turn some of the features off! The common.inc patch adds these lines:

+ if(function_exists('i18n_url_rewrite')){
+ $path = i18n_url_rewrite($path);
+ }

The behavior I get is that my URL gets rewritten with "/es/" prepended to it, even after I have disabled the i18n module (and even after I have english set as my default language).

This only happens in the admin/modules part of the Drupal menu hierarchy. Is there a way to fix this line of code so that the i18n rewriting won't happen when the i18n module is disabled?

-rc

Coding issues

Dries commented 30 November 2004 at 22:50

I spent 2 minutes looking at the il8n code.

IMO, you could do a better job adhering to Drupal's (implicit) coding conventions. In particular with regard to spaces/tabs, and the placement of curly brackets.

Also, in user output we do not 'Capitalize Multiple Words in a Sentence'; instead we 'Capitalize the first word only'.

As far as SQL queries are concerned, we try to upper-case SQL keywords (SELECT, FROM, WHERE, ...). Also, I urge you to update your SQL queries to use the %d and %s directives rather than injecting variables yourself. Your code is insecure! People should refrain from using this module until this gets fixed. (I demoted this forum topic from the main page and notified Jose.)

Lastly, we try not to abbreviate words: for example, we write 'information' not 'info', or 'statistics' not 'stats'.

Ok Ok Ok KO :-( I'm

Jose Reyero commented 30 November 2004 at 23:19

Ok
Ok
Ok
KO :-(

I'm not going to say in my discharge it was just a 'preview', so I'm not saying it, am I? ;-)

All of this will be addressed before any new -kind of- release.

Thanks for looking at the code.

https://reyero.net

Error in SQL syntax when previewing

gjost commented 2 December 2004 at 19:39

I'm getting the following error when i first create a story or page. This morning it's only happening during the initial preview - there are no errors when i submit the story. yesterday it seemed to happen on every preview, but maybe my memory is faulty. :)

user error: You have an error in your SQL syntax.
Check the manual that corresponds to your MySQL server version
for the right syntax to use near 'where n.nid !=' at line 1
query: select n.nid, n.lang, n.title, a.trid from node n
inner join i18n_node a on n.nid = a.nid
inner join i18n_node b on a.trid = b.trid and b.nid = where n.nid !=
in /usr/local/web/drupal/includes/database.mysql.inc on line 125.

Great work on the i18n module so far! i'm looking forward to building a multilingual site with it.

Extra node.module and taxonomy.module

gjost commented 2 December 2004 at 19:44

I'm seeing some strange behavior on the Administer > Modules page. Extra node and taxonomy appear after i18n, in addition to the regular ones. I can enable/disable the new ones independently of the originals. It doesn't seem to affect anything else, but i'm new to Drupal, so i can't be quite sure...

delete i18n/patched_files

chx commented 2 December 2004 at 19:49

Jose had some patch problems, so he created this directory with the patched files, not needed.

--
Drupal development: making the world better, one patch at a time. | A bedroom without a teddy is like a face without a smile.

Update: Release coming

Jose Reyero commented 2 December 2004 at 23:02

Most of the issues here have been addressed, others don't apply any longer, and the new module is not in my sandbox anymore.

You can get the cvs version which will work with Drupal 4.5.1 and -I've not tested this- should work also with Drupal 4.5.0

From now on, please, report bugs, fill support requests, etc... in the Internationalization module

Don't forget, the new version is still 'cvs', not 4.5.0. The 4.5.1 version will be released soon, if I figure out how to create a new release with this 'project' thing.... :-)

Thanks for all the feedback.

P.S: For the ones already using this 'preview', you will have to manually update the database field names: from 'lang' to 'language' in these tables: 'node' and 'term_data'

https://reyero.net