Add content (node) translation support to Drupal core

subscribing

yes, we really really want this in 6. so lets work to get it ready. if we run out of time, we should seriously consider making an extension for this patch. thats not so uncommon. the described approach sounds sane.

as for the admin interface, i am hoping to write a Views powered node management page. See http://drupal.org/node/125810. Maybe we just leave the admin UI in Contrib?

Moshe,

if we run out of time, we should seriously consider making an extension for this patch. thats not so uncommon. the described approach sounds sane.

Yes, I will be including the parts of this that don't get into Drupal core for next version into i18n.module

as for the admin interface, i am hoping to write a Views powered node management page. See http://drupal.org/node/125810. Maybe we just leave the admin UI in Contrib?

We have some work done for the admin interface, you can check it out here http://svn3.cvsdude.com/devseed/sandbox/i18n/
But that solution won't scale well for a big number of languages so we are looking for something better

I agree -- this is something we want for D6. It's at the core of what people expect us to deliver.

* An option to toggle translation links for content types? Like we have toggles for author info.

This could be added later.

* Native names for languages in node links? Or translated names in the language used, or both? Now we have translated links. Only themers can alter that now.

Not sure I understand the difference between 'native names' and 'translated names', but the names should always be in the native language (assuming 'native language' means the visitor's language). If you go to a Chinese website, and it has an English translation available, it shouldn't communicate that in Chinese. You want to recognize the word 'English' somewhere, or you would not be able to figure out how to proceed.

* Help text and documentation! Obviously we need docs for the "translate" tab, once basic things are settled.

Yep.

* New content type setting, to enable translation per node type? Like we have "language enabled" node types now.

Does that need a new setting, or can we reuse the existing setting?

In the patch, what does the 's' of 'svid' stand for?

I'm wondering why we need the translation sets -- can't we just add a field to the node table that keep the nid of the source node?

Subscribing

Reviewed and seems to work. Please note that you have to enable Content translation module for this.

One bug: Create a node, add translation. Then view your translation and click Edit. Language selection box will have "Language neutral" instead of "My language"

By tying translations to revisions, you impose a hidden workflow. In other words: if you want the translations to be marked as 'out of sync', you need to create a new revision. If you don't want the translations to be marked as 'out of sync', you can't create a new revision. That might cause problems as I might want to update a node, create a new revision and not mark the translations as 'out-of-sync'. From what I understand, this would be impossible with the proposed module.

Furthermore, it requires that the author know the revision system. Many site don't care about keeping revisions, and that's perfectly valid, IMO.

Personally, I prefer an approach that is both more transparent, more flexible and more explicit. Just add a check box "mark translations as out of date" and keep a flag.

So why not a source node and an extra flag/status field to mark the node as up-to-date.

Reworked a big part of the module.
- Removed all the vid related information. New fields are 'snid' for source node and 'status' (default 0, will be 1 if translation needs updating)
- Followed Dries remarks about 'hidden translation workflow'. Now there are some settings in the node form
a) If the node is the translation source, there will be an option to mark all the translations as 'need updating'
b) If the node is not the source, the 'status' field will be shown, asking whether the translation must be marked as outdated or not.
- Fixed the bug mentioned by meba in #9
- Node links with native names
- Added db_rewrite_sql for the query loading translations

Note that we are now keeping track of the translation source at two levels: the first source nid will be the translation id (trid), while we still store the source that has been used for each specific translation. Why? Because it is easy, it keeps all the information about what has been translated from what and the info is just there if someone else needs some more complex translation workflow.

So, I guess this may need some more work yet, but I think the most important issue is to find some solution for that translation workflow. Please, feedback.

Is it normal ?
- one 'english node', with an associated 'french' translation node
- editing the nodes, I don't have the ability to change their language, but I can set them to 'language neutral' ?

Additionnally, if I do set the english node back to 'neutral', it still shows a link to the french node - and the french node displays an 'undefined index warning (translation.module line 134)

I created a node with French content, and then clicked Translate to make an English version. That worked fine.

Then I edited the French version of the node and made it Language neutral.
1) The Translate tab went away.
2) The English link remained.
3) When clicking the English link, or on teaser view, get:

notice: Undefined index: in /home/.honor/snarkles/webchick.net/i18n/modules/translation/translation.module on line 134.

4) Translate tab is active on the English version. I click it, and I can add a new French translation.
5) Now the original language neutral patch shows both French and English links.
6) So, naturally, I went back to the now Language Neutral version ;) and attempted to change its language. Language Neutral is the only choice there, so that's good.
7) I delete the original "parent" node. The two translations remain. not sure if this is "by design" or not.

And this, folks, is why evil webchick shouldn't test patches. ;) I will try and test something non-edge-casey now. :P I don't even know if this is helpful, although that notice should probably be fixed. ;)

This isn't exactly a bug, so I apologize if this is the wrong place to post this. However, seeing as its related to this patch I though it would be OK here.

I was wondering if the "select node" option for translations is still on the cards? Its extremely useful - I might even go so far as to say critical. Should we submit a patch to the latest patch for this, or should we hold off for a while.

Polls:
1) Created a French poll.
2) Clicked "Translate"
3) BUG: The title is pre-filled with French content, but the choices are not.
4) BUG: I vote on the English side, this is not reflected on the French side.

(note: there are also a bunch of notices, but those are there without the patch.)

#15 brought up an interesting philosophical discussion between yched and myself... ;)

Is #4 actually a bug?

On a Canadian site, you probably want the poll data aggregated; although you have French and English content, most of your audience will intermix regardless of language.

On an "international" site, your French/English/Japanese members might be cordoned off in their own areas and not even know the other exists. In that case, it doesn't make sense for the English poll to increment when a Japanese visitor votes.

It seems we need the ability to configure how this works... we talked about enabling an "Aggregate content" option per node type, but that's ambiguous, since comment-enabled types will do one type of thing, voting-enabled types another, and polls something different again.

About the [lack of] sharing of data between nodes: Gábor's above explanations for the initial patch (#0) would mean that #15.4 is not a bug. :-)

new patch fixes the bug described in #12 - #13 : setting back to 'language neutral' keeps the node in the translation set (and leaves inconsistent state)

+ fixes some } else { code style 'bugs'.

wait, this is no good - dlesieur pointed on IRC that if we remove the 'source' node from the translation set, we have to pick a new 'source' amongst the remaining ones...

Actually this is even a little more tricky :
the nodes in the translation set do not necessarily have the 'mother node' (nid = trid) as a source (snid)...
Say English (1) is mother
Create French (2) from English
Create German (3) from French
trid = English , French snid = English, German snid = French...
remove French, and you have to update German...

Which leads me to : Do we _really_ have to keep distinct concepts of trid and snid ?

Anyway, updated patch, tries to handle 'node removal' nicely with the current state of trid and snid :
- sets a new 'mother' (trid) for the translation set if the current 'mother' is removed,
- sets a new source for the nodes that had the removed node as source.

See the patch, and talk about convoluted code / logic... Suggestions / simplifications welcome

and prevent 'single node' translation sets...

I guess this will be all for tonight.

Reviewed and didn't find any bugs

Some usability remarks that came up during the last Montreal Drupal Meetup:

1. Once the translation module is activated, it would be nice if the Language negotiation settings moved from 'None' to 'Path prefix with language fallback' (the third option), as this seems to be the best option for a true multilingual site.
2. There is currently no way to see which node is the "source" node (maybe it could be shown under the Translation tab).
3. Looking at what is shown under the Translate tab, there's no clear indication that a translation needs to be updated, even though it can be marked as "updated" in the node edition form.

Wanted to quickly say that I tested this patch here:
http://drupal6.dev.openconcept.ca/?q=english_example

and liked how it functions. Quite similar in many ways to the current i18n module. Would love to see this module brought into core.

I was at first confused about applying the patch for this module as the files didn't exist. Anyways, created the directory and then applied:
patch -u < translation_module_04.patch

I liked the "This translation needs to be updated" feature. Would certainly be nice to have other items listed under the "Translation settings" such as - original language.

Mike

@Gabor : I heard you plea here, and I guess having rolled a debug for this patch makes me the one who should stand up here.
I'm quite willing to do so, unfortunately there's absolutely no way for me to make it happen before late thu. 31 at best :-(. If anyone else is ready to pick it up before that - please do !

Also, I'm not sure I understand the thing about a 'three way selector' for node language ?

Got it :-)

I'll try to post an updated patch (well, module) in 2 or 3 days if noone beats me to it.

Ok, here is an updated patch, as promised (and not sooner than promised, but we're still in May in France :-) ).

- trid = snid, so we keep only the snid
- removed the {translation} table, the two translation fields live in {node}
- added a 3rd option for content types settings ('multilingual, with translations')
(sorry, still 1 todo : update the '#description' for the radios accordingly.)
- I also made clearer on the translation overview which node (which language) is the 'source' node.

Previous patch was missing node.module.

So this is not really something targeted for Drupal 6 core (although contrib modules will be able to implement it just fine).

This appears to be an essential feature. Having to re-enter dates, telephone numbers, or whatnot can be a real pain for the user. Being able to share fields seems key to provide a good user experience towards content editors?

Dries: with the current patch, it is currently up to each module to pre-fill the node form elements they control with the values from the source node when creating a new translation (hook_nodeapi('prepare translation')).
Provided CCK implements this behaviour, I think this is a good compromise.

Besides, I'm not sure how we could 'automate' that without relying on the relevant modules ? (sort of) clone the source node and unset nid ?

latest clone.module does what yched describes: http://drupal.org/node/110294. one tricky bit is attachments and filefield/imagefield fields.

Moshe : thanks for the hint. Do you have any idea if the recent changes in files management would maybe alleviate the files / upload issues in D6 ?

About linking other content like files, to different nodes, the only problem is the way upload module in drupal handles the files, which doesn't really allow to link one file to more than one node so that's really a different problem IMHO.

For synchronizing data accross translations, you cannot do it blindly, just copying over all the fields, and not even hardcoding it for a given content type. You need a whole set on configuration options.

Example: We have an event, linked with a location, in two language versions, English and Spanish.
Case a: It is really the same event, same location, same date. Ok, for this case it makes sense to copy field contents
Case b: They are two 'language versions' of the event, like you have a talk for English speaking people, and then another one for Spanish speaking, different places, different times.

Another example: A weight for a product
it's a numeric field so we can think it's the same for all languages. But maybe we'd like to use kilograms in Spanish, pounds in English, so we cannot just copy over the value.

The point is, synchronizing translations is not something you can just do copying over all non text fields. It's a 'higher level' decision and unless we provide a whole bunch of settings we better leave it for contrib modules for now. Because any solution we provide for all translations in general will cause more trouble than benefits for site specific implementations.

Note: I'm implementing a solution for this for i18 module/Drupal 5, and it consists on providing the whole list of fields for each content type the administrator can set as 'synchronized' or not. And the only thing that would help here is to have some _fields hook for all node types which provides human readable information on what each field means.

There is a lot of potential flexibility that you might want for Drupal 6, but I don't think we need to wait until we've got a whole new configuration mechanism to allow admins to choose what fields they want to be translated.

- there needs to be a means to have the translations behave like they did in 4.7 or 5.x (it may seem like a bug, but people may be dependent on it). This should be a simple checkbox in the config.

- we need to implement a new default for synchroniziing translated nodes. Having numeric fields be linked by default will provide a better user experience for most instances. Ideally it will be customizable for each node type. Each module should be able to define an array of linked fields that they want to be the default for multilingual data. It would be quite complicated to do this for CCK node types. The default set of tables could be patched produce a different set of linked resuts.

- Once we've got core modules using this then I think it would be a good time to add in the user interface changes so that users will be able to select which fields they want to have translated for each node type. I think that could wait till Drupal 7 though.

Using the latest patch :

1. Three languages (English, French,German)
2. Create a page, set its language to English, submit
3. Click "Translate" tab

Expected : English is marked as "Translated"
Issue : English is marked as "Not translated", so it's possible to translate the English page into English.

If you translate the English page into French or German, and click "Translate" tab, information are correct : English is marked as "Translated" and "source", French or German is marked as "Translated".

Another small issue :

1. In "admin/settings/language", disable German language
2. Create a page, set its language to English (German does not appear in the list, as expected), submit
3. Click "Translate" tab

Expected : German language is not listed
Issue : German is listed, and clicking "Add a translation" opens a page creation form, its language being set to "Language neutral".

This was already missing in my patch in #33 / #34 :
the update function does not create the indexes on tnid and translate columns

Also, now that we removed the ability set translated nodes back to 'Langage neutral', i don't think this branch in translation_nodepi('submit') is still needed :

if (isset($node->translation) && $node->translation && !empty($node->language)) {
  ...
}
else {
  // Remove node from translation set if it was in one.
  translation_remove_from_set($node);
}

And last for tonight : function translation_path_get_translations($path) does not seem to be called anywhere.

All these points were already valid objections to my previous patches :-/

Gabor, for the node selection option, a patch was already accepted into i18n that added an autocomplete input field above the list of available nodes. For more information, see http://drupal.org/node/137049. Jose mentioned that he did some minor changes to the patch I submitted, so you might want to double check with him exactly what changes he made.

I had another look at this, and have a number of comments. Note that NOT all these comments need to be addressed. It's more of a brain dump, although I would expect other people to wonder exactly the same. It would be great if we could give them some additional thought.

* It seems like a good idea to remove any code that is not being used. I don't think other modules will make assumptions, or that they are not able to deal with things themselves.

* How often do you want to add or remove a node to/from a translation set? I don't have a lot of experience translation Drupal site, but this requirement never occurred to me (although I can see it being useful). Getting the UI right for this seems to be the biggest challenge.

* In the translate.module's global help text, it might be useful to drill down on the architecture a bit. I'd add a second paragraph that describes the different database fields that are relevant to content translations, and how they are being used. That sort of information seems to be a natural extension to the first paragraph.

* I'm not sure whether "one source content" is valid English.

* From a user point of view, the UIs seem to make sense. The one thing that might be confusing is the word "translation set" and "source node". Both sound rather techie to me. I think non-technical people would simply refer to it as "translations" (instead of "translation set") and "the original post" (instead of "source node"). I think we need to think about this some more. First, "Translations" might be the better word, especially in the UI. Second, I wonder why the user should care about there being a "source node" to begin with.

(I can foresee a scenario where there is an English source node that is translated to Hungarian, and where the Hungarian translator might decide to mark the source node as "outdated". At that point, the notion of there being a "source node" is rather moot. At least from the user's point of view.)

To make a long story short -- we might be able to take this one step further and eliminate two slightly technical terms.

* The UI leaves me with a couple of question though: what happens when I delete the source node? Will the translations be deleted too? This is something that we'll want to document because I can't predict the behavior. The fact that there is a source node suggests that the translations will be deleted too. If there is no longer a source node, I would not know what to expect.

* The UI leaves me with another question. On the node edit page you can mark translations as outdated. The form description reads "Mark all the content of the translations as outdated". My question is simple: what does it mean for a node to be marked as outdated? Does that mean that the node will no longer be accessible? To me, it means as much as "flag this node". There are two things missing from this description: the purpose and the effects? Why is this useful? What will this cause to happen? Both are undefined.

* Feature request: if would be great if we could integrate the translations module with the forthcoming actions module. Like that, we create more interesting workflows (i.e. send an e-mail to the Hungarian translation team). This might be a good opportunity to review and evaluate the actions patch.

* Is there an easy way to see all outdated translations? How does a translator identify all the missing or outdated translations that he/she has to work on? I realize that this can be solved by a contributed module, but maybe it is as simple as adding something to the filtering operations on the node administration page. Not sure but maybe worth thinking about. Maybe this requires further integration with the node.module (and the status field) to get this 100% right?

* For mere mortals, the difference between "locale" and "translations" might be a little hard. The administration page shows "locale" and "content translation". I wonder if we can make this better. I know we talked about this before, but it something that keeps bugging me. I don't have a definite answer and I don't expect one either. Just dumping my raw thoughts.

Thanks for the explanation. That helped clarify various things -- especially regarding the 'source node'. New users might still have the same questions as I have, so it is worth communicating the essence of this to the user. You explained it to me, but you might also want to explain it to the user. ;) It's the kind of information that helps the user wrap his head around this. In the help text, explain in 2 or 3 sentences what a "source node" is and how that concept is useful for the user.

Source node deletion: I don't think we would like to see all translations removed at once. Well, how can we communicate this better?

I'm not advocating that we'd delete all the translations. I just think that we need to be explicit about this. Print something like: "This post has x translations. Deleting this post, will not delete the translations.". And then maybe another sentence to explain the implications (if any).

That still leaves me with some questions, but these are certainly less important. One example question: does that mean there will be a new source node that I can use to /fork/ new translation from? What are the implications of orphaning translations. Does that break the integrity of my data?

Flag as outdated: I have updated the labels and help texts there, hopefully it is cleaner now. Unfortunately we don't know what *will* happen, maybe an action will fire if the user set it up, so the post could be unpublished or the translators can be notified or anything. Anyway, I have included a help text, which indicates that only the flag will be set, other properties of posts will not change.

Interesting but still somewhat confusing. I think this part of the patch is somewhat under-developed. There a handful of things that _could_ happen when the source node is updated: we could just flag the translations and wait for translation teams to pick it up, we could flag the translations and unpublish them, we could optionally e-mail the translators, etc. As a user, even as an administrator, you don't really know what is going to happen and you don't have control over it either. Try to put yourself in the position of a user that does not know how translations work.

* I might want to trigger different combinations of these events depending on how important that node is.
* If I don't know what is going to happen, I'm not likely to check that box. I'm going to search for a good explanation first.

So two problems here: (i) lack of control and (ii) not clear what is happening under the hood. I'm not convinced this checkbox is the proper solution. It just adds an extra level of indirection, it seems.

I'm not saying that we need to fix these now. It's not a show-stopper, but it is certainly something where the modules comes short, IMO.

"Locale" and "content translation": I bet you mean the module list?! Otherwise these do not display anywhere... There is no such menu item or page titled "content translation". Well, we can name the locale module "interface translation" there in the module list, if that would not confuse people. I think it could easily confuse them.

Yep, I was talking about the module configuration screen.

I agree that we can add these through contrib modules or incremental improvements. As I said, it's not a show-stopper. Feel free to commit this. :)

we need indexes added on the new fields... yched you think single column indexes on tnid and translate would be fine? I see tnid is queries in many WHERE's on its own, but 'translate' is not queried as it seems... I am not sure we need that index.
Yep, single column indexes should be fine.
The 'translate' column is used in an ORDER BY in translation_remove_from_set (pick the new 'source node' for tha translation set), and in a WHERE in the queries generated by the 'up to date / outdated' filter on admin/content/node overview.

So, well, I kept an index on 'translate' for now.

Updated patch takes care of defining the indexes in node.schema and adding them in system_update.
I also went ahead and removed the 'language switcher'-related translation_path_get_translations() for now.
I think we should commit this without cruft if it's ready, and then add the function back in the language switcher patch where it belongs.

Woops.

Attached patch removes translation_alter_translation_link() as well.