Break up locale.module, but how?

One suggestion that @sun, @catch and @Jose Reyero voiced in the related issues and discussions is the following:

1. Move all language management UI and API out of locale module and current language includes to a language.module.
2. Make language module required for *all* Drupal sites.

Upsides:

A. This would get us a clear namespace of language_*() for clearly language related functionality and we can do the UI translation part in another module (current working name locale, but their suggestion did not reach into discussing a name for that).
B. Those who want to work with a pluggable localization solution (eg. string overrides) can still use the language list functionality separately.
C. You'd be able to assign language information to things without also using UI translation. Eg. if you catalog card games on a webshop, you can mark the games with their language with this functionality, even though your shop might only be in English. (Yes, you can do this with taxonomy just as well).

Downsides:

A. All Drupal sites will have a language module always enabled. This will show a UI which might look strange for those who did not want to deal with different languages.
B. Even English Drupal sites would need to consult the language database table for language information.

I hope I adequately summarized this proposal on behalf of the ones named. Comments welcome!

BTW Drupal currently uses drupal_multilingual() (I'd argue this belongs to the language_*() namespace) and the variable used in language_default() to keep performance good when only one language is enabled (even if that is a foreign language). Since both depend on / use regular Drupal variable_*() functions, their data is in memory when the variables are bootstrapped, so for single language sites, there is no lookup on the languages table. We can theoretically keep similar performance workarounds in Drupal 8 (although the object in language_default() is not well regarded, see #1272862: Clean up default language handling).

Wanted to mark this patch for performance and usability, but since it seems like performance can be the same using the same "workarounds" as D7 does, just marking it for usability.

Well, if this depends on the promise of the CMI, then maybe we should return this later when it makes it possible to do this more lightweight. I know / assume the goal of the CMI would be to do everything via that system which are not nodes, and that is great for unification. So I'd assume if we get there, the language list would use that system too.

@Alex UA: yes, the UX impact of this would at least be a "Languages" item in the Regional category, that would have English enabled by default and let you add more languages. Currently when you have the "language list enabled" = "locale enabled" ATM, you get to have a language dropdown for example on your node forms, that has "language neutral" in that additionally to English. Now, since all sites would have the language list, they might or might not want to have that. This is a UX issue we need to rework. There were various suggestions elsewhere to have a language list specific to actions, like a language list you want for your language switcher might be different to what you want for your node submission (eg. later might include language neutral, the former would not). Anyway, since enabling a language list would not be an explicit operation, all the things that come with it need to be considered.

Also, another UX problem is that when you actually add a new language, "nothing" will happen. It does not really help the 4/5 of the world in itself (your number estimation), since they'd still need to enable the locale (or whatever the name for the UI translation) module first to see real action when they add a language, and actually get some UI translated too (which is their goal). So this becomes a process tumbling block for these people IMHO. Maybe we need to prompt people when they try to add a new language if they also want to enable interface translation functionality with it or just want to have the language lying around to be able to mark posts with that language, but keep the UI all English still.

All-in-all I think there are some subtle and not so subtle process details to consider here additionally to the code organization, performance, etc. questions.

@Crell: yes, your second point is being addressed in #1266318: Make English a first class language. On your first point, how do you make the UI "autoload" in the same way? At this point a language module would have all the common (already in common always included include files) API functions available in its code as well as a hook_menu() and then branch off to include files for admin pages. That would be the D7 way of doing it at least, which is at this point equal to the D8 way of doing it. Do you consider that non-issue that to avoid having a required module, we need to have certain of its APIs in include files / autoloading classes, whatever, but its UI is still in the module (and using the same namespace)?

@Lars: I don't think that it solved our problem if (in your scenarios) either Locale or Language module handles languages. It is not much of an improvement compared to what we have now that either the base system or locale module handles them. We are trying to move it to one place not move the two separate pieces around :)

@sun:

- Starting with ii) would separate functionality from locale module BUT would still have the language_* namespace spread around. You don't seem to have a solution for that here (even on the long run).
- You don't list the language negotiation UI at any place in this model, but you already have its API in i).
- My understanding is that in your model, the user process for enabling ii) would be enabling a module. What would be the user process for enabling iii)? Such as being able to assign language information to a node/entity/piece of content? If its already there, how is ii) separate from iii)? If its not there, does it come with enabling yet another new module?

@Jose Reyero: that sounds interesting. I agree we need to structure this discussion a bit more to put people on the right rails for this conversation so to say. In fact I was working on two little figures to help illustrate the site types we are working with. I've designated 5 types, although admittedly these are not necessarily 100% distinct. These figures attempt to document the status quo to some degree but also attempt to distinguish features that could be separate depending on whether people need them for different site types. Some notes below.

(Find this figure at https://docs.google.com/drawings/d/1psJW4Jvv0ZYRWtzjHV7zWz02eJQAC-w9HsW5... too).

Here I imagine/explain the situation where the installer creates either of the two most typical site types. Fully English (SE) or fully single foreign language (SF). Because English is special in Drupal in that the built-in software UI is English, the SF site already needs UI translation but would only need to know of one language. No language selection involved. Content language UI is only needed if we want to offer the "Language neutral" option for content, and that is needed either way for SE as well, if people want it. Needs to be configurable IMHO. Jose's simplified single language idea fits in here.

Then you can move from SE by adding more languages (to SEM), but only using them for content (eg. you have an English site but want to post foreign posts occasionally, such as some bloggers). You don't need UI translation, content translation or config multilanguage support because you only want to manage content in some other languages. Konstantin Kaefer's blog used to be a good example for this (edit: in fact, its still a good example, see http://kkaefer.com/journal/page/4/ ATM has English and German posts, all UI is English). Language negotiation/selection is optional, depending on whether you need to have access to your content / listings language aware. Not required.

If you move from SF to add more languages for a multilingual site (or from SEM to add UI translation for those languages). In this case, you most probably want to have configuration in separate languages too and want to employ language negotiation and selection to drive the user to the right language version. Note that translation is not necessarily required. You can set up a gate page with language selection and people just get to the right language version of the site. Or do domain access for separate language areas. You name it. http://www.toughbook.eu/ is a good Drupal site example for this. There are no internal language switchers, you get a big welcome page for that. It might not have any translation relation between content on the subsites. All this needs is a UI to tell this content is in this language or this configuration is in this language, and it will use the right one on the right language page. (Edit: http://www.beyonceonline.com/ is another good (Drupal site) example, which seem to have editors working separately on the separate language posts, if you switch languages, the set of posts you see are entirely different. I'm not sure this site has any translation relations for posts.)

Finally, you can move to add translation relations (ie. UI for content and config translation) (SFF), which I consider the full multilingual site.

Note that even a very simple foreign language website needs UI translation, because the UI is in built-in English out of the box. However, all other stuff (content+config) is user provided and can be marked with proper language info, so only the highest level multilingual site needs actual translation for those.

Now how do we do progressive enhancement to expose features relevant for the sites in question without piling all features into one module or piling unrelated features into the same modules? The current module set for these features is kind of all around the place. See the right end of the figure.

Here is a figure of the same site types with the transitions marked with what needs to be enabled / added at that point to the site in terms of functionality/setup:

(Find this figure at https://docs.google.com/drawings/d/1bBZ78F8ReSigCgV3aQJmMNYceHOCg1J1Zsoc... too).

I hope this helps structure the conversation to figure out how to build our functionality to make sense for people with different needs. Please let me know if I made mistakes in identifying the site types or the related features needed.

(Smaller images to not break the layout horribly).

BTW language path prefix and domain configuration is moving to negotiation configuration itself, where its used / makes sense, so adding languages is getting to be a very simple and lightweight operation already. This fits in very well with not always needing language negotiation when you need to have a list of languages. See #1280530: Decouple domain/path negotiation setup from language configuration.

@plach: hm, well part of my point is there are different building blocks configured differently based on site type. I'm not sure we should have plugins per site type, that would duplicate code. Or you mean these plugins would pre-configure other plugins? Hm. I think we should look if we can make the components configurable so that people can cook up their combination. Here is a different take with the component relations based on the above figures. Language assignment should always happen, so even if you go single language, sometime later you might go multilingual, at which point we need all stuff to already have language info. Then depending on your number of languages and needs, you might need UI translation or language negotiation, and if you have multiple languages, you might want to go to a site with translation. I hope this makes sense and helps put an even clearer perspective on the issue.

(Figure can also be seen at https://docs.google.com/drawings/d/1jnabdm_uvEfvSMK55Bw6HSKZ1yRWmP9a_WsH...).

@Crell: well, hook_hook_info() is for autoloading module/hook files possibly per hook, so it can be used to do autoloading for alter hooks, right? I'm not saying its not a hack in cases, ie. I'd put the menu hook at least in the module file plus maybe a hook_hook_info(). I think the real problem is with general API functions that people generally want to use, if those are in include files that manually need to be loaded, it becomes very cumbersome. I understand the autoloaded include classes would work for these, I'm still looking forward to the patterns to use for them :)

All right, this sounds like my careful figure compositions halted this whole conversation :) Too much to digest? Everyone agrees on those breakdowns? We still need to define the steps and break up functionality :)

On the way is #1260860: Rework language list admin user interface to clean up language configuration and #1280530: Decouple domain/path negotiation setup from language configuration to make it much simpler. With these two, adding and configuring languages becomes much simpler. There is also #1296566: Improve usability of add language screen for the add-language screen itself which needs feedback. So all these make the language setup experience simpler. They still don't serve the single language site maybe as good as we can. If you install Drupal in the right language, everything is good, but if you install in English let's say and later need to switch languages, you need to add a new language, delete an old one, etc. Cumbersome. It is not as easy as changing timezones from a dropdown. Granted there is data related to your site language at that point, so making it simple to switch away from that might not be a good idea. Do we want to build language migration tools into core or solve this in some other way for the single-language website when they switch language? Assuming they'll have it right from the installer is not really possible IMHO.

The other ubiquitous part in #33 was language assignment. Well, that work landed for path module in #1236680: Move path language settings from Locale to Path module and should be worked on for node module at #540294: Move node language settings from Locale to Node module. These are being generalized for a form API field for language assignment for other type of data in #1280996: New language_select element type for form API. Eventually we need to assign language information to all pieces of data in Drupal and we need infrastructure support for this in core.

Sounds like these are all issues that Drupal core (without locale module as we know it now) would need to have as per the ubiquity of language definition and language assignment requirements discussed above. Please focus your powers helping with those issues or debate the need for them :)

@Jose: yeah! I'm not trying to address the whole problem at once, I'm trying to figure out the best way forward. In #1216094: We call too many things 'language', clean that up we could not agree if we'll support "languages" or "locales" in Drupal 8, so even renaming the "languages" table to "language" seemed like a "not sure yet" idea, since we might eventually want to support locales in relation to languages in some way (but we don't know how). However, to make it possible to decouple locale from language handling, #1260488: META: Introduce real language APIs was very active to add insert/update/delete support for languages with hooks for others to hook in (so we can decouple JS and page cache clearing from language addition once we separate the two), and we are busy in #1280530: Decouple domain/path negotiation setup from language configuration to decouple negotiation (domain/path) settings from language. We are getting closer to be able to move out language list config from locale with these, so I've opened #1301040: Move language listing functionality from locale.module to a new language.module and made it postponed on the ongoing decoupling/cleanup issues.

There is also no clear consensus above as to whether we should have negotiation in that module or in a separate module. I think once we remove all extraneus fields from language, it comes down to values we already have in code for a list of predefined languages, namely language code, name and direction. So we can build in a very simple select list to the base system for single language sites, so they don't need the language.module at all. In that case, we should probably integrate negotiation there, since if you have more languages, you are likely needing that (but honestly not need it in all cases). If we do need language.module for any foreign language site, then we maybe don't want to integrate negotiation, since single language sites will never need it. So these are the questions I've opened this issue for, while we work "in the background" to prepare locale for being split either way :)

Ok, here is more of a proposal to debate about then :) Instead of just documenting what we have, I've attached possible module names to the items.

1. Language assignment: clearly needs to be delegated to each module on their own. This is happening for path and node modules as linked above. Each module should be responsible for storage and assignment of language information on its data.

2. Single foreign language vs. multiple foreign languages. I've put lots of thought into this. Putting in a language configuration page to modify the value of 'language_default' variable we have now is *very* easy in system module. However, we stil need configuration as to whether we expose the Language neutral option for selection for example and what to apply languages to. If you then enable the ui translation module for example, it should be able to report its status for the language (and handle the system default fallback language too in a way). So all-in-all for consistency it seems like best to require language.module to be enabled for any site that wants to change the default language from English even if its a single foreign language site.

3. The ui translation portion of locale module should go to a dedicated module I named interface_translation in this figure. This reads naturally (vs. translation_interface). Debate welcome.

4. Translation.module in core is becoming generalized for entities, and becomes entity_translation.module (again, not translation_entity.module to read naturally).

5. Configuration translation goes to configuration_translation.module. Core currently only has this for date formats, and it has badly implemented solutions for language name translation and role translation, which are being eradicated. See #1301148: Stop pretending we have configuration translation for languages for the former.

6. This leaves us with language negotiation. As explored in detail above, this is only needed if you do have multiple languages AND you want it to actually change your UI language (ie. you have interface_translation.module too) or you want it to select from content/config languages (ie. you have one or both of entity_translation and configuration_translation). As discussed above, this is (in the majority of cases) not needed for single foreign language sites and not needed for a minority of multilingual sites either.

IMHO the two big questions are with (2) and (6). What do you think?

This figure is an extension of the above and can still be reached at https://docs.google.com/drawings/d/1jnabdm_uvEfvSMK55Bw6HSKZ1yRWmP9a_WsH... too.

Clarified figure a bit to avoid confusion in labels.

@good_man: please take some time to read up the above discussion, especially #29 and #33, and ask specific questions or dispute the points discussed there. The proposal is supported by those details and those could possibly answer all the questions you've asked.

Here is another view of what the plan is for modules in core and function distribution based on the above discussion and related plans in the initiative:

This is also a Google Drawing available at https://docs.google.com/drawings/d/1kTE8bX_ts16dCx35i4DRRhKfqz4Wi_BDsjee... for higher res viewing.

Please voice your concerns if any!

I get that people don't understand modules and would rather want to enable one huge module to have all functionality in a topic area. However, locale module is already a very confusing monster in terms of DX since it tries to implement a language list, language negotiation, language assignment on behalf of other modules, date (config) translation and UI translation. Add to this the whole l10n_update module, extended negotiation, entity_translation, and a whole new configuration translation layer and its going to be a monster you've never seen before (at least in core, but maybe even in contrib). Telling which hook is implemented for what sub-functionality would kill the DX of this module. That sounds like an entirely horrible scenario. If we truly want to expose this as one module for people, we can do sub-modules like field_ui has and hide them and enable all of them for the user when they enable the master module. I don't see how to make all hook implementations and UIs baked all into one module and not go insane on the way.

Also, we do want replaceable, interchangeable parts, right. So how do we do that with one monolithic module? Is hook_menu_alter() and trying to kill existing hook implementations for other hooks are our best to offer? Is that even possible with hook_hook_info_alter()? (I never tried). Or hook_menu() and all the hooks become pluggable classes in Drupal 8? Should I just assume that and go with a monolithic module? I'm trying to work best with the tools we have.

Here are some examples from the (7.x) locale.module code. Can you tell whether these belong to the (1) language management (2) negotiation configuration (3) interface translation (4) config (date) translation capability? Do you still think we should add even more sub-functionality modules to make this more confusing?

locale_help()
locale_menu()
locale_init()
locale_permission()
locale_form_alter()
locale_theme()
locale_language_types_info()
locale_modules_enabled()
locale_modules_disabled()
locale_modules_installed()
locale_css_alter()
locale_library_alter()
locale_block_info()
locale_form_comment_form_alter()

I mean we are already cramping the functionality of various "modules" inside locale module. Do we really want to mix up the permissions, menus, theme definitions, init hooks, css alters, whatever for all these modules into one huge pile of code? I can understand that those who don't have a good overview of the translation/language functionality don't see that there are really huge and not that much interrelated functionalities in question here. But the interface translation, entity translation and config translation stuff, all three, are very different and very non-related. They work off different data sources (text from code, fields in the DB (field storage), and nested config (XML) structures. They also work with different languages, interface text is provided in English by default, whereas content is almost never pre-provided and config is sometimes pre-provided. Config and content also can use different default languages per entity/config instance. Their interface is entirely different. Content translation is in-place (via local tasks/tabs), interface translation is out of place on its dedicated translation screen, and config translation we don't really know yet...

While these all try to solve different areas of the problem space, they are not one thing! It is not just going to be horrible developer experience to cramp these together to maintain it (who wants to work in excessively deep namespaces like language_interface_translation_*), but also for extensibility. If a hook_modules_installed() needs to stuff for interface and content translation, a module that wants to override either would need to kill the tangled code for both. Where is the Drupal 8 unofficial framework initiative goal here to untangle interrelated code and make things separately pluggable? I don't get this. Anybody but @plach agree with me?

One more thing. Last time I tried to just break at least the negotiation configuration out of locale.admin.inc, I was clearly wasting my time without support. I think until we have our hook system and our $modulename.admin.inc and $modulename.pages.inc patterns, it sounds like a pretty bad nightmare to try and cramp five module's functionality into one module and maintain both sane users and developers.

What is a module then anyway? I thought it was supposed to be an interchangeable unit of functionality covering a specific use case. I tried to spend my time extensively above to argue for the use cases and the user and developer need for "disablability" / interchangeability.

This came up again on the D8MI meeting with @webchick strongly in favor of keeping the modules together. So I decided to write up a short example with what I'm trying to avoid and see if @webchick or @Crell can show the standard D8 way they'd like to see it. Let's name our master module language for this example. Let's assume content and config translation are controlled about a master switch for simplicity.

function language_permission() {
  $perms = array(... some permissions);
  if (variable_get('language_ui_translation_enabled', FALSE)) {
    $perms += ....;
  }
  if (variable_get('language_content_and_config_translation_enabled', FALSE)) {
    $perms += ....;
  }
  return $perms;
}

function language_form_something_something_alter(&$form, $form_state) {
  if (variable_get('language_ui_translation_enabled', FALSE)) {
    $form['something'] = 'something';
  }
  if (variable_get('language_content_and_config_translation_enabled', FALSE) && some_other_thing_also_happens()) {
    $form['something_else'] = 'something_else';
  }
}

Is this the kind of ideal DX we want to pursue? Why is it better to have a checkbox to enable certain mega-features at an obscure place somewhere on the admin UI compared to actually enabling mega-features on the modules page? (Apart from our modules page sucking hard, which really should just be solved?)

This is bad DX because you never know if a function is related to just handling languages, or does UI translation stuff, or also does config or content translation stuff in cases. If you want to provide an alternate UI for your content translation let's say, which functions do you look at? You need to check all of them one by one if their code has any functionality referring to content translation. And no, pulling most of this logic to include files does not help. Where do conditionals for the implementation happen? Is that in the form_alters and hook_permissions? Then its basically the same code above, where I purposefully abbreviated what they actually do and just left the skeleton. If the logic happens in include files, then its even harder since you need to first track the include files. Then obviously we can introduce one more leve of hooks for these to separate them like so:

function language_permission() {
  $perms = module_invoke_all_type_of_call_for_object_methods('language_permission');
  return $perms;
}

Then those included objects would implement that method-hook thingy. We'd nicely replicate all hooks as method search&invokes, which to me sounds like a parallel hook system. Is this a "best practice" then for core rather then the "best practice" outlined above? Reimplement our choice of hooks one by one in custom ways?

Yes, I know our module page sucks. Yes, I know people don't get modules. We should fix those problems. Hacking around that with custom quirks like the above are things contrib modules did, yes. Are we adopting this practice in core for Drupal 8? Do you have better ideas?

I consider this hindering UX a lot as well, because as long as the DX is so confused of language support, people will not bother to be compatible with it and support this functionality in their modules and Drupal contribs will just not be multilingual still. The DX of what we do is very important to get right, because real meaty functionality exists in contrib that people want to use, and they will only consider our language support is anywhere near complete if contrib adopts what we work out.

How do you imagine a hook_permission() or hook_form_FORM_ID_alter() work in a scenario explained above and be good DX/UX when included all in one module? Also is our direction to reduce the number of modules and instead have settings at different custom places for huge chunks of functionality?

@webchick: thanks, I put in *lots* of time to consolidate the supporting arguments above, so it would be great to read your updated feedback after reading the above.

@webchick: care to follow up with an extended response? Its been a month... :)

@webchick: I think solution modules are really a pipe dream... Say you want a wiki. You need to understand content types, configure one with permissions, then add markdown input format, then add diff module, then add a bueditor or a WYSIWYG editor for markdown, then add talk pages and/or to display comments separately, etc. And really, wikis are very simple. But Drupal core and contrib has these lego bricks that you use. If you want an event calendar, voila, you have calendar module, but that needs date field, which needs understanding of fields, and also needs views to do the actual display and then some theming to make it look less gray on white.

All right, well, wikis and calendars are not simple. So now add on top that you want to make your wikis and calendars and forums and blogs, and ... multilingual. It is not even a side-by-side feature of the complexity of wikis or calendar, for which you already legoed your pieces together. Its a whole layer on top. Should it really be a goal to tuck everything together in one module to solve every problem and use case that might arise for multilingual sites, but should we go the way of the wikis and calendars and offer you building blocks so you use and configure the pieces you need (and can swap pieces in with replacements if you need to)?

Just like Drupal has no one switch solution for wikis and calendars, because we don't know what's your needs are, Drupal does not have a one switch solution for multilingual sites either. Let's say you want to rebuild lanyrd.com in Drupal. Its everybody's current favorite web 2.0 event calendar site. Is it multilingual? Well, I can see Hungarian events on there: http://lanyrd.com/2011/wudhu2011/schedule/, I see French events there: http://lanyrd.com/2011/scrumdayparis/coverage/. Do they have a Hungarian or French UI? No. Its not in their interest (at this point?) to have a multilingual UI (it is a lot of time and QA investment) but its in their interest to have all those events from all parts of the world, in all kinds of languages represented. I can easily imagine a classic DIWD "rebuild with Drupal" session that builds lanyrd.com in Drupal. It would need the language module but not any interface translation (or language negotiation for that matter).

Yes, this is not the 80% use case. The 80% use case is that people want to build wikis and event calendars and multilingual sites and Drupal does not have one stop solutions for them. People buy books like Using Drupal to understand how to build these solutions. Should we make it easier to build wikis, event calendars and multilingual sites? Yes, we should.

We discussed this in our multilingual UX planning meeting with Bojhan this week, and he posted a summary at http://groups.drupal.org/node/194493. In short, its already lots of steps to configure your site for multilingual and we should lead with (a) good defaults where possible (b) maybe a wizard type solution for the rest.

The D8MI initiative has a clear path for usability improvements, we made the installer language selection much simpler, removed confusing pieces of language configuration like the English/native name fields and moved path prefix and domain configuration where they make sense, so if you don't need them, you are not bothered. The D8 language config screen is leaps and bounds ahead of D7. We want to have default language negotiation settings, so that it has working settings out of the box instead of you needing to go in and do config to even make it work.

There are various aspects to take into account though. Usability and developer experience are two key goals to improve for D8MI. Having a language.module vs. a locale.module is in itself a usability improvement. We don't deal much with locales in Drupal, people I talked to at conferences often did not even know what it means or how to pronounce it. People helping with the D8 initiative meeting minutes regularly misspell it. (Introducing real locale specific functionality has its own issue at #1345758: META: Provide locale (regional) formats framework for automated translation of non textual data ). Separating concerns/modules is also a huge bonus for developer experience. If we have an easy to understand structure in our solutions, people will much more likely work with us.

I'm clearly seeing that separating modules could/would be a usability problem in the onboarding process for people building multilingual sites. Its the same problem that building wikis or event calendars in Drupal results in. But multilingual is not a smaller problem either, so I don't see an issue in applying Drupal practices to this as well and then help with the onboarding tools.

Just cycling back the discussion from our last IRC meetings earlier this week (http://groups.drupal.org/node/196963 with chat log). The current plan is that the negotiation part would go with the core language.module and not as its own module. The interface translation functionality (coming from locale and l10n_update) go to an interface translation module.

So the diff from http://drupal.org/node/1293304#comment-5145100 would be that negotiation does not exist as its own module in the plan but as part of language.module.

Tagging for base language system.

With negotiation being cleaned up to move in with language module and code implemented before in locale module on behalf of other modules is moving out to the respective modules, I think we can close this off (although the moves did not fully happen yet, the discussion results are being put into practice).

All right, now #1468930: Clean up and move the code for the negotiation functionality from locale module to language module is on!

We seem to be having the module set we wanted (minus config translation that will need to be a contrib module). Please weigh in at #1833184: Find a consistent naming scheme for translation-related modules on naming for the modules. :) Thanks!

Add English first class issue link.