Download & Extend

Unify language_list() and locale_language_list()

Project:Drupal core
Version:8.x-dev
Component:language system
Category:task
Priority:normal
Assigned:Unassigned
Status:closed (fixed)
Issue tags:API clean-up, D8MI, language-base

Issue Summary

There are two functions to get a list of languages in Drupal 6/7 (and currently in 8 too). This is pretty confusing. The difference between language_list() and locale_language_list() are as follows:

- language_list() in in bootstrap and therefore can be invoked anytime, no module dependency, locale_language_list() is in locale module
- language_list() returns an array of objects, locale_language_list() returns an array of property values
- language_list() always returns the whole list of languages, locale_language_list() returns the enabled ones by default and needs to be instructed to return the whole set

In short, the co-existence of these two functions is very confusing and should be fixed. To make that easy, I'm proposing we add an $only_enabled argument to language_list() which defaults to FALSE and can be set to TRUE to filter to enabled languages only. Then the return data in fact needs custom modifications at most places where it is used, so use the resulting list with whatever modifications are needed at places on top of that.

(The patch also simplifies locale_language_name() which is also about to be removed later in a followup).

Change records for this issue

AttachmentSizeStatusTest resultOperations
unify-language-list-functions.patch13.92 KBIdleFAILED: [[SimpleTest]]: [MySQL] 34,384 pass(es), 1 fail(s), and 0 exception(es).View details

Comments

#1

Status:needs review» needs work

The last submitted patch, unify-language-list-functions.patch, failed testing.

#2

Issue tags:+Less code

Just reviewed the code and couldn't find anything wrong with it.

Adding the coolest tag in Drupal, let's see if we can make it and still fix the test :)

#3

Status:needs work» needs review

The $only_enabled argument did not work due to $lang being used in the code instead of $language in language_list(). Changing to more standard $language as per the code standards should fix this proper.

AttachmentSizeStatusTest resultOperations
unify-language-list-functions-3.patch14.2 KBIdleFAILED: [[SimpleTest]]: [MySQL] 34,385 pass(es), 1 fail(s), and 0 exception(es).View details

#4

Status:needs review» needs work

The last submitted patch, unify-language-list-functions-3.patch, failed testing.

#5

Status:needs work» needs review

Need to diversify the caches in language_list() so it takes into account all arguments passed for how the return value should be produced. Therefore adding a specific cache $key there computed from the two arguments. Should now properly return the language list filtered when requested.

AttachmentSizeStatusTest resultOperations
unify-language-list-functions-5.patch14.54 KBIdlePASSED: [[SimpleTest]]: [MySQL] 34,383 pass(es).View details

#6

BTW this is barely less code I hope its easy to see it is a great unification :)

$ diffstat unify-language-list-functions-5.patch
includes/bootstrap.inc                 |   27 ++++++++++++++-------
modules/locale/locale.admin.inc        |   25 +++++++++-----------
modules/locale/locale.bulk.inc         |   37 +++++++++++++++++------------
modules/locale/locale.module           |   41 ++++++++-------------------------
modules/locale/locale.pages.inc        |   11 +++++---
modules/path/path.admin.inc            |    8 +++++-
modules/translation/translation.module |    3 --
7 files changed, 77 insertions(+), 75 deletions(-)

#7

Issue tags:-Less code

Remove tag accordingly.

#8

#5: unify-language-list-functions-5.patch queued for re-testing.

#9

@tstoeckler: what do you think of the current patch? Can you help review/test it?

#10

#5: unify-language-list-functions-5.patch queued for re-testing.

#11

Status:needs review» needs work

Here's that review. :)
I'll see if I can fix that and try it out tomorrow or so if no one beats me to it.

+++ b/core/includes/bootstrap.inc
@@ -2672,6 +2672,8 @@ function language_types() {
  * @param $field
  *   (optional) The field to index the list with.

@@ -2681,7 +2683,7 @@ function language_types() {
+function language_list($field = 'language', $only_enabled = FALSE) {

Is the fact that we're using 'language' here and not 'langcode' still a remnant of D7's total confusion of these terms, or is that actually meant to stay?

The $field argument is always 'language' in this patch. Also in locale_language_list() it was used differently, as you probably know (I just found out reading the patch... :) ). So do we really need this parameter? If you really need the languages keyed by name, you're still one foreach away... I didn't check, though, how language_list() is called outside of this patch, so that might be a bad suggestion.

And thirdly, regarding the default value of $only_enabled: It's always TRUE in this patch. Now, again, that might only be because of the scope of this patch, but I thought I'd at least ask if TRUE made more sense here.

+++ b/core/includes/bootstrap.inc
@@ -2701,20 +2703,27 @@ function language_list($field = 'language') {
+  // Produce an array indexed by the right field, filtered as requested. This
+  // will use keys for the static cache like language0, language1, weight0,
+  // weight1, etc. keeping the 'language' list the unfiltered master cache.

Isn't 'language0' then identical to 'language'?

+++ b/core/includes/bootstrap.inc
@@ -2701,20 +2703,27 @@ function language_list($field = 'language') {
+      // Some values should be collected into an array.
       if (in_array($field, array('enabled', 'weight'))) {
...
+        $languages[$key][$language->$field][$language->language] = $language;
       }
       else {
...
+        $languages[$key][$language->$field] = $language;
       }

I don't get that, the comment should be expanded, I think.

+++ b/core/modules/locale/locale.admin.inc
@@ -423,14 +423,13 @@ function locale_date_format_language_overview_page() {
+  $languages = language_list('language', TRUE);
...
+    '#markup' => check_plain($languages[$langcode]->name),

I might be wrong, but couldn't we just use language_load() here since it's a single language we want?
We could even inline that, if I'm not mistaken.

+++ b/core/modules/locale/locale.bulk.inc
@@ -13,22 +13,25 @@ include_once DRUPAL_ROOT . '/core/includes/gettext.inc';
+  $languages = language_list('language', TRUE);
+  $language_options = array();
+  foreach ($languages as $langcode => $language) {
+    if ($langcode != 'en' || locale_translate_english()) {
+      $language_options[$langcode] = $language->name;
+    }
   }
...
+    $language_options = array(
+      t('Already added languages') => $language_options,
       t('Languages not yet added') => language_admin_predefined_list()
     );

Wow, I stumbled on that one. Can we rename the first variable for clarity, maybe $added_languages or something?

+++ b/core/modules/locale/locale.module
@@ -757,37 +762,11 @@ function locale_get_plural($count, $langcode = NULL) {
+function locale_language_name($langcode) {

It seems this might be a useful function, and I really see no reason why it shouldn't live in language.module.
Since it doesn't seem to work (see below), it doesn't really seem to be used, so maybe just drop it?

+++ b/core/modules/locale/locale.module
@@ -757,37 +762,11 @@ function locale_get_plural($count, $langcode = NULL) {
+  $languages = language_list('language', TRUE);

Like above, couldn't this use language_load()?

+++ b/core/modules/locale/locale.module
@@ -757,37 +762,11 @@ function locale_get_plural($count, $langcode = NULL) {
+  return ($langcode && isset($list[$langcode])) ? $list[$langcode]->name : t('All');

That's not going to work, there's no $list anymore. :)

-5 days to next Drupal core point release.

#12

@tstoeckler: thanks for this detailed review!

1. NON-ISSUE: The language/langcode problem is being addressed in #1357918: Missing update for language_default in language langcode update. Its a 87k patch in itself (and is RTBC yay). Once that is committed first, this patch will (need to) be updated to reflect that change. For now, langauges use the language property for the language code.

2/3. DISCUSS / NEEDS WORK?: locale.module, system.module, etc. uses language_list() with the 'enabled' property instead of language code. Those are existing use of language_list() so we either need to support them or add copy-paste code for them all the places. I think its best to leave supported. Yes, $only_enabled is introduced in this patch because the base locale_langugae_list() usecase was to filter for enabled languages only. The existing uses of language_list() ask for the 'enabled' property and then use $return_value[1] to have the enabled languages only when needed. We can convert those places to use $only_enabled and if there are no other property requests left, we can leave off the property argument and leave $only_enabled the only argument.

4. NEEDS WORK: Yes, language0 will be the same as language. We can special case this to avoid duplicate storage inside the function.

5. NEEDS WORK: "Some values should be collected into an array." was an existing pattern but I agree it could use more comments and seen people ask about it before. Also, we might be able to get away without this (see 2/3, so maybe we should first explore that).

6. NEEDS WORK?: I like to use language_list() when we need data about multiple languages. I agree it makes sense to use language_load() when the langauge module is loaded and we only need one language.

7. NEEDS WORK?: Yes, the locale.bulk.inc use is a bit complex to support different scenarios. I'll look into how we can make that simpler.

8. FOLLOWUP: yes, locale_langauge_name() should go away. It is only module_invoke()d from node and path modules. Once we unify language_list() and locale_language_list(), that should be the only function to get rid of (or move to bootstrap.inc, to be discussed in a different issue). We should get this patch in first and then that is the only function left in locale.module that deals purely with language.

9. NEEDS WORK?: I'll look into whether it makes more sense to use language_load() there, thanks.

10. NEEDS WORK: woops, thanks for noticing, definitely needs fixing.

#13

Assigned to:Anonymous» Gábor Hojtsy

#14

Status:needs work» needs review

Ok, here is an updated core patch that takes into account your concerns.

- #1357918: Missing update for language_default in language langcode update got committed, so the patch stopped applying but 'language' on languages is now 'langcode', yay :) This solved one of your concerns.

- I've removed the $field argument on language_list(), making it only take $only_enabled. Found one place where it really needed both the disabled and enabled languages in separate lists, so had the code added there to separate them. All other places, where 'enabled' was used for key immediately turned to using the $languages[1] list only (which $only_enabled is there to cover). I also found only one place where there was another key used (besides 'enabled' or 'langcode'), and that was weight in the language detection system. Since the language list is already ordered by weight, and the language negotiation code actually had code to flatten the weight indexed array again (getting back what you'd have gotten from language_list() without the weight property requirement at the first place), removing the property request there actually made that code simpler. Heh.

- Found a previously unseen occasion of module_invoke() on locale_language_list() in node module that is now remedied.

- The language0 identical to language problem is now irrelevant given we don't have arbitrary keys. I did rename the cached lists to 'all' and 'enabled' though for clarity. These are the only two cached lists now.

- Used language_load() at more places where appropriate. I'm sure we'll find some more in the future, not sure we should cover that here. It is really a listing vs. loading function question and we are unifying two listing functions here, so let's keep that focus.

- Clarified the locale.bulk.inc nested select code/comment a lot.

- Fixed locale_language_name(). Again, to be removed/obsoleted in a later patch on the way to make path, node, etc. modules rely solely on language module for their language assignment capability.

Another review would be welcome. Thanks! (Yeah, the patch doubled in size unfortunately due to the changes related to loosing the property argument on language_list() but this makes the API much-much simpler especially that $only_enabled serves the 90% use case for the previous property argument).

AttachmentSizeStatusTest resultOperations
unify-language-list-functions-14.patch30.58 KBIdleFAILED: [[SimpleTest]]: [MySQL] 34,570 pass(es), 40 fail(s), and 356 exception(es).View details

#15

Status:needs review» needs work

The last submitted patch, unify-language-list-functions-14.patch, failed testing.

#16

The $grouped_languages array was not properly populated in translation_form_node_form_alter().

AttachmentSizeStatusTest resultOperations
unify-language-list-functions-16.patch30.59 KBIdlePASSED: [[SimpleTest]]: [MySQL] 34,606 pass(es).View details

#17

Status:needs work» needs review

#18

Super awesome!
That really does make the code a lot clearer.
I have run out of things to complain about.
Since I think I'm the only who's looked at this, I would like to read through this one more time later today before marking RTBC.

#19

Status:needs review» needs work

Well, I did find something. :)

+++ b/core/includes/bootstrap.inc
@@ -2664,53 +2664,45 @@ function language_types() {
+ *   (optional) Whether return only enabled languages.

Whether TO return...

+++ b/core/includes/bootstrap.inc
@@ -2664,53 +2664,45 @@ function language_types() {
+ *   An associative array, keyed by the language code, ordered by weight

Perhaps: an assoc. array OF LANGUAGES?

+++ b/core/modules/locale/locale.bulk.inc
@@ -11,24 +11,32 @@ include_once DRUPAL_ROOT . '/core/includes/gettext.inc';
+  // Initialize a language list to the ones available including English if

Missing comma: ...available, including...

Will fix this up now.

-10 days to next Drupal core point release.

#20

Status:needs work» needs review

Here's the re-roll.
I wanted to open a follow-up issue to remove locale_language_name(), but it's not used anywhere in core?! I did a grep and only the function definition turned up. So I ripped it out, let's see how that goes. :)

AttachmentSizeStatusTest resultOperations
unify-language-list-functions-20.patch30.41 KBIdlePASSED: [[SimpleTest]]: [MySQL] 34,612 pass(es).View details

#21

Assigned to:Gábor Hojtsy» Anonymous
Status:needs review» needs work

Well, if you grep for language_name, among very few other pieces, you'll find these:

core/modules/locale/locale.module:function locale_language_name($langcode) {
core/modules/node/node.admin.inc:      $value = $value == LANGUAGE_NONE ? t('Language neutral') : module_invoke('locale', 'language_name', $value);
core/modules/path/path.admin.inc:      $row['data']['language'] = module_invoke('locale', 'language_name', $data->language);

The removal of locale_language_name is required to make node and path modules independent of locale module for their language support, so that language.module fulfills its potential finally. However, it should/will be its own debate, due to how it attempts to be language independent, so it should not directly call language_load() and would be ugly to continue using module_invoke() for this anyway. Apparently that functionality does not have good test coverage, since you managed to keep tests pass even though you removed the function. Well. We'll need to add better test coverage when we remove that.

For this issue, let's keep our scope. The road forward is to make distinct, well contained and easily defined changes, so we don't get sidetracked trying to solve too much at once.

Looking for an update that keeps locale_language_name() there for now updated to language_list() as used above before your removal.

#22

Status:needs work» needs review

Ahh, sorry, you had mentioned that before. Yes, I'll open a new issue for that. *slapsforehead*

AttachmentSizeStatusTest resultOperations
unify-language-list-functions-22.patch30.56 KBIdleFAILED: [[SimpleTest]]: [MySQL] Unable to apply patch unify-language-list-functions-22.patch. Unable to apply patch. See the log in the details link for more information.View details

#23

Status:needs review» reviewed & tested by the community

Ok, all concerns seems to be fixed now, so this looks good to go IMHO. Important step in making language module an independent entity and also simplifying DX by removing parallels in the API and simplifying the existing API. The existing test coverage validates the changes as proven by above test failures when we did not change certain things in the patch.

#24

Issue tags:+Less code

I can't help myself and re-add my favorite tag in the world :)
Let's do this!

#25

lol, great tag :)

#26

Status:reviewed & tested by the community» needs review
Issue tags:-Less code, -Novice

As we're changing the API and function signature of language_list() as part of this issue:

Looking through the patch, almost all invocations of language_list() are passing the TRUE argument.

Therefore, I wonder why we return all languages including disabled by default? That's a bit unnatural, no? I'd compare it with a hypothetical module_list() that happens to return all modules, instead of only the enabled; i.e., the most common use-case.

I'm not sure whether we have a consistent pattern for a $include_disabled or $all arguments in core, but I'd assume the following should be OK:

* @param bool $include_disabled
*   (optional) Whether to return all languages, including disabled.
*   Defaults to FALSE.
*/
function language_list($include_disabled = FALSE) {

#27

I agree with sun on this one. We should always try to use arguments for the less common case.

#28

@sun: of course the patch has language_list() calls passing TRUE, since the locale_language_list() we are merging in had that default. However, in total, there are 55 invocations of language_list() that I found in Drupal 8 core (with the patch applied). Only 22 of them call it with only enabled, all others call it with all. Your suggestion would make this default to only enabled, so we'd have 33 places call it with an argument vs. 22 now. Of course we can do a piece by piece rundown of the 55 places it is called and investigate whether we need disabled or enabled languages there, and I hope to do that sometime. However, this patch is a cornerstone to move forward with making language module a standalone piece for language assignment, so it would be great to do the piecemeal semantic analysis in a followup. I'm merely unifying two functions here and applying the signature of an existing one on the code.

Also, the patch already doubled from 15k to 30k with the extension of scope to remove the property name argument since it was not used much, and that already involved existing language_list() dependent code, which was not really part of the goal here.

What do you think?

core/includes/language.inc:    $languages = language_list(TRUE);
core/includes/language.inc:    $fallback_candidates = array_keys(language_list());
core/includes/locale.inc:  $languages = language_list(TRUE);
core/includes/locale.inc:  $languages = language_list(TRUE);
core/includes/locale.inc:    $languages = language_list(TRUE);
core/includes/locale.inc:      $languages = language_list(TRUE);
core/includes/locale.inc:    $languages = language_list();
core/includes/locale.inc:    $languages = language_list();
core/modules/field/field.multilingual.inc:  return array_keys(language_list() + array(LANGUAGE_NONE => NULL));
core/modules/language/language.admin.inc:  $languages = language_list();
core/modules/language/language.admin.inc:  $languages = language_list();
core/modules/language/language.admin.inc:    $languages = language_list();
core/modules/language/language.admin.inc:  $languages = language_list();
core/modules/language/language.admin.inc:  $languages = language_list();
core/modules/language/language.admin.inc:  $languages = language_list();
core/modules/language/language.admin.inc:  $languages = language_list();
core/modules/language/language.module:  $languages = language_list();
core/modules/language/language.module:  $languages = language_list();
core/modules/language/language.test:    $enabled_languages = language_list(TRUE);
core/modules/language/language.test:    $enabled_languages = language_list(TRUE);
core/modules/locale/locale.admin.inc:  $languages = language_list(TRUE);
core/modules/locale/locale.admin.inc:  $languages = language_list(TRUE);
core/modules/locale/locale.admin.inc:  $languages = language_list(TRUE);
core/modules/locale/locale.bulk.inc:  $languages = language_list(TRUE);
core/modules/locale/locale.bulk.inc:    $languages = language_list();
core/modules/locale/locale.bulk.inc:  $languages = language_list(TRUE);
core/modules/locale/locale.bulk.inc:    $languages = language_list();
core/modules/locale/locale.bulk.inc:    $langcodes = array_keys(language_list());
core/modules/locale/locale.install:  $languages = language_list();
core/modules/locale/locale.module:  $languages = language_list(TRUE);
core/modules/locale/locale.module:    $languages = language_list(TRUE);
core/modules/locale/locale.module:  $languages = language_list(TRUE);
core/modules/locale/locale.pages.inc:  $languages = language_list();
core/modules/locale/locale.pages.inc:  $languages = language_list(TRUE);
core/modules/locale/locale.pages.inc:  $languages = language_list();
core/modules/locale/locale.test:    $languages = language_list(TRUE);
core/modules/locale/locale.test:    $languages = language_list();
core/modules/locale/locale.test:    $languages = language_list();
core/modules/locale/locale.test:    foreach (language_list() as $node_langcode => $node_language) {
core/modules/locale/locale.test:      foreach (language_list() as $langcode => $language) {
core/modules/node/node.admin.inc:    $languages = language_list(TRUE);
core/modules/node/node.admin.inc:  $languages = language_list();
core/modules/node/node.module:    foreach (language_list() as $key => $entity) {
core/modules/openid/openid.module:      $enabled_languages = language_list(TRUE);
core/modules/path/path.admin.inc:    $languages = language_list(TRUE);
core/modules/path/path.test:    $languages = language_list();
core/modules/system/system.module:  $languages = language_list(TRUE);
core/modules/translation/translation.module:    $languages = language_list();
core/modules/translation/translation.module:    $languages = language_list(TRUE);
core/modules/translation/translation.module:    $language_list = language_list();
core/modules/translation/translation.pages.inc:  foreach (language_list() as $langcode => $language) {
core/modules/translation/translation.test:    $languages = language_list();
core/modules/translation/translation.test:      $languages = language_list();
core/modules/translation/translation.test:    $languages = language_list();
core/modules/user/user.module:  $language_list = language_list();

#29

Status:needs review» reviewed & tested by the community

I do think the default should be switched, since modules should not attempt to work on disabled configuration in the first place. But I'm perfectly fine with a separate follow-up issue.

#30

#31

I reviewed this patch and it looks RTBC. It may need a re-roll though. Asking the testbot to give it a run.

#32

#33

Status:reviewed & tested by the community» needs work

The last submitted patch, unify-language-list-functions-22.patch, failed testing.

#34

Status:needs work» needs review

Rerolled, so should apply again. Let's see if it still passes all tests as well.

AttachmentSizeStatusTest resultOperations
unify-language-list-functions-34.patch30.56 KBIdlePASSED: [[SimpleTest]]: [MySQL] 33,303 pass(es).View details

#35

Status:needs review» reviewed & tested by the community

#36

Status:reviewed & tested by the community» fixed

Thanks for the re-roll. Committed to 8.x. :-)

#37

Added changelog entry at http://drupal.org/node/1414264

#38

@tstoeckler: the followup issue for locale_langauge_name() is on! #1414314: Make node and path depend on language module only for language support, get rid of locale_language_name. That should be a great boost for delivering on the promises of language.module being the gate for langauge assignment support, so after that path and node modules would let you assign language and they would display and use language information and all without locale module ever being enabled :)

#39

Issue tags:+language-base

Tagging for base language system.

#40

Status:fixed» closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.