Problem/Motivation
As per #1216094: We call too many things 'language', clean that up we are converting schemas and parallel APIs to use 'langcode' instead of 'language' where a language code is involved. The use of 'language' inconsistently for language objects and language codes in Drupal 7 keeps throwing people off the cliff and makes understanding the API much harder.
This issue is about field data storage needing the same conversion to move from language to langcode. Will need updates and upgrade tests but first of all it would be a well focused search and replace wherever field language is involved. The tricky part is that field language maps to generated schema pieces in the database for example, so is used in schemas under all kinds of dynamic names.
Proposed resolution
Rename all occurences of language into langcode
Remaining tasks
Notify devel : #1292294: Respect field translatability when generating content, otherwise it gets lost
API changes
Use 'langcode' instead of 'language' for field queries.
Original report by Gábor Hojtsy
As per #1216094: We call too many things 'language', clean that up we are converting schemas and parallel APIs to use 'langcode' instead of 'language' where a language code is involved. The use of 'language' inconsistently for language objects and language codes in Drupal 7 keeps throwing people off the cliff and makes understanding the API much harder.
This issue is about field data storage needing the same conversion to move from language to langcode. Will need updates and upgrade tests but first of all it would be a well focused search and replace wherever field language is involved. The tricky part is that field language maps to generated schema pieces in the database for example, so is used in schemas under all kinds of dynamic names.
Comments
Comment #1
clemens.tolboomYou can use
to check for language.
http://api.drupal.org/api/drupal/core!includes!bootstrap.inc/function/dr...
Using patch drush: #1396178: Add properties output as a --format and
gives a nice list
comment.fields.language.type = varchar
field_data_comment_body.fields.language.type = varchar
field_revision_comment_body.fields.language.type = varchar
field_data_body.fields.language.type = varchar
field_revision_body.fields.language.type = varchar
field_data_field_tags.fields.language.type = varchar
field_revision_field_tags.fields.language.type = varchar
field_data_field_image.fields.language.type = varchar
field_revision_field_image.fields.language.type = varchar
node.fields.language.type = varchar
date_format_locale.fields.language.type = varchar
users.fields.language.type = varchar
Comment #2
clemens.tolboomComment #3
clemens.tolboomI start working on alpha order
which leaves out
which are covered by other issues.
Comment #4
clemens.tolboomFirst stab @ field_sql_storage , field.multilingual and locale.module. Let's see how much dies.
Comment #5
gábor hojtsyLooks good, just one minor comment:
Let's say "language code" when in 'docs', not langcode :)
Comment #6
clemens.tolboomFixed the text from #5
Another stab ... changed all(?) code occurences
'language' and ->languagefor field and field_uiThere are a lot of langcode related strings and function names still left for change.
[edit]
Following modules use field alteration of some kind
Comment #7
clemens.tolboomComment #9
clemens.tolboomFailure of TextTranslationTestCase was my bad ... that test case is test a node/add/article
Failure of LanguageUpgradePathTestCase needs some work as the column is changed from language to langcode.
Same goes for FilledUpgradePathTestCase
Failure of EntityFieldQueryTestCase was my bad ...
I expect now only failure of both upgrade path test.
Next step is to http://drupal.org/node/1429136 (UpgradePath test cases)
Comment #11
clemens.tolboomHmmm ... EntityFieldQueryTestCase still fails :(
One down ... three tests to fix.
Comment #12
clemens.tolboomI failed so solve EntityFieldQueryTestCase completely (6 out of 8 are solved)
Checking the documentation for these test give me no clue :(
We still need a
function field_update_8000() {to change the language column into a langcode column.Comment #13
clemens.tolboomI'm puzzled how to alter the schema for this unknown amount of fields having a language column. A code example would be welcome ;)
This needs API Change record for contrib modules to use langcode instead of language.
Comment #13.0
clemens.tolboomUpdated issue summary.
Comment #15
fagoI think we can do that upgrade path for the mysql field storage engine only. Afaik amitai came up with some working code for that over at #1319040: Remove "target_type" column from db (d7) in entity reference, which could serve as guidance. I don't think we should rely upon field_info_fields() though, but better use low-level functions for retrieving info about existing fields.
Comment #16
clemens.tolboom@fago tnx
What about other storage engine? That sounds like a D8 release blocker is it?
Not using field_info_fields was what I thought too.
The link #1319040: Remove "target_type" column from db patch from #37 was useful.
I guess we could use code like field_read_fields without invoking modules.
I give it a try this week(end).
Comment #17
clemens.tolboomI added a
function field_sql_storage_update_8000(&$sandbox) {That seems to be the best place.@fago
I'm still concerned with "We only upgrade mysql". Please elaborate on that.
Let's see what the test bot says.
Comment #19
clemens.tolboomNice ... compared to #12 we have 2 exceptions down.
The failures are in
which reads garble to me. The code that is :(
So some guides needed still.
Comment #20
clemens.tolboomThese tests needs way more info.
and
Comment #21
clemens.tolboomI reverted the language_group arguments back to 'language'
Added extra debug info on assertEntityFieldQuery failure.
The 2 failures are still here so this needs help :(
Comment #23
clemens.tolboom#21: field-language-rename-1439692-21.patch queued for re-testing.
Comment #24
gábor hojtsy@chx pointed out http://api.drupal.org/api/drupal/modules%21field%21modules%21field_sql_s... which loops through the conditions, and it looks up 'language_group' by concatenating the foreach value before '_group'. Since you renamed that to 'langcode' but not the key itself, all you got was a missing key, so no good results. If we rename 'langugae_group' to 'langcode_group' too, we get those tests pass again, since they will find their values :)
Attached the interdiff as well. For EFQ to work properly, the first part of the last line change was important. The others I've renamed for consistency.
Comment #25
plach@clemens.tolboom:
AFAIK we support other storage engines only on the API level, we don't account for them in update functions as it would be nearly impossible to avoid hook invocations, which in this context are BAD!
Actually also the display language is a display language code :)
I guess
$languagesshould become$langcodes.$languageshould be$langcode. In this light$available_languagesshould become$available_langcodes.I think we want
_update_7000_field_read_fields()here to avoid hook invocations."langcode conditions": do we really want to go this far?
26 days to next Drupal core point release.
Comment #26
clemens.tolboomI've applied the suggestion from @plach except the hook_update_8000 as that lies outside my comfort zone right now.
I also changed text by changing language when langcode is meant afaik. This needs a review.
See inter-diff.
Comment #27
clemens.tolboomComment #28
plachComments do not wrap at column 80.
This should be
field_available_languages(). I'm wondering how the hell is possible for this code not to fail tests.Here and below
field_languagecodesshould befield_langcodes.'is' shoud go one line up: comments should wrap as close as possible to column 80.
As I wrote previously
field_read_fieldsshould be safely replaceable with a_update_7000_field_read_fields.Missing trailing dots.
languagecodes -> langcodes
$languages -> $langcodes
$all_languagecodes -> $all_langcodes
field_langcode
fallback_language -> fallback_langcode
21 days to next Drupal core point release.
Comment #29
dawehnerThis is indeed interesting. There seems to be a problem, maybe the tests are not completed.
Here is a new patch which reroles the patch and fixes the issues from above.
Comment #31
dawehnerThere is another patch for the testbot.
Comment #32
das-peter commentedReviewing...
Comment #33
clemens.tolboomOpen points still stainding according franskuipers and clemens.tolboom
We should delete this function.
This is a Drupal 7 function which we should not use.
Is this ok to use?
Is this hook-less?
Comment #34
das-peter commentedTo test the upgrade path I extended the node type "Article" to have two instances (single & multivalue) of every core field type and generated 50 nodes with content.
The only issue I cam across was the fact that the scheme wasn't rebuild after the update and thus the site had a WSDO until a full cache flush was done.
Adding a
drupal_get_schema(NULL, TRUE);after the schema update solves this issue.Comment #35
clemens.tolboomWe (FransKuipers and me) rewrote field_sql_storage_update_8000 which is now without hook invocation.
Please review :)
We commented out
as it's unclear to us _AND_ it involves invoking hook_schema.
Comment #37
clemens.tolboomWe fixed the _revision primary key.
Comment #38
das-peter commentedTested the patch again and this time the explicit rebuild of the schema cache wasn't necessary :)
Thus I removed the lines I added earlier.
Comment #39
dawehnerJust a leftover debug statement
I'm wondering whether it's intended to rename the hooks and some other functions as well.
Just from the perspective of a module developer i would though preper the current version.
I read the full big patch and didn't saw any problem with the renaming of language to langcode.
Comment #40
clemens.tolboom@dawehner good catch for the debug.
But are we changing the hooks? We only changed the hook parameter names from language* to langcode* right? Or do I miss something? One could argument the hook names should be changed too.
Thanks for the checks :)
Comment #41
plachI don't think we want to change also hooks and function names, but we should hear Gabor's voice on this.
Comment #42
gábor hojtsyI definitely replied in IRC that I think it should not be part of this issue, if ever :) We should debate that elsewhere definitely. This is about the schema and its update, lets try to focus on that.
Comment #43
das-peter commentedAttention: Only the name of the variables, used for hook invocation, are changed. There's no such thing as a renamed hook.
As this debug call is conditional it might make even sense to keep it (at least it doesn't seem to harm a thing), but I don't know what the rules for such constructs are.
Comment #44
clemens.tolboom@das-peter: let's keep the debug then.
@Gábor Hojtsy: Who must review this to get this RTBC?
Comment #45
gábor hojtsy@Clemens: I don't have a specific list! @fubhy, @plach, @dawhenever, @Clemens (yes), etc. were very great reviewers so far, many people worked on this patch, so cross-reviewing with them seems like would be great in itself. It is not about new architecture or shiny new stuff, it is about DX improvements, so I think its fine if we cross-review.
Comment #46
clemens.tolboom@plach could you please review the last hurdle mentioned by you in #25 + #28 and resolved similar to #35 in patch #37 concerning
I think #38 is then RTBC :)
(as we keep the debug and don't change hooks #39 - #44 )
Comment #47
plachI fixed some language occurence left. I also rewrote the update function to use the (supposedly?) proper update API function. We really need @catch to have a look to it sooner or later. Maybe we should assign this to him when this is RTBC (and now IMO it is).
$language => $langcode
$langcode_suggestion
Wrong comment wrapping.
Here and below $requested_langcodes should be singular, as only a single language is requested.
If we want to make this change, this should be plural here and below.
-4 days to next Drupal core point release.
Comment #49
plachWeird, I manually tested the upgrade path and it completed nicely...
Comment #50
plachHmm, here is the culprit :)
Attached a working patch plus interdiff with #38.
-4 days to next Drupal core point release.
Comment #51
plachGábor Hojtsy:
Since the only part that still needs review is the update function and no one here seems to be comfortable with reviewing it, what about marking this RTBC and assigning it to @catch so he can provide us any feedback about possible pending concerns?
Comment #52
gábor hojtsyLooks like the question is down to whether we want _field_sql_storage_revision_tablename() duplicated for the update and whether _update_7000_field_read_fields() should be renamed?
Comment #53
clemens.tolboomThis function (name) may never be used for D8. AFAIK it is a helper function for D6 to D7 upgrades.
That's why I used the db_query.
Should this be langcode as well?
'If we have a langcode suggestion and the suggested langcode is available,'
Good catch over the plural form.
I guess $display_langcodes should now also singular instead of plural.
Comment #54
clemens.tolboomComment #55
gábor hojtsy@clemens: yes, that is why I'm saying if we want to use the _update_7000_field_sql_storage_write() function in the D8 upgrade, we'd want to rename it maybe to _update_8000_field_sql_storage_write() or somesuch?
Comment #56
clemens.tolboom@Gábor Hojtsy: He ... we past each others comment :)
It could be good to transform that function but should we do it for this issue? I somehow guess yes.
If we do that it needs a test as I think its data column contains values also available through the individual columns. Not 100% sure.
Comment #57
plachFixed the language suggestion comment and another 'language' occurrence.
I recall from the deadly #1164852: Inconsistencies in field language handling that it's absolutely discouraged to directly query
field_config, since we cannot rely on it being always there. I really think using _update_7000_field_read_fields()is the right thing to do here, since it encapsulates the field retrieval logic, and no, I don't think renaming it from 7000 to 8000 belongs to an issue dealing with language code renamings. My point is we should really ask someone knowledgeable, as @catch.Comment #58
clemens.tolboom@plach: as you put it that way we certainly have to use a function like that :)
So we agree this function should be reused by renaming it to _update_8000_field_read_fields or maybe a better name as @Gábor Hojtsy suggested too.
What about adding hookless into it's name and make it public. It is now used only by hook_update_8000 but is that it's only use case?
It is similar to http://api.drupal.org/api/drupal/core!modules!field!field.crud.inc/funct... right?
Comment #59
plachSounds as good plan but for the life of me I don't get why we have to discuss it here, however the update prefix means exactly hookless. I don't think we need a new terminology.
Comment #60
clemens.tolboom@plach : You just want to use the function whatever it's name. Sounds like a brilliant plan :) ... or I completely misunderstood.
@Gábor Hojtsy guess we should follow along @plach suggestion from #51 and mark this as RTBC and assign to catch.
Comment #61
plach(OT)
@clemens.tolboom:
Trying to explain myself better: having a better name for that function, making clear it's an API function pertaining the upgrade path sounds like a good plan. I don't think the hookless terminology is something we need to introduce since we already have the "update_*" prefix.
Comment #62
gábor hojtsy_update_7000_field_read_fields() would be easy to rename, it is not invoked ever in Drupal 8 as far as my grep shows. I think it was merely missed from being removed when the D6 -> D7 update functions were removed. I'd say that it is the task of patches introducing update path use of functions to make sure those are available properly provided as pure update functions. Therefore I'd at least rename this function to _update_8000_....().
Then there is _field_sql_storage_revision_tablename() which is a 6 line function with a very simple condition. Whether we should keep using that is a good question really. Safest is to duplicate that too as an update function. That would not hurt. Either way, once the _update_7000_...() function is renamed, I'm comfortable marking this RTBC in any case. I can easily predict @catch would ask for the 7000 rename :)
Comment #63
plachThere is a bunch of _7000 functions in field.install, not mentioning another one in node.install and one in field_sql_storage.install. None of them is used, should we rename them all for consistency?
Comment #64
gábor hojtsy@plach: I think *that* could be for another issue. Patches are responsible for their own update stuff :)
Comment #65
plachHere it is.
Comment #66
gábor hojtsyI think that looks good to go now.
Comment #67
dries commentedCommitted to 8.x. Thanks!
Comment #68
plach@catch:
Would you please confirm
field_sql_storage_update_8000()is ok?Comment #69
catchNo it's not quite right, there's no reason to do this since the schema for field config hasn't changed, those functions are tied to schema version, not API version.
See also #1134088: Properly document update-api functions.
Comment #70
clemens.tolboom@catch so we shouldn't have renamed the _update_7000_field_read_fields ?
Leave it 'as is' as the schema we are querying is still a D7 schema? Makes sense.
So all _7000_ functions are meant to work on a D7 scheme in a D8 context.
(I've read and commented on #1134088: Properly document update-api functions)
Comment #71
gábor hojtsy@catch: all the update functions, quite a few of which work with the Drupal 7 schema state are named _800* too. The previous examples in #1134088: Properly document update-api functions, before your latest example today at http://drupal.org/node/1134088#comment-5849018 showed update helper functions in the _8000 space used in update functions in the 8000 space. Your latest example makes the differentiation better, and makes it clear that we should name this back. Thanks for that.
Comment #72
catch@ Gábor Hojtsy: yes anything named 8000 that's not specifically related to a Drupal 8 schema change is named incorrectly, that's discussed in #1239370: Module updates of new modules in core are not executed which is another pile of evil :(
I've committed/pushed the follow-up, thanks!
Comment #73
gábor hojtsyWhile attempting to write a change notice node for this, I found that we updated the schema handling in _update_7000_field_sql_storage_write(), but did not rename that function. As per the above discussion, that should have been renamed. Verified that with @catch in IRC. This function is not invoked anywhere, so there is no use to keep a D7 specific version around. Later update functions would invoke the D8 version (and set an update function dependency on field_sql_storage_update_8000().
Verified that this is not invoked via http://api.drupal.org/api/drupal/core%21modules%21field%21modules%21fiel... and looking for invocations in the committed patch too.
Comment #74
catchOK, committed that one too.
Comment #75
gábor hojtsyOk, wrote up an initial version of a change notice node at http://drupal.org/node/1525236
Comment #76
gábor hojtsyRemoving sprint tag too.
Comment #77.0
(not verified) commentedUpdated issue summary.