Problem/Motivation

Follow-up of #2284111: Exceptions caught and printed from index.php should not return a 200 status code.

basic_auth loads a full user entity and sets that as current user. That results in a circular recursion due to LocaleLookup trying to get the current user roles and t() calls while fetching the field definitions to load them.

This is what happens:

-> t()
  -> LocaleTranslation::getStringTranslation
  -> new LocaleLookup ...
    -> getRoles()
    -> UserRole::baseFieldDefinitions
    -> t()
      -> new LocaleLookup
        ->getRoles()
          ->...

Proposed resolution

Three layers of defense:

1. @dawehner came up with a way to lazy-load the cache ID within lazyLoadCache(). Because that sets the already-loaded flag before actually trying to load, any following calls will stop there. They will then result in cache miss lookups and resulting cache writes, but at least we can prevent a fatal error/loop.

2. Then, we add TranslationWrapper's to the few known current t() calls during early bootstrap. That prevents those t() calls for core/current HEAD.

3. To prevent the cache writes, we reset the persist list after building the cache ID. Then the worst thing that can happen if someone adds more super-early t() calls is a few uncached translation lookup queries.

With that, I feel comfortable with this as a solution to this problem.

Remaining tasks

Can we test this somehow? (the cache misses)

There is also an issue that wants to look into loading the user entity for the session backend too, to get rid of the hardcoded queries there, now that we have the entity cache. This is a hard blocker for trying that. We might encounter more t() calls there, we can fix them there.
Follow-up task in #1751274: Do not query user_roles directly, not to be done here.

User interface changes

API changes

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Berdir’s picture

Status: Active » Needs review

Status: Needs review » Needs work

The last submitted patch, circular-dependency-2284111-9.patch, failed testing.

bbinkovitz’s picture

I'm trying to reproduce this bug so I can test the patch but I'm not getting it. I've installed from the latest Drupal 8 dev with the language set to German and then I followed the instructions on https://www.drupal.org/documentation/modules/basic_auth which culminate with "try various routes like router_test/test11". When I try that route, I don't get any indication of a circular recursion. I just get an "Access Denied" message, which I believe is what one would expect?

Status: Needs work » Needs review
Berdir’s picture

I can certainly still reproduce this when installing basic auth, making de the default language and then going to a rest route.

Adding random new TranslationWrapper()'s instead of t()'s seems to be a start in fixing this, but ugly, as mentioned.

Status: Needs review » Needs work

The last submitted patch, circular-dependency-2284111-9.patch, failed testing.

The last submitted patch, 5: circular-dependency-2284111-5.patch, failed testing.

The last submitted patch, 5: circular-dependency-2284111-5-test-only.patch, failed testing.

Berdir’s picture

Status: Needs work » Needs review
FileSize
17.55 KB
1.86 KB

Making sure that DataDefinition and EntityType return strings for their labels and descriptions.

The test-only test did fail as expected (out of memory due to endless loop)

dawehner’s picture

@berdir
It seems not so nice if you have to create TranslationWrappers for your own all over the place. One simple way could be to introduce \Drupal::t() which creates these objects?

+++ b/core/lib/Drupal/Core/Entity/Annotation/ConfigEntityType.php
@@ -36,7 +37,7 @@ class ConfigEntityType extends EntityType {
+    $this->definition['group_label'] = New TranslationWrapper('Configuration', array(), array('context' => 'Entity type group'));

Do we write code in german now ;) Let's lowercase "New".

Berdir’s picture

"seems not so nice". I'd say it's ugly as **** :)

I was just trying to check if this is enough or if we need something more. I don't know what a good API would look like.

If we wouldn't have the ever-recurring problem of having to be able to identify translatable strings, we could just say that DataDefinition will take care of translating the title/description and you can simply pass in the string.

dawehner’s picture

Another idea I had was to pass along some kind of object to baseFieldDefinition

Gábor Hojtsy’s picture

Issue tags: +D8MI, +language-content

Was this happening due to the nonsensical logic in #2240459: Applying default language fallback rules to interface translation has impressively bad results that was fixed since then? Or still happening? Will send the test only patch for retest.

Gábor Hojtsy’s picture

Berdir says still a problem.

[5:35pm] berdir: GaborHojtsy: so when you call t() the first time, it needs to get the roles for the currently logged in user. and if that is an entity (and not a plain object like CurrentUser), that needs to load field/property definitions to get that. and those call t() => loop
[5:35pm] GaborHojtsy: berdir: hm, not clear why that is a loop, the t()s should return?
[5:35pm] GaborHojtsy: berdir: they are not looking at the same thing to translate?
[5:36pm] berdir: GaborHojtsy: LocaleLookup::__construct()
[5:37pm] berdir: GaborHojtsy: the first call to t() will call that, that calls t() somewhere down in getRoles(), then that calls the constructor again because it has not yet been initialized, then we call getRoles() again...

Needs to be updated in the summary.

dawehner’s picture

FileSize
4.42 KB

[5:37pm] berdir: GaborHojtsy: the first call to t() will call that, that calls t() somewhere down in getRoles(), then that calls the constructor again because it has not yet been initialized, then we call getRoles() again...

Okay, so as always, it is a bad idea to execute any code beside $this->foo = $foo; in the constructor.
What about deferring the call a bit. I haven't actually tried whether the approach works, but could be interesting.

The last submitted patch, 5: circular-dependency-2284111-5-test-only.patch, failed testing.

Berdir’s picture

That will not change anything. I thought about this before and I think there is a cache collector where we do this, although I would have to check which one it is. However, this isn't something that we happen to just initialize early... We actually *use* it early, and while I agree that it would be cleaner to not do this in __construct(), it will not change anything for this issue.

Combining your patch with my test should prove that.

dawehner’s picture

FileSize
5.84 KB

So that approach is at least green.

So the thing is that now the t() call goes like this:

-> t()
  -> LocaleTranslation::getStringTranslation
  -> new LocaleLookup ...
  -> $translation = $this->translations[$langcode][$context]->get($string);
    -> getCid()
    -> UserRole::baseFieldDefinitions
      -> t()
      -> LocaleTranslation::getStringTranslation
      -> $this->translations[$langcode][$context]->get($string)
      -> return the base field title

So as you see the deferring helps for the next call.

In HEAD it works like the following:

-> t()
  -> LocaleTranslation::getStringTranslation
  -> new LocaleLookup ...
    -> UserRole::baseFieldDefinitions
    -> t()
      -> new LocaleLookup
        ->
          ->
Berdir’s picture

We discussed this, and the reason the patch works is that lazyLoadCache() sets the flag about being loaded before loading before we actually load it. Then the next call assumes that the cache was loaded, looks if it is in the cache, and because it is (obviously) not. That calls resolveCacheMiss(), which loads from the database.

That's not too bad, but the real problem is that also marks that string to be written into the cache. So on every request, we'd write the cache again, which also involves a lock. That's not pretty. But at least this prevents the fatal error/loop.

However, we debugged this a bit and we only found two/three calls that happen that early. That's those default languages in LanguageManager that currently trigger the initial t() call and that's StringItem::propertyDefinitions(), which is called from FieldItemBase::__construct() for the roles field item. So we can add TranslationWrappers for those, so we prevent the cache write problem on HEAD with basic authentication.

Finally, to prevent the cache write problem, we can clear the persist list after getting the cache ID. Then the worst thing that can happen if someone adds more super-early t() calls is a few uncached translation lookup queries.

Berdir’s picture

dawehner’s picture

Great!

+++ b/core/lib/Drupal/Core/Entity/Annotation/ConfigEntityType.php
@@ -36,7 +37,7 @@ class ConfigEntityType extends EntityType {
-    $this->definition['group_label'] = $this->t('Configuration', array(), array('context' => 'Entity type group'));
+    $this->definition['group_label'] = new TranslationWrapper('Configuration', array(), array('context' => 'Entity type group'));

+++ b/core/lib/Drupal/Core/Entity/Annotation/ContentEntityType.php
@@ -36,7 +37,7 @@ class ContentEntityType extends EntityType {
-    $this->definition['group_label'] = $this->t('Content', array(), array('context' => 'Entity type group'));
+    $this->definition['group_label'] = new TranslationWrapper('Content', array(), array('context' => 'Entity type group'));

Should we explicit document these cases and why we do this? PS: Does potx has support for this already?

Status: Needs review » Needs work

The last submitted patch, 20: 2288123-20.patch, failed testing.

Berdir’s picture

Ouch, that looks like we're somehow writing those language types out into config and fail to deal with TranslationWrapper there?

@dawehner: Honestly, I just updated those to be consistent with everything else in entity annotations. Right now, they would end up being cached translated as removed the language from the cache key. They're not strictly related to this issue, should work without that.

Gábor Hojtsy’s picture

@dawehner: yes, potx supports TranslationWrapper.

@berdir: we have language.entity.und and language.entity.zxx already pre-shipped with the language module, we don't write them from the code modified here. Or you referred to something else that is written out into config?

Berdir’s picture

Status: Needs work » Needs review
FileSize
12.01 KB
785 bytes

Weird, it's just that test that's trying to write those languages, so I used installConfig() there. I haven't been able to figure out why exactly we are doing it, all tests pass fine without that logic. But I don't want to change something there, so switching to the installConfig() call.

dawehner’s picture

+++ b/core/modules/system/src/Tests/Entity/EntityLanguageTestBase.php
@@ -106,11 +106,7 @@ protected function setUp() {
+    $this->installConfig(array('language'));

So this would also include language.(negotation, mappings and types).yml
do we really want those as well? I guess it is not problematic at all.

Berdir’s picture

I don't think that is a problem. In a real installation, that default config exists too.

dawehner’s picture

Me neither now :)

Gábor Hojtsy’s picture

The fix looks as good as it can get I guess :) This is a tricky circular dependency.

+++ b/core/lib/Drupal/Core/Entity/Annotation/ConfigEntityType.php
@@ -36,7 +37,7 @@ class ConfigEntityType extends EntityType {
-    $this->definition['group_label'] = $this->t('Configuration', array(), array('context' => 'Entity type group'));
+    $this->definition['group_label'] = new TranslationWrapper('Configuration', array(), array('context' => 'Entity type group'));

+++ b/core/lib/Drupal/Core/Entity/Annotation/ContentEntityType.php
@@ -36,7 +37,7 @@ class ContentEntityType extends EntityType {
-    $this->definition['group_label'] = $this->t('Content', array(), array('context' => 'Entity type group'));
+    $this->definition['group_label'] = new TranslationWrapper('Content', array(), array('context' => 'Entity type group'));

+++ b/core/lib/Drupal/Core/Field/Plugin/Field/FieldType/StringItem.php
@@ -39,7 +40,7 @@ public static function defaultSettings() {
-      ->setLabel(t('Text value'));
+      ->setLabel(new TranslationWrapper('Text value'));

+++ b/core/lib/Drupal/Core/Language/LanguageManager.php
@@ -197,13 +198,13 @@ public function getDefaultLockedLanguages($weight = 0) {
-      'name' => $this->t('Not specified'),
+      'name' => new TranslationWrapper('Not specified'),
...
-      'name' => $this->t('Not applicable'),
+      'name' => new TranslationWrapper('Not applicable'),

Is there some hard rule to where should we use TranslationWrapper? I mean people may be puzzled that one field type uses it while others don't. IMHO we should explain it with comments or do some other kind of long term knowledge preservation.

////

Also do we want to pursue any of the remaining tasks in this issue? I think not, but then would be good to say it there. That section is to be used for things still outstanding for commit on an issue.

Berdir’s picture

Issue summary: View changes

Thanks, yes adding comments makes sense, I will add them to Language Manager and StringItem. The group label changes are IMHO just a logical thing to do as all other annotation labels are tranlation wrappers too. But I guess I could add that as a comment.

There is not really a hard rule, that was what I was worried about it my patches. That adding another call would make it fail again. We're protected against that now, as explained in the issue summary, so the rule is basically "We use TranslationWrapper when we know that something is called early".

The first task I'm not sure if possible, I don't know how. We would have to do very dark things (like manually messing with the cached data).

andypost’s picture

It would be great to mention in some update docs/CRs that TranslationWrapper is replacement for st() ;)

Gábor Hojtsy’s picture

@andypost: No, not at all a replacement. st() did translation right away using an array in memory reading from a specific file in the filesystem, while TranslationWrapper() does not do translation until cast to a string sometime later and will use all the regular services that t() would use at the time, same backend. So the behaviours have nothing to do with each other.

Gábor Hojtsy’s picture

Status: Needs review » Needs work
Issue tags: +sprint

@Berdir: looks like this is down to the comment cleanups discussed.

@all: special thanks for maintaining a stellar issue summary.

Berdir’s picture

Status: Needs work » Needs review
FileSize
12.34 KB
1.45 KB

Those comments OK?

Gábor Hojtsy’s picture

Status: Needs review » Reviewed & tested by the community

Looks fine by me. While there is some black magic to which ones to apply the TranslationWrapper treatment, there does not seem to be a possibility to do this cleaner.

catch’s picture

Status: Reviewed & tested by the community » Fixed

Looks good to me as well. TranslationWrapper is the only way we have at the moment to resolve these circular dependencies, so as long as we explain why we're using it I'm happy.

  • catch committed 2d6516a on 8.0.x
    Issue #2288123 by Berdir, dawehner: Fixed Basic authentication broken...
Gábor Hojtsy’s picture

Issue tags: -sprint

Yay, thanks.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

Berdir’s picture