Add Google's new robots tags (D8)

This is going to be interesting. Currently we use a checkboxes element and simply concatenating the values. Now we have additional properties, separated by a colon.

1. max-snippet and max-video-preview requires a number field
2. max-image-preview requires a select field

Related: #3068572: Add support for googlebot_news meta tags (D8)

I have tested the above patch and can confirm it's output is correctly shown on the configured page.

I have also checked the PHPUnit test and it seems to fail on this line

   // Confirm that no changes have been made yet.
    $this->assertNoText('Overridden');

Just before that we go to a view

 // Load the Arcive view.
    $this->drupalGet('/admin/structure/views/view/archive');
    $this->assertResponse(200);

I do not think this test fail is related to the patch provided above, seems unrelated. It fails because this view contains a contextual filter "Content: Created year + month" which is "Overridden"

The test failures need to be resolved.

@DamienMcKenna, the test fails on an unrelated view('/admin/structure/views/view/archive'), not sure why we would fix this in this issue? I looked over it and we simply cannot do the assertion:

   $this->assertNoText('Overridden');

because with a clean Drupal 8 install, this text already exists.

Will this feature be backported to D7 ? We really need it, and I suppose several of News sites still under D7 do it as well. Google removed description and all snippets of my site's lines appearing in search results - only a poor title shows - according to it's statement on EU Directive 2019/790 (see above @druhu #3)

I added a new issue for a backport: #3106692: Add Google's new robots tags (D7)

@DamienMcKenna: Is it possible to adapt code from D7 to get this also working on D8?

Problem is, that current provided D8 patch breaks robots metatag when saving a content, because in setValue() method new form structure is not handled correctly.

We currently get articles with something like "robots" or "robots, max-image-preview" in content of robots metatag. So it seems that default and/or selected settings will get lost and maybe also corrupt data is saved into metatag fields table for your entities.

Given patch in #6 is not working. Please find the attached one.

This should work

This should work

@gkannan25: fyi your file uploads and comments were not visible because your account was so new, but I've fixed that so now you can upload patches.

Let's see what testbot thinks of the new patch.

Trying another patch

the patch works so far, even if the patch tests fail. however, i always have this error when editing a node or calling the metatag admin interface. this is a bit inconvenient. occurs under drupal 9.2.0.

Notice: Undefined index: max-video-preview in Drupal\metatag\Plugin\metatag\Tag\Robots->form() (line 135 of /var/www/html/docroot/modules/contrib/metatag/src/Plugin/metatag/Tag/Robots.php)

all in all, it is a very important specification and the patch should come in core if possible.

I modified #27 to be working with views metatags and compatible with current dev state.
Values' join logic moved from MetatagDefaultsForm::save() to the Robots::validateTag() to get it working on views options form submit as well.

Thanks for the update @Eugene, however this cannot be applied. Metatag version 8.x-1.16, Drupal 9.2.0.

Could not apply the patch! Skipped. The error was: Cannot apply patch https://www.drupal.org/files/issues/2021-07-06/added-new-robots-tags-3083743-31.patch

@zcht: FYI patches are preferably built against the current dev codebase, not against the last stable release, so I'm putting this back to "needs review".

Yes, the patching files in the dev already differ from 1.16. I've made variant for Metatag 8.x-1.16

@DamienMcKenna right, didn't take this into consideration. Let's hope for a release soon :)

@Eugene, thanks for the patch, works great. Karma++

Patch #34 for 8.x-1.16 RTTB++

I have found a bug here, the behaviour actually only occurs with this patch. As soon as the patch is removed, everything is normal and functional again.

I use the following: Metatag version 8.x-1.16, Drupal 9.2.0, Group 1.4

I have configured the metatags for one content type and allow this content type to be used in a group. As long as migrated content is accessed via this group, all robot tags are correct, including those from this patch.

If I edit and save the node as an administrator, the output is still correct.

However, as soon as the node is saved by a user IN the group, i.e. just open and save, the output is completely wrong and strange. After the configuration export, I have the following in the field.field.node.page.field_meta_tags.yml

default_value:
  -
    value: 'a:1:{s:6:"robots";s:20:"robots, robots-keyed";}'

Output in the frontend looks like this: <meta name="robots" content="robots, robots-keyed">

An entry is made in the table node__field_meta_tags for this updated node and only if the node is updated via the group. Editing the node again as administrator does not help, all checkboxes are deselected.

Thank you @zcht for pointing to this and provide the additional information. You are right, there is an issue with validateTag() function. It works well with global configured tags, but not with node's individual. I'll try to solve this. And obviously, it requires to add tests for that.

These are modified #31 and #34 for dev branch and 8.x-1.16 which should solve #36 issue. But they don't contains tests yet.

Thanks @Eugene, for the adaptation of the patch. Looks very good so far, all settings remain as they should.

Karma++ :)

This patch differs from #38 only in the tests. I suggest to add a call of method named {$tagname}TestFormValues (if presented) in the Drupal\Tests\metatag\Functional\MetatagTagsTestBase::testTagsInputOutput to get the keys/values array for the edit form in the tests. This will allow use this method to test tags that use more than one form element for editing.

With the current version 1.18 this patch can unfortunately no longer be applied. It would be really nice if we not only update this function but also actually bring it into the core of the module. These metatags are very relevant especially for content platforms.

@zcht: This would "bring it into the core of the module", it's additions to the "robots" tag that is part of the main Metatag module.

Rerolled.

Rerolled correctly X-)

works perfectly with 1.19, thanks for the reroll

Karma++ :)

I've also tested #45 with 1.19, works like a charm.

Replace t() with $this->t().

Latest patch #48 doesn't apply against the 1.21 release, I'm adding a re-roll

#49 Does not seem to work. I set max-image-preview:large for content type article under /admin/config/search/metatag but its not set by default for already created articles (which do not have overridden the robots meta tag). Creating a new article it's not selected and not added to the robots after saving. Manually setting the value works, but in database it now appears as overridden, which shouldn't because it's the default setting.

I had problems with the robot meta tags, but they are likely not related with this issue, but somehow with language settings.

I have a config files sync/metatag.metatag_defaults.node__article.yml and sync/language/de/metatag.metatag_defaults.node__article.yml. While updating default settings for node article meta tags only sync/metatag.metatag_defaults.node__article.yml got updated

-  robots: 'index, follow'
+  robots: 'index, follow, max-image-preview:large'

but not sync/language/de/metatag.metatag_defaults.node__article.yml (there I still had robots: 'index, follow'). Very likely that way the option for max-image-preview:large was not selected by default on /node/add/article.

Then I removed all robot settings from node article metatag defaults and saved. Both config files sync/metatag.metatag_defaults.node__article.yml and sync/language/de/metatag.metatag_defaults.node__article.yml where updated. Now the default value was used from node metatag defaults, where I enabled max-image-preview:large which also worked for /node/add/article form.

Last test was setting robots metatag default for node back to "index, follow" and for node article to "index, follow, max-image-preview:large". Now again only sync/metatag.metatag_defaults.node__article.yml but not sync/language/de/metatag.metatag_defaults.node__article.yml was updated. After that the option for max-image-preview:large was selected by default on /node/add/article.

I guess it is not related with issue but who knows, maybe this helps somebody.

Latest patch #49 doesn't apply against the 1.22 release.

Rerolled #49 against 8.x-1.x, not sure if rerolling the tests went well.

Made a merge request with all the work done up to now. I attempted to fix tests and I also added the new `unavailable_after` robots tag.

Setting for review.
Thanks

Since tests are green I'm toggling visibility of previous patches.

#61 MR works for us. I set the issue status to RTBC. Thanks.

When I applied this patch (MR), form doesn't store well robots checkbox options like "nofollow", if you check and save and then edit again the choices are disappeared, and the html it will be:

@jansete can you check again please? Making sure you cleared caches etc? It works flawlessly on my test. Screenshot is after saving and revisiting the edit page:

Metatags options page is also working ok, and of course all the metatags appear as expected in the DOM.

Settings this back to needs review and will bump it to RTBC if I don't hear from you in the coming days.
Thanks

I see, you are right.

The problem is only when using with the asynchronous patch from this issue https://www.drupal.org/project/metatag/issues/3057523

Sorry!

Copy of patch from https://git.drupalcode.org/project/metatag/-/merge_requests/73#note_140080 attached, to be used with composer, see "Patches from drupal.org merge request URLs are dangerous?".

Patch provided in comment #66 works great for me.
thank you :)

This patch is only relevant at the moment, but has also gotten several literation. would be nice if the work can be incorporated into the core of the module.

Working on making sure this is compatible with v2's tests.

Updated patch with test coverage for the 2.0.x branch.

Because the equivalent change went into 7.x-1.x already, I plan on adding this to 8.x-1.x but needed to get the tests working on the 2.0.x branch first.

Committed. Thank you.