Allow use of glossaries for translations

Due to limitations of the deepL API we cannot use the glossaries created on the website or in the deepL apps.
In this case we have to implement a custom solution based on drupal entities for creating glossaries via the API documented in https://www.deepl.com/docs-api/managing-glossaries/.

More information about the topic can be found here: https://support.deepl.com/hc/en-us/articles/4405021321746-Managing-gloss...

@daveiano
Feel free to contribute.
RIght now, i've not planned to integrate the feature, cause i'm involved in a time consuming project with not so much time left for extra contributions.

While implementing the feature, we should take care of the following things

• administer glossaries as custom drupal entities (allow multiple glossaries depending on API key of configured deepL translator)
• glossary entity with following fields
  • Internal name (only used for drupal)
  • glossary id (used as unique identifier for drupal and API)
  • Source language
  • Target language
  • Glossary entry with Source Target values (Multiple value field)
• possibility to manage 1-x glossaries
• restrict selectable source/ target languages
• custom validation for glossary entries
• API integration
  • TBD: possibility to sync glossary to API or "add new + delete old" on saving glossary entity
  • check if entries have changed
  • show latest sync date
  • sync glossaries (multi user scenario) / kind-of content locking if multiple users try to edit same glossary
• passing glossary while translating content
  • auto-select latest glossary while tranlating based on source/ target language
  • let user select glossary (restricted to available glossaries based on source/ target language)

As you can see, there are several things, we need to take care, while integrating the glossaries.
For storage purposes a custom entity type would be best fit - syncinc between drupal < > DeepL could be achieved by running a cron/ manually sync process.

API documentation: https://www.deepl.com/de/docs-api/managing-glossaries/

Since i'm currently working on the issue, i'll set the status to Active.

Current progress is available at https://git.drupalcode.org/issue/tmgmt_deepl-3232134/-/tree/3232134-glos...

@daveiano:

The deepl_glossary entity won't get translated, cause you build up a glossary entity with its entries for a specific language pair. While running checkout, those glossary entities will be available for selection in the translation checkout process (since we have access to source/ target language in the checkout, we can check for possible glossary entities to let the user choose from).

Just have a look at the API documentation https://www.deepl.com/docs-api/managing-glossaries/

This is done roughly on my dev machine, but needs some polishing to get pushed to the branch.
Furthermore a cron will be added, to fetch glossaries from the deepl API automatically.

@daveiano:

Let me explain.
The glossary entity stores entries for a specific language combination e.g. EN (English) → DE (German).
A translation of those entities would make no sense for this case.

DeepL uses the entries of the glossary and simply replaces our defined subject with the given definition.
This can be used, if you want a really specific translation for a given term, that differs from the translation deepl would provide while translating.

For that purpose, we have to provide a matching glossary with its entries while requesting a translation.
You can think of multiple glossaries per language pair or just single combination. But in all use cases, we need a selection of the glossary, we want to use for a given job.

The checkout form is the best place to add those informations to the current translation job (you won't need to build a continous job for that purpose). It's just the default translation workflow provided by the underlying tmgmt module. You may take tmgmt_file as an example, how to apply custom checkout settings or other tmgmt related translator providers.

Hope it helps to understand, how the whole glossary "thing" is working. The API documentation also provides lots of infos, how this is working internally. Just check the link in my previous comments.
A recent blog post of deepL also describes the use of glossaries https://www.deepl.com/en/blog/translate-your-way-with-the-deepl-glossary

@daveiano
This is not a bug of the module. The module only stores the subject/ definitions in drupal and pushes those entries into a deepl glossary.
While translating we only pass the glossary id, that should be used by the translation.
The replacement part is happening on the deepL site.
You may use the desktop app of deepL to check, how glossary entries get replaced. While using your example, you'll get the same result. The word (allgemein) is also getting translated unless it should be part of the glossary entry.
This issue could only be addressed by deepL.

@mrshowerman
Right now the quick checkout (auto accept finished translations) won't support glossaries. But the suggestion you think ok could be integrated.
But we should think about having multiple glossary entities with the same target and source language for a translation job.
Which glossary should be used in this case?
Do we need any setting to enable/ disable the use of automatic glossary selection while quick checkout (auto-accept).

Editing / deleting of glossary entries is possible, since the module just uses a standard multiple field widget for editing the entries.
Deleting a single item in a multiple items field is another issue, but this is already being addressed in https://www.drupal.org/project/drupal/issues/1038316.

The API of deepL is "kind of limited" in its editing capabilities. You cannot edit existing glossaries directly, that's why, we delete/ recreate the glossary on "deepL side".

@vistree
We don't have a release plan for the feature, since most of the development is happening besides our main projects.
But the current MR can be tested.
Feel free to do so and add comments in this issue.

Short instructions for testing:

• install tmgmt_deepl and configure the free and/ or pro endpoint with your API key
• install tmgmt_deepl_glossary
  • a new section DeepL Glossaries will be available at admin/tmgmt
  • start adding new glossaries at /admin/tmgmt/deepl_glossaries
  • create new translations via default tmgmt workflow - in case glossary is existing for source -> target language it will be used in your translation
  • if you have multiple glossaries for source -> target language combination, you can select between those glossaries (option needs to be set in tmgmt translator settings

There is still need for some unit tests and polishing of error messages, but the main functionality is working.
I'll rebase against latest 2.1.x/ 2.2.x release, so don't be suprised about the commits in the next comment..

@vistree
Currently this is not possible, because deleting a glossary would result in an API Call, which also deletes the glossary on deepL to keep things consistent.
If you have multiple sites, which should use the same glossary, you can run into some problems due to the API.

• glossary entries cannot be edited directly via the API - instead, we have to do a delete/ add operation -> leads to new glossary_ids with every change
• need of a cron to synchronize glossaries, instead of manual action on the glossary overview page
• not deleting would lead to obsolete glossaries on API site (we cannot delete those via any UI besides the module)

Since the glossary is directly tied to the deepL API token (free or paid), you can create one glossary per project and use it standalone.

Another approach for sharing multiple glossaries would be a "main site", were the glossaries can be edited/ administered for all projects. But for this approach the module needs some additional features like:

• more granular permissions for deleting/ editing glossaries
• cron for automatic synchronisation of glossaries

I set this issue to "Needs review" to get some feedback from the community.
Please play around with the feature and leave comments in the issue for getting a 2.2.0 release.
Alpha release is available for testing.

@DuaelFr: I justed wanted to ask, if everything went fine, while implementing the feature?

@mrshowerman: Further work on this topic should be covered within a new issue. May you check the Proposed resolution there and comment in case i'm missing sth..
Thx.

Yep. Just leave a comment in here. I'll take care of those issues. Thx.

@mrshowerman Thx for the findings. I've changed the code according to your suggestions.
All occurences of deepL where changed to DeepL (like in their API documentation) and the description text was updated.
I also run a phpstan/ phpcs test against the tmgmt_deepl_glossary submodule and added some fixes.

The time was sponsored by my lovely daughter, who is currently at the swimming competition. (Dad taxi 🚕🚕🚕)