Message Templates - Allow rendering & previewing of translated messages #24174

totten · 2022-08-08T10:04:36Z

Overview

Allow MessageTemplates to be translated. This PR builds on prior work (ext/message_admin) which allowed the administrator to create+edit translations, but this update is needed to render, preview, or deliver the translated templates in fully localized form. There are a few major sections in this branch.

(1) APIv4 Translation Mode

Suppose you read a record (such as a MessageTemplate):

$m = MessageTemplate::get()
  ->addWhere('workflow_name', '=', 'xxx')
  ->execute();

This returns the canonical record. However, there may exist Translation records for this MessageTemplate. If you actually want to use those translations, then you must lookup the Translations and synthesize an effective/de-facto template.

The PR adds an option $translationMode which will automatically lookup the best available translation. It currently supports two modes, fuzzy (opt-in) or off (default/status-quo). For example:

$m = MessageTemplate::get()
  ->addWhere('workflow_name', '=', 'xxx')
  ->setLanguage('fr_CA')
  ->setTranslationMode('fuzzy')
  ->execute();

This first loads the canonical version of the xxx template, and then it attempts to translate (replacing msg_subject, msg_html, msg_text with localized variants). It will use the active language (fr_CA) and look for the closest matching template. The choice of "closest" is based in Civi\Core\Locale::renegotiate():

If there is an exact translation (fr_CA), use that.
If there is a similar translation (fr_FR), use that.
If there is no translation, it continue with the original/canonical template.

Note that the translation-mode is emphatically superficial. It modifies the output of a basic get request. It does not support other operations, such as writing-back translated data or filtering/grouping translated data. It is difficult to define an automatic/one-size-fits-all behavior for those operations. (You can use the Translation API instead.)

However, for basic application-usage (fetch and display record X), it is much easier to read data this way.

Recap:

Before: Load individual field-translations via Translation API.
After: Load a translated record, based on the closest-available locale.

(2) Message Template Runtime

There are various APIs for using MessageTemplates. These are slight permutations on the same basic mechanism:

CRM_Core_BAO_MessageTemplate::sendTemplate()
CRM_Core_BAO_MessageTemplate::renderTemplate()
Civi\Api4\WorkflowMessage::render()

In each case, it is possible to either (a) pass in a specific template or (b) autoload a template by name. The PR changes the autoloading rule:

Before: Always load the canonical message-template.
After: If available, load a translated message-template. Pick a template based on fuzzy match to the recipient's preferred_language. Otherwise, continue with the canonical message-template.

(3) Message Admin UI

The message_admin UI allows you to create and preview translated templates. However, it needed a fix to render the tokens in the intended locale:

Before: When previewing a message, tokens are rendered in the language of the web-user. Tokens that rely on formatting or translation may appear inconsistent (e.g. French prose with American currency formatting).
After: When previewing a message, tokens are rendered in the assigned locale of the template.

Comments

There is a long list of locales supported for communication purposes (ie preferred_language; ie languages) - it is longer than the list of locales supported by the CIviCRM application. This is not necessarily a problem - but it can be. (Some mail-tokens may require l10n resources. But tokens are discretionary, and there are often similar/close-enough resources available.) To get the best support for many languages, enable the "Partial/Mixed Locales" option from #24403.

Development History

This PR builds on #24116 and #23844. Major differences relate to:

Splitting setPreferredLanguage() in two:
- setLanguage() is basically as before, but promoted
- setTranslationMode() is the new mechanism that autoloads translations, and it only applies to DAO get
Move the proposed $moneyFormatLocale into an object (along with $tsLocale and $dbLocale).
- Don't sprinkle so many globals+envvars for l10n into random methods.
- Do define the major locale-based flags as a Locale object and make fallbacks more general

This PR was used as an exploratory branch for patches in inter-related areas, so the history is quite long. Several elements were subsequently re-split. This includes smaller cleanups (#24281, #24284, #24369, and #24400) and more visible functionality (#24269, #24430 and dev/tranlsation#78's #24403).

civibot · 2022-08-08T10:04:39Z

(Standard links)

If this is your first pull-request for CiviCRM, please browse CONTRIBUTING.md for information about the development and testing processes.
If you are reviewing this pull-request, you may wish to consult the test sites and the Review Standards (long template, short template).

eileenmcnaughton · 2022-08-10T11:00:27Z

That is an odd conflict @totten - maybe we should just merge that unhide since we should be able to sort the rest before the next forking (we REALLY want to wrap it up this week or next)

eileenmcnaughton · 2022-08-16T19:22:58Z

I just rebased out the 'unhide msg_admin' commit as it is otherwise merged

eileenmcnaughton · 2022-08-16T21:28:27Z

The error is an uncaught Exception

Fatal error: Uncaught PEAR_Exception: DB Error: no such field in /home/jenkins/bknix-dfl/build/core-24174-78r6x/web/sites/all/modules/civicrm/vendor/pear/pear-core-minimal/src/PEAR.php on line 944
 DB_Error: DB Error: no such field in /home/jenkins/bknix-dfl/build/core-24174-78r6x/web/sites/all/modules/civicrm/CRM/Core/Error.php on line 955

PEAR_Exception: DB Error: no such field in /home/jenkins/bknix-dfl/build/core-24174-78r6x/web/sites/all/modules/civicrm/CRM/Core/Error.php on line 955

  6.9619   67052344  24. CRM_Core_Smarty::singleton() /home/jenkins/bknix-dfl/build/core-24174-78r6x/web/sites/all/modules/civicrm/ext/greenwich/greenwich.civix.php:94
    6.9619   67053744  25. CRM_Core_Smarty->initialize() /home/jenkins/bknix-dfl/build/core-24174-78r6x/web/sites/all/modules/civicrm/CRM/Core/Smarty.php:163
    6.9621   67056000  26. CRM_Core_I18n::uiLanguages($justCodes = ???) /home/jenkins/bknix-dfl/build/core-24174-78r6x/web/sites/all/modules/civicrm/CRM/Core/Smarty.php:128
    6.9621   67056376  27. CRM_Core_I18n::languages($justEnabled = ???) /home/jenkins/bknix-dfl/build/core-24174-78r6x/web/sites/all/modules/civicrm/CRM/Core/I18n.php:278
    6.9626   67106952  28. CRM_Core_OptionValue::getValues($groupParams = ['name' => 'languages'], $values = NULL, $orderBy = 'weight', $isActive = TRUE) /home/jenkins/bknix-dfl/build/core-24174-78r6x/web/sites/all/modules/civicrm/CRM/Core/I18n.php:183
    6.9626   67108552  29. CRM_Core_DAO::executeQuery($query = '\nSELECT\n   option_value.id          as id,\n   option_value.label       as label,\n   option_value.value       as value,\n   option_value.name        as name,\n   option_value.description as description,\n   option_value.weight      as weight,\n   option_value.is_active   as is_active,\n   option_value.icon        as icon,\n   option_value.color       as color,\n   option_value.is_default  as is_default\nFROM\n   civicrm_option_value  as option_value,\n   civicrm_option_group  as option_group  WHERE optio', $params = [2 => [0 => 'languages', 1 => 'String']], $abort = ???, $daoName = ???, $freeDAO = ???, $i18nRewrite = ???, $trapException = ???, $options = ???) /home/jenkins/bknix-dfl/build/core-24174-78r6x/web/sites/all/modules/civicrm/CRM/Core/OptionValue.php:449
    6.9626   67109704  30. CRM_Core_DAO->query($query = '\nSELECT\n   option_value.id          as id,\n   option_value.label       as label,\n   option_value.value       as value,\n   option_value.name        as name,\n   option_value.description as description,\n   option_value.weight      as weight,\n   option_value.is_active   as is_active,\n   option_value.icon        as icon,\n   option_value.color       as color,\n   option_value.is_default  as is_default\nFROM\n   civicrm_option_value  as option_value,\n   civicrm_option_group  as option_group  WHERE optio', $i18nRewrite = TRUE) /home/jenkins/bknix-dfl/build/core-24174-78r6x/web/sites/all/modules/civicrm/CRM/Core/DAO.php:1637
    6.9626   67109704  31. CRM_Core_DAO->query($string = '\nSELECT\n   option_value.id          as id,\n   option_value.label       as label,\n   option_value.value       as value,\n   option_value.name        as name,\n   option_value.description as description,\n   option_value.weight      as weight,\n   option_value.is_active   as is_active,\n   option_value.icon        as icon,\n   option_value.color       as color,\n   option_value.is_default  as is_default\nFROM\n   civicrm_option_value  as option_value,\n   civicrm_option_group  as option_group  WHERE optio') /home/jenkins/bknix-dfl/build/core-24174-78r6x/web/sites/all/modules/civicrm/CRM/Core/DAO.php:472
    6.9626   67109704  32. CRM_Core_DAO->_query($string = '\nSELECT\n   option_value.id          as id,\n   option_value.label       as label,\n   option_value.value       as value,\n   option_value.name        as name,\n   option_value.description as description,\n   option_value.weight      as weight,\n   option_value.is_active   as is_active,\n   option_value.icon        as icon,\n   option_value.color       as color,\n   option_value.is_default  as is_default\nFROM\n   civicrm_option_value  as option_value,\n   civicrm_option_group  as option_group  WHERE optio')

eileenmcnaughton · 2022-08-16T21:36:06Z

I pulled out a commit to get it merged & reduce the noise of this #24281

Not too sure about that exception though

totten · 2022-08-17T04:25:07Z

Rebased to hide some bits that we're merged. Added a fix for the !$x instanceof AbstractAction bug, which is what caused broad failures in the test-suite.

eileenmcnaughton · 2022-08-17T06:43:08Z

nope - that didn't solve it

79jh5/web/sites/default/files/civicrm/upload/', 'customPHPPathDir' => FALSE, 'customTemplateDir' => FALSE, 'templateCompileDir' => '/home/jenkins/bknix-dfl/build/core-24174-79jh5/web/sites/default/files/civicrm/templates_c/', 'configAndLogDir' => '/home/jenkins/bknix-dfl/build/core-24174-79jh5/web/sites/default/files/civicrm/ConfigAndLog/'] }) /home/jenkins/bknix-dfl/build/core-24174-79jh5/web/sites/all/modules/civicrm/ext/greenwich/greenwich.php:14
    7.3751   67058128  24. CRM_Core_Smarty::singleton() /home/jenkins/bknix-dfl/build/core-24174-79jh5/web/sites/all/modules/civicrm/ext/greenwich/greenwich.civix.php:94
    7.3751   67059528  25. CRM_Core_Smarty->initialize() /home/jenkins/bknix-dfl/build/core-24174-79jh5/web/sites/all/modules/civicrm/CRM/Core/Smarty.php:163
    7.3752   67061768  26. CRM_Core_I18n::uiLanguages($justCodes = ???) /home/jenkins/bknix-dfl/build/core-24174-79jh5/web/sites/all/modules/civicrm/CRM/Core/Smarty.php:128
    7.3752   67062144  27. CRM_Core_I18n::languages($justEnabled = ???) /home/jenkins/bknix-dfl/build/core-24174-79jh5/web/sites/all/modules/civicrm/CRM/Core/I18n.php:278
    7.3756   67112720  28. CRM_Core_OptionValue::getValues($groupParams = ['name' => 'languages'], $values = NULL, $orderBy = 'weight', $isActive = TRUE) /home/jenkins/bknix-dfl/build/core-24174-79jh5/web/sites/all/modules/civicrm/CRM/Core/I18n.php:183
    7.3756   67114320  29. CRM_Core_DAO::executeQuery($query = '\nSELECT\n   option_value.id          as id,\n   option_value.label       as label,\n   option_value.value       as value,\n   option_value.name        as name,\n   option_value.description as description,\n   option_value.weight      as weight,\n   option_value.is_active   as is_active,\n   option_value.icon        as icon,\n   option_value.color       as color,\n   option_value.is_default  as is_default\nFROM\n   civicrm_option_value  as option_value,\n   civicrm_option_group  as option_group  WHERE optio', $params = [2 => [0 => 'languages', 1 => 'String']], $abort = ???, $daoName = ???, $freeDAO = ???, $i18nRewrite = ???, $trapException = ???, $options = ???) /home/jenkins/bknix-dfl/build/core-24174-79jh5/web/sites/all/modules/civicrm/CRM/Core/OptionValue.php:449
    7.3756   67115472  30. CRM_Core_DAO->query($query = '\nSELECT\n   option_value.id          as id,\n   option_value.label       as label,\n   option_value.value       as value,\n   option_value.name        as name,\n   option_value.description as description,\n   option_value.weight      as weight,\n   option_value.is_active   as is_active,\n   option_value.icon        as icon,\n   option_value.color       as color,\n   option_value.is_default  as is_default\nFROM\n   civicrm_option_value  as option_value,\n   civicrm_option_group  as option_group  WHERE optio', $i18nRewrite = TRUE) /home/jenkins/bknix-dfl/build/core-24174-79jh5/web/sites/all/modules/civicrm/CRM/Core/DAO.php:1648
    7.3756   67115472  31. CRM_Core_DAO->query($string = '\nSELECT\n   option_value.id          as id,\n   option_value.label       as label,\n   option_value.value       as value,\n   option_value.name        as name,\n   option_value.description as description,\n   option_value.weight      as weight,\n   option_value.is_active   as is_active,\n   option_value.icon        as icon,\n   option_value.color       as color,\n   option_value.is_default  as is_default\nFROM\n   civicrm_option_value  as option_value,\n   civicrm_option_group  as option_group  WHERE optio') /home/jenkins/bknix-dfl/build/core-24174-79jh5/web/sites/all/modules/civicrm/CRM/Core/DAO.php:479
    7.3756   67115472  32. CRM_Core_DAO->_query($string = '\nSELECT\n   option_value.id          as id,\n   option_value.label       as label,\n   option_value.value       as value,\n   option_value.name        as name,\n   option_value.description as description,\n   option_value.weight      as weight,\n   option_value.is_active   as is_active,\n   option_value.icon        as icon,\n   option_value.color       as color,\n   option_value.is_default  as is_default\nFROM\n   civicrm_option_value  as option_value,\n   civicrm_option_group  as option_group  WHERE optio') /home/jenkins/bknix-dfl/build/core-24174-79jh5/web/sites/all/modules/civicrm/packages/DB/DataObject.php:1829

Found EXITCODES=""

Fatal error: Uncaught Exception: String could not be parsed as XML in /home/jenkins/bknix-dfl/extern/phpunit-xml-cleanup.php on line 27

eileenmcnaughton · 2022-08-18T21:43:35Z

test this please

eileenmcnaughton · 2022-08-19T00:34:42Z

That's better - twelve clear fails - I suspect this might account for more than one...

CRM_Utils_TokenConsistencyTest::testDomainNow
TypeError: Civi\Core\Locale::getLocalePrecedence(): Argument #1 ($preferred) must be of type string, null given, called in /home/jenkins/bknix-dfl/build/core-24174-7coap/web/sites/all/modules/civicrm/Civi/Core/Locale.php on line 248

eileenmcnaughton · 2022-08-20T01:54:00Z

CRM/Core/BAO/Translation.php

+      $languages[$field['language']][$index] = $field;
+    }
+
+    $bizLocale = $userLocale->renegotiate(array_keys($languages));


This really stumped me - I couldn't figure out what Biz Locale is - 'Business Locale' 'Bees Locale' 'Random Prefix Locale'?

I feel like maybe this is the ResolvedLocale?

eileenmcnaughton · 2022-08-20T06:50:15Z

Fails are getting more clearly related now & testRenderTranslatedTemplate failing to correctly format currency to the selected language

api_v3_MultilingualTest::testOptionLanguage with data set "APIv3" (3)
Invalid getsingle resultArray
(
[error_code] => no such table
[sql] => SELECT *
FROM civicrm_contact_type_
[nativecode=1146 ** Table 'core241747ew7ctest_6nqrv.civicrm_contact_type_' doesn't exist]
[tip] => add debug=1 to your API call to have more info about the error
[is_error] => 1
[error_message] => DB Error: no such table
[debug_information] => SELECT *
FROM civicrm_contact_type_
[nativecode=1146 ** Table 'core241747ew7ctest_6nqrv.civicrm_contact_type_' doesn't exist]
[entity] => OptionValue
[action] => getsingle
[trace] => #0 /home/jenkins/bknix-dfl/build/core-24174-7ew7c/web/sites/all/modules/civicrm/api/api.php(173): civicrm_api3('OptionValue', 'getfields', Array)

eileenmcnaughton · 2022-08-24T05:06:32Z

I pulled out a couple of test tweaks to a separate PR. I don't hold up much hope of this unblocking us but it makes me feel a little less helpless :-) #24369

totten · 2022-08-24T23:45:49Z

Thanks for pulling those out @eileenmcnaughton. I'm rebasing and squashing a couple of the "fixup" commits.

I'm glad the test-results actually show the failures now... :)

eileenmcnaughton · 2022-08-30T21:57:27Z

still passing - we need the next chunk huh?

eileenmcnaughton · 2022-09-01T02:55:59Z

Attempt to pick off language not working yet - #24430

The functionality is bigger than `MessageTemplate`, and the test is fairly long.

Move hook_civicrm_translateFields from message_admin to core Move hook_civicrm_translateFields from message_admin to core m

…der the translated version

…n get/render the translated version * Updates for APIv4 calls * Set `$language` and `#ranslationMode()` instead of `$preferredLanguage` * Read 'actual_language' instead of `getTranslationLanguage()` * Updates for tracking global locale properties * Use `$loacleObj->moneyFormat` instead of `$GLOBALS['moneyFormatLocale']` and `IGNORE_SEPARATOR_CONFIG` * Use `$tokenContext['locale']` instead of `$GLOBALS['moneyFormatLocale']` and `IGNORE_SEPARATOR_CONFIG` * Split `testRenderTranslatedTemplate()` in two (for different configurations)

eileenmcnaughton · 2022-09-01T06:51:40Z

I rebased out the merged changes

eileenmcnaughton · 2022-09-01T21:35:39Z

Yay - the bug is fixed - talked with Tim & we are ok to merge

totten · 2022-09-02T00:10:39Z

@eileenmcnaughton I did a pass on updating the description.

eileenmcnaughton · 2022-09-02T01:09:44Z

unrelated fails

eileenmcnaughton · 2022-09-02T01:10:03Z

yay - thanks for that update

civibot bot added the master label Aug 8, 2022

This was referenced Aug 14, 2022

Simplify multilingual check #24130

Closed

Unhide message_admin extension #24269

Merged

eileenmcnaughton force-pushed the master-lang branch from 5253c13 to 2c601ee Compare August 16, 2022 19:22

eileenmcnaughton mentioned this pull request Aug 16, 2022

(REF) I18nSubscriber - Rename parameter to be more evocative #24281

Merged

totten mentioned this pull request Aug 16, 2022

(NFC) Mark various tests with @group locale #24284

Merged

eileenmcnaughton mentioned this pull request Aug 17, 2022

dev/core#3656 Translations - if a message_template has been translated then get/render the translated version #23844

Closed

totten force-pushed the master-lang branch from 2c601ee to 2f6ef72 Compare August 17, 2022 04:02

totten force-pushed the master-lang branch 2 times, most recently from ac2a248 to 8169749 Compare August 18, 2022 10:34

eileenmcnaughton reviewed Aug 20, 2022

View reviewed changes

eileenmcnaughton mentioned this pull request Aug 20, 2022

APIv4 - Add generic setLanguage, to all APIs #24116

Closed

eileenmcnaughton mentioned this pull request Aug 24, 2022

Test only fixes #24369

Merged

totten force-pushed the master-lang branch from 613c39f to 7a3baa2 Compare August 24, 2022 23:45

totten changed the title ~~(EXP) Various experimental patches in i18n~~ (EXP) (dev/translation#78) Various experimental patches in i18n Aug 24, 2022

totten force-pushed the master-lang branch 3 times, most recently from f0af670 to c618487 Compare August 25, 2022 07:46

totten force-pushed the master-lang branch from 1c0e5ae to e7be2a3 Compare August 28, 2022 23:56

totten mentioned this pull request Aug 28, 2022

(REF) Tests - Misc updates in prepartion for l10n updates #24400

Merged

totten changed the title ~~(EXP) (dev/translation#78) Various experimental patches in i18n~~ (EXP) Various experimental patches in i18n (partial locales + API $language + MessageTemplate) Aug 29, 2022

totten force-pushed the master-lang branch 3 times, most recently from b821944 to f148983 Compare August 29, 2022 23:35

totten and others added 9 commits September 1, 2022 18:51

APIv4 - Declare option $translationMode (for DAOGetAction)

09f31d4

APIv4 - Implement option $translationMode via TranslationGetWrapper

9048f45

(REF) Move test for $translationMode to its own class

f61325e

The functionality is bigger than `MessageTemplate`, and the test is fairly long.

APIv4 - Add generic 'language', 'translationLanguage' to all APIs

c17f575

Move hook_civicrm_translateFields from message_admin to core Move hook_civicrm_translateFields from message_admin to core m

Translations - if a message_template has been translated then get/ren…

5ea3ea0

…der the translated version

Translation BAO - Minor strictness cleanups

6a02902

Pass language into msg_admin api calls

01b6270

MessageTemplateTest - Rearrange with data-provider. Add more tests.

0fa49e7

eileenmcnaughton force-pushed the master-lang branch from f148983 to 0fa49e7 Compare September 1, 2022 06:51

Fix UI bug

669dcbf

eileenmcnaughton marked this pull request as ready for review September 1, 2022 21:33

eileenmcnaughton changed the title ~~(EXP) Various experimental patches in i18n (partial locales + API $language + MessageTemplate)~~ Fix message templates to preview & render in for specific locales, using translationMode Sep 1, 2022

eileenmcnaughton added the merge on pass label Sep 1, 2022

totten changed the title ~~Fix message templates to preview & render in for specific locales, using translationMode~~ Message Templates - Allow rendering & previewing of translated messages Sep 1, 2022

eileenmcnaughton merged commit d1f11af into civicrm:master Sep 2, 2022

totten deleted the master-lang branch September 2, 2022 05:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Message Templates - Allow rendering & previewing of translated messages #24174

Message Templates - Allow rendering & previewing of translated messages #24174

totten commented Aug 8, 2022 •

edited

Loading

civibot bot commented Aug 8, 2022

eileenmcnaughton commented Aug 10, 2022

eileenmcnaughton commented Aug 16, 2022

eileenmcnaughton commented Aug 16, 2022

eileenmcnaughton commented Aug 16, 2022

totten commented Aug 17, 2022

eileenmcnaughton commented Aug 17, 2022

eileenmcnaughton commented Aug 18, 2022

eileenmcnaughton commented Aug 19, 2022 •

edited

Loading

eileenmcnaughton Aug 20, 2022

eileenmcnaughton commented Aug 20, 2022

eileenmcnaughton commented Aug 24, 2022

totten commented Aug 24, 2022 •

edited

Loading

eileenmcnaughton commented Aug 30, 2022

eileenmcnaughton commented Sep 1, 2022

eileenmcnaughton commented Sep 1, 2022

eileenmcnaughton commented Sep 1, 2022

totten commented Sep 2, 2022

eileenmcnaughton commented Sep 2, 2022

eileenmcnaughton commented Sep 2, 2022

Message Templates - Allow rendering & previewing of translated messages #24174

Message Templates - Allow rendering & previewing of translated messages #24174

Conversation

totten commented Aug 8, 2022 • edited Loading

Overview

(1) APIv4 Translation Mode

(2) Message Template Runtime

(3) Message Admin UI

Comments

Development History

civibot bot commented Aug 8, 2022

eileenmcnaughton commented Aug 10, 2022

eileenmcnaughton commented Aug 16, 2022

eileenmcnaughton commented Aug 16, 2022

eileenmcnaughton commented Aug 16, 2022

totten commented Aug 17, 2022

eileenmcnaughton commented Aug 17, 2022

eileenmcnaughton commented Aug 18, 2022

eileenmcnaughton commented Aug 19, 2022 • edited Loading

eileenmcnaughton Aug 20, 2022

Choose a reason for hiding this comment

eileenmcnaughton commented Aug 20, 2022

eileenmcnaughton commented Aug 24, 2022

totten commented Aug 24, 2022 • edited Loading

eileenmcnaughton commented Aug 30, 2022

eileenmcnaughton commented Sep 1, 2022

eileenmcnaughton commented Sep 1, 2022

eileenmcnaughton commented Sep 1, 2022

totten commented Sep 2, 2022

eileenmcnaughton commented Sep 2, 2022

eileenmcnaughton commented Sep 2, 2022

totten commented Aug 8, 2022 •

edited

Loading

eileenmcnaughton commented Aug 19, 2022 •

edited

Loading

totten commented Aug 24, 2022 •

edited

Loading