Align most of the tokens in the token processor handling with the legacy handling

[eileenmcnaughton]

Overview

There are 2 ways in the code that tokens can be rendered -

• legacy Utils_Token class (the main way)
• the token processor (the preferred way)

We want to consistently use the token processor - but in order to do so we need to align it as a replacement.

Currently it is not rendering a few fields and presenting the db values, rather than the formatted values of the custom fields.

Before

The 'main' token routine was rendering tokens per the image on the right - the token processor was rendering them per the image on the left.

After

Both render them per the image on the right

The main differences are that the token processor was outputing the data directly from the database but now it is formatting. In addition it was unable to render multiple option values (that is not captured in the screen shot as I had to fix that to get the test to complete.

Technical Details

This revists 2 changes

1. https://issues.civicrm.org/jira/browse/CRM-13161 - the expectations that $value might hold an array like [0 => 'id', 1 => 'label'] no longer appear to be true -and I've added tests for all fields that are offered up
2. The decision not to format this data at this point seems like a non decision a0fcf99#diff-3f93091ccbe99a7c1ae88473057539b1021099b37852cfea68ff948f9f8d1a86R66 - it looks like the custom data was not available and hence code was added to fetch it whereas in all other places it was already available and code was added to format it

Comments

If we merge this it unblocks a lot of code rationalisation but it DOES change token behaviour and we need to warn people. I believe it affects scheduled reminders and CiviMail in some cases.

We probably should just do a pre-upgrade message and warn people.

This is groundwork for #19806

[civibot]

(Standard links)

• If this is your first pull-request for CiviCRM, please browse CONTRIBUTING.md for information about the development and testing processes.
• If you are reviewing this pull-request, you may wish to consult the test sites and the Review Standards (long template, short template).

[eileenmcnaughton]

This line causes an enotice without this cleanup - it was added as part of a fix for https://issues.civicrm.org/jira/browse/CRM-13161 - however, all those greetings and gender fields are now tested & don't appear to have regressed (based on the tests I ran locally) with this change and I don't think the format described is used anymore.

The enotice directly mapped to the token processor having multi-value custom field data

[mattwire]

@eileenmcnaughton Needs rebase

[eileenmcnaughton]

thanks @mattwire - looks like it included something I was reviewed - removed

[eileenmcnaughton]

I now have a table of the specific impacts which we can link to in the preUpgrade hook - perhaps

"CiviCRM has standardised some of the tokens for contact custom fields. If you use these in scheduled reminders or CiviMails or other custom flows then you might need to confirm you are still happy with the output. In general we are switching from outputting the values stored in the database to the display values"

Data type or token	CRM_Utils_Token	Token Processor
Custom date field	Site formatted date eg. 01/20/2020 12:00:00.am	SQL date eg2020-01-20 00:00:00
Option values	Labels eg. Purple	Names e.g P
Contact reference fields	Name eg.Mr Orson Welles,	Id eg.7
Link	Html for link eg	Raw eg.https://civicrm.org

[magnolia61]

For consideration: token values vs token labels. The values make it a lot easier in smarty message templates to use conditionals, as labels can be changed and reworded for custom fields over time. best of both worlds would be to offer both. Maybe with an extra parameter: token:value / token:label Probably not within the scope of this consolidation effort but I though i;d mention it here.

[totten]

@eileenmcnaughton Brilliant research and brilliant test :)

There's a small MC in a test-class, but I don't think it's substantive. I'll give this some r-run and file a longer review.

[eileenmcnaughton]

@totten thx - rebased

[totten]

• General standards
  • (r-explain) Pass (Provisional, maybe tweak with info about r-tech)
  • (r-user) Discussion: Doesn't affect UI, but it does affect token compat. More discussion under r-tech.
  • (r-doc) Discussion (Depends on what you think of my r-tech comments)
  • (r-run) Pass: I tested this by creating several custom fields of different types -- and then creating an Activity and a matching scheduled reminder rule, and including a long list of tokens in the scheduled reminder. I got the same results reported by Eileen.
• Developer standards
  • (r-tech) Discussion: So this obviously is the main issue. I think that -- for dates -- the change is clearly better. For option-values, contact-refs, and links, I think it is better to be aligned with traditional templates -- but I also agree with @magnolia61 that the old output is useful, and there may be some templates/use-cases for which it really is better to use the raw value. Alas, adding a notation like :label or :value would be a considerable sub-project. But I don't think it's too hard to support a notation like {raw_contact.created_date}. So... I think what we should is throw up something like https://lab.civicrm.org/extensions/rawtoken/ and tell anyone who cares to go use that extension.
  • (r-code) Pass
  • (r-maint) Pass
  • (r-test) Pass (provisional, last run)

[eileenmcnaughton]

@totten right those are the same conclusions I drew - other than me not knowing about the raw_token

I have communicated this in more than one dev digest and even included the change table - so I think we should merge this and wordsmith up some sort of pre-upgrade hook message to pre-warn people before upgrading

[totten]

@eileenmcnaughton I guess there isn't a gitlab issue for this? So maybe the upgrader message should link to the PR for discussion?

[eileenmcnaughton]

@totten that sounds good - would you be OK to write the message as a follow up PR (I have a feeling you would do so much copy editing you might be better to write it & I can review :-)

[eileenmcnaughton]

I would link to the actual table here - #19806 (comment)

[totten]

would you be OK to write the message as a follow up PR (I have a feeling you would do so much copy editing you might be better to write it & I can review :-)

That seems like some kind of tennis move. 😉

Yeah, merging, and I'm working on the upgrader.

[totten]

Just to cross-reference:

• Raw token extension: https://lab.civicrm.org/extensions/rawtoken/
• Upgrade docs: https://docs.civicrm.org/sysadmin/en/latest/upgrade/version-specific/#token-format
• Upgrader alert: Upgrader - Display note about token formatting #19920