Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix loading, saving, and upgrade of CiviContribute settings #17188

Merged
merged 8 commits into from
Apr 30, 2020

Conversation

totten
Copy link
Member

@totten totten commented Apr 28, 2020

Overview

Historically, preferences related to tax and invoice were stored in the setting contribution_invoice_settings. Circa 5.23, contribution_invoice_settings was deprecated and decomposed into several smaller settings (invoice_due_date, tax_term, etc) . There is a transitional mechanism to convert between the old-style/large setting and the new-style/small settings, but it has issues. This PR updates the transitional mechanism.

This patch should fix transitional issues for most deployments. However, if you rely on the "CiviContribute Component Settings" to customize the "Tax and Invoicing" options, and if you've previously used an affected version (5.23.0 - 5.24.5), then you may wish to review the settings to ensure they are correct.

See also:

Before

  1. The page "CiviContribute Component Settings` does not reliably save/load the values under "Enable Tax and Invoicing".
  2. If you start with an older version (eg 5.22), customize contribution_invoice_settings, and upgrade... then the settings are not converted.
  3. Depending on the order in which settings are read and written, the values may not be consistent in the new-style settings and old-style settings.
  4. Explicit-values are stored in new-style settings (invoice_due_date, tax_term, etc), but default-values are defined in old-style contribution_invoice_settings - leading to inconsistent behavior.
  5. The values in contribution_invoice_settings do not reflect overrides ($civicrm_settings).

After

  1. The page "CiviContribute Component Settings` does reliably save/load the values under "Enable Tax and Invoicing".
  2. If you start with an older version (eg 5.22), customize contribution_invoice_settings, and upgrade... then the settings are converted.
  3. One may more freely interchange old-style and new-style.
  4. Both default-values and explicit-values are tracked in the new-style settings. Values never come from the old-style setting, which is purely virtual.
  5. Any mix of default-values, explicit-values, and mandatory/override-values can be reflected in contribution_invoice_settings.

Comments

Why does the "Overview" suggest that admins review the "CiviContribute Component Settings"? As a precaution. The main consideration would arise if someone attempted to reconfigure the options while using a version (5.23.0-5.24.5) where the form didn't save reliably. That could produce confusion. The clearest antidote is to simply check the final settings and see if they are correct.

eileenmcnaughton and others added 4 commits April 28, 2020 19:38
It seems that, depending on the order in which settings are loaded, this is not always populated. Moving it here
addresses, although medium term we should address all references to it
…irtual

The `SettingsBag` uses the `$combined` property to (locally) cache a full
view of the active settings (based on combining different layers of data --
default-values, explicit-values, and mandatory-values). This design is
good for read-mostly data.

This patch changes the `__ContributionSettings` to be another layer in the
construction of `$combined` -- the virtual-values layer.  It fixes issues
wherein:

1. The virtual value is recomputed during every call to `get($key)`.

2. The virtual value is only based on explicit-values -- it disregards
   default-values and mandatory-values.
The default values were previously stored in 'contribution_invoice_settings', but
the updates are stored in individual settings, which makes it difficult to reconcile
the default values and explicit values.

With this revision, 'contribution_invoice_settings' is strictly virtual, and
all the other settings (like 'invoice_due_date') are canonical.
@civibot
Copy link

civibot bot commented Apr 28, 2020

(Standard links)

@civibot civibot bot added the 5.25 label Apr 28, 2020
@eileenmcnaughton eileenmcnaughton changed the title (WIP) Fix contribution settings (with itemized settings as canonical) Fix contribution settings (with itemized settings as canonical) Apr 29, 2020
@totten totten changed the title Fix contribution settings (with itemized settings as canonical) Fix loading, saving, and upgrade of CiviContribute settings Apr 29, 2020
@totten
Copy link
Member Author

totten commented Apr 29, 2020

Note: on my local, I've done some testing to make sure that "CiviContribute Component Settings" reads data from previous versions - and can save/load updates - when starting from:

  • 5.21 DB with customized invoice settings
  • 5.22 DB with customized invoice settings
  • 5.22 DB with default invoice settings
  • 5.25.beta DB with customized invoice settings

@totten
Copy link
Member Author

totten commented Apr 29, 2020

jenkins, test this please

@totten totten force-pushed the 5.25-contrib-virt-itemcanon branch from b6fa397 to 2bb2705 Compare April 29, 2020 07:20
@seamuslee001
Copy link
Contributor

I tested the upgraded and it worked correctly and the form loaded correctly, The code looks correct from my POV. Lets see what Jenkins makes of it

@totten
Copy link
Member Author

totten commented Apr 29, 2020

Hmmblarneyblerg. So a previous test-run with the last commit (dev/core#1724 - Backward-compatible handling of 'contribution_invoice_settings.invoicing') showed a failure in CRM_Contribute_Form_Contribution_ConfirmTest which I was able to reproduce. So I'm expecting Jenkins to complain again. My interpretation is that the test has traditionally run with invoicing disabled -- and, if it incorrectly believes that invoicing is enabled (e.g. due to confusion about the format of contribution_invoice_settings), then it runs some extra code which raises warnings.

I'm confused by how we're boxed in on the content of contribution_invoice_settings.invoicing -- i.e. is it a bool or an an array containing a bool.

  • If contribution_invoice_settings.invoicing is an array containing a bool, then the settings form works.
  • If contribution_invoice_settings.invoicing is a bool itself, then snippets like this work:
          $invoiceSettings = Civi::settings()->get('contribution_invoice_settings');
          $invoicing = $invoiceSettings['invoicing'] ?? NULL;
          if ($invoicing) {

The perplexing thing is that both of those seem older than the regression. (And this feels like an exercise in obscura... but we do want upgrades to work...) Maybe I need to step away and look at it again tomorrow. 😕

@totten totten force-pushed the 5.25-contrib-virt-itemcanon branch 2 times, most recently from 8388462 to 60e3ae8 Compare April 29, 2020 22:39
@totten
Copy link
Member Author

totten commented Apr 29, 2020

OK, I think I got it. The data-type of contribution_invoice_settings.invoicing depended on whether the preference was to enable or disable the option:

  1. If enabled, the value was an array with trueish subkey: ['invoicing'=>1]
  2. If disabled, the value was a numeric zero.

Which is... bonkers. 🤷

So I've pushed an updated form of the last commit which uses that format. Retested a few cases locally, and these seem good:

  • Upgrade 5.21 DB with customized invoice settings - and view/edit settings
  • Upgrade 5.22 DB with default invoice settings - and view/edit settings
  • Upgrade 5.23 DB with customized invoice settings - and view/edit settings
  • Install new 5.25 DB - and view/edit settings

Additionally, CRM_Contribute_Form_Contribution_ConfirmTest::testPaynowPayment passes locally and looks to have passed in the current/on-going Jenkins run.

@totten totten force-pushed the 5.25-contrib-virt-itemcanon branch from 60e3ae8 to db5de31 Compare April 29, 2020 23:33
totten added 3 commits April 29, 2020 16:34
Consider this use-case:

* Install 5.22
* Configure `CiviContribute Component Settings`
* Upgrade to 5.25.beta

The settings are lost during the upgrade.
…_settings.invoicing'

In the old setting `contribution_invoice_settings`, the `invoicing`
property (aka "CiviContribute Component Settings" => "Enable Tax and
Invoicing") could take one of two values:

1. If enabled, it's an array with trueish subkey: `['invoicing'=>1]`
2. If disabled, it's a numeric zero.

This encoding is weird, and `contribution_invoice_settings` is generally deprecated
and has been replaced by a virtual rendition which needs to be consistent with the
old representation -- so that a couple-dozen things continue working.

For testing, it helped me to try these commands at different points:

```
cv ev 'return CRM_Invoicing_Utils::isInvoicingEnabled();'
cv api setting.get return=invoicing
cv api setting.get return=contribution_invoice_settings

select * from civicrm_setting where name like 'contrib%' or name like 'invoic%';
```
@totten totten force-pushed the 5.25-contrib-virt-itemcanon branch from db5de31 to 84d5298 Compare April 29, 2020 23:34
…rtion

In practice, for the old setting `contribution_invoice_settings`, the
`invoicing` property could take one of two values:

1. If enabled, the value is an array with trueish subkey: `['invoicing'=>['invoicing'=>1]]`
2. If disabled, the value is a numeric `0`: `['invoicing'=>0]`

Following this means better compatibility.

The format is counter-intuitive, and the unit-test sets the value to enabled
using a more intuitive format (`['invoicing'=>1]`).  It's currently valid to
`set()` this way, but `get()`ting the value produces the
more-compatible/historical format.
@seamuslee001
Copy link
Contributor

Test fail unrelated

@seamuslee001 seamuslee001 merged commit 658971c into civicrm:5.25 Apr 30, 2020
@totten totten deleted the 5.25-contrib-virt-itemcanon branch May 15, 2020 23:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants