dev/core#3890 Rebuild smart groups included in mailing on scheduling

[larssandergreen]

Overview

Smart group recipients should be totally up to date when a mailing is scheduled.

Before

Smart group recipients for a mailing were based on the smart group cache, potentially resulting in out of date recipients.

After

Smart groups that are included or excluded from a mailing are rebuilt at time of scheduling, so recipients are fully up to date. Parent groups are rebuilt as well. This also includes hidden smart groups (mailings created from search results using All NNN Contacts).

Technical Details

This may not look like the most logical place for this to happen (but if this is in getRecipients, it will run a dozen times by the time a mailing is created and sent) or the simplest way to go about it (you'd think the params we'd need would be there - they are earlier, but they aren't at scheduling time).

[civibot]

(Standard links)

• If this is your first pull-request for CiviCRM, please browse CONTRIBUTING.md for information about the development and testing processes.
• If you are reviewing this pull-request, you may wish to consult the test sites and the Review Standards (long template, short template).

[agileware-justin]

Smart groups that are included or excluded from a mailing are rebuilt at time of scheduling, so recipients are fully up to date.

@larssandergreen thanks very much for working on this issue. Would it be possible to move the rebuild recipients list to just before the mailing is sent, rather than scheduled? The reason being that end users expect that the included / excluded mailing groups will be used at the time the mailing is sent, not when it was scheduled.

This is particularly relevant if you schedule the mailing to be sent few days / weeks / months - there can be a LOT of changes to the included / excluded mailing groups. And not using the latest recipient list can be very problematic.

For example: monthly membership newsletter is scheduled on 14th and will be sent out on the 21st to all current members. 50 new members join between the 14th and 20th. On the 21st the newsletter is sent, those 50 new members do not receive the newsletter and complain to the management committee. The management committee have reduced confidence that CiviCRM is functioning correctly.

There is an extension available which did solve this CiviCRM problem, however this should really be a core feature change not an option, https://github.com/3sd/civicrm-recalculate-recipients

Thanks for listening 🎧

[seamuslee001]

@agileware-justin whilst that probably would work with most systems (because in most systems everyone has visibility of all contacts), if you had a system that had a set of ACLs it might be a bit more difficult because I presume the user that would be triggering the re-calculation at sending time would likely be the cron user and not the possibly ACLed user that scheduled the mailing so the 'which contacts can this user see' becomes a problem.

[larssandergreen]

@agileware-justin I totally agree with you that recipients should be determined at sending time. More than one person in my organization has been pretty surprised when I explained that's not how this works.

It does seem like a somewhat controversial issue though and every time it is discussed, it seems to gather a bunch of different opinions on what should happen (even in the issue for this change). I'd welcome a full discussion on how this should work and if there's agreement to change it, would be willing to help out with that. It does seem like at the end of the day, most people probably expect the recipients to be calculated at send time and we should meet that expectation. Leaving smart groups aside, it's even strange that if you remove a bunch of contacts from a group after scheduling but before sending, they will still receive a mailing.

But this change here in this PR is pretty simple, just rebuilding the smart group caches for the relevant groups right before building the recipients list. If the building of the recipients list is changed from scheduling to sending time, then this change should be moved along with it.

[larssandergreen]

@seamuslee001 We could, in theory, manually add the appropriate where clauses to the queries that select the recipients based on the approver_id for the mailing.

[agileware-justin]

whilst that probably would work with most systems (because in most systems everyone has visibility of all contacts), if you had a system that had a set of ACLs it might be a bit more difficult because I presume the user that would be triggering the re-calculation at sending time would likely be the cron user and not the possibly ACLed user that scheduled the mailing so the 'which contacts can this user see' becomes a problem.

I guess you're referring to the Australian Greens here, in which case sure I understand their specific requirement for restricting mailings.

Storing the mailing user id and/or mailing approver id and using this for querying the recipients at the time of send would be the logical change here. Sure you've thought of that too @seamuslee001

[tschuettler]

@agileware-justin there is a dedicated issue for this: https://lab.civicrm.org/dev/core/-/issues/2351 following an earlier discussion at: https://lists.civicrm.org/lists/arc/civicrm-dev/2017-12/msg00012.html
There are legitimate use cases for both recalculating at time of scheduling and at time of sending the mail.

[civicrm-builder]

Can one of the admins verify this patch?

[larssandergreen]

jenkins test this please

[larssandergreen]

@agileware-justin @seamuslee001 @tschuettler Anyone able to give this a review so we can get it merged? Just to recap, there has been some discussion here about the age-old question of when the mailing recipients should be calculated, but all this PR does is make sure the smart groups are up to date when the recipients are calculated, to avoid cached smart groups causing unexpected recipients. This doesn't change when the recipients are calculated.

[agileware-justin]

@larssandergreen yep can do next week. Will have some free time to review.

[agileware-justin]

Tested looks good to me, thanks @larssandergreen

CiviCRM Review Standards

• General standards
  • (r-explain) Pass
  • (r-user) Pass:
    Behaviour matches the CiviCRM End User documentation, see https://docs.civicrm.org/user/en/latest/email/mass-mailings-using-civimail/#choosing-recipients-when-mailing-recipients-are-set
  • (r-doc) Pass
  • (r-run) Pass:
    Test 1: Created a Mailing, set Recipients based on Smart Group. Scheduled Mailing. Changed some Contacts to meet Smart Group criteria. When Mailing was sent the new Contacts did NOT receive the email. This is correct (but I do not like!)
    Test 2: Created a Mailing, set Recipients based on Smart Group. Saved Mailing. Changed some Contacts to meet Smart Group criteria. When Mailing was Scheduled the Smart Group was re-calculated and included the new Contacts. When Mailing was sent the new Contacts received the email. This is correct.
• Developer standards
  • (r-tech) Pass:
    The change reflects the end user documentation, so it is likely already assumed by CiviCRM Developers and CiviCRM End Users alike that this is the current behaviour.
  • (r-code) Pass
  • (r-maint) Pass
  • (r-test) Pass

[agileware-justin]

@eileenmcnaughton ready for you to do what you do best 😄

[larssandergreen]

Thanks for the review, @agileware-justin!

[eileenmcnaughton]

Thanks @larssandergreen & @agileware-justin