dev/core/544 Add report support for filter on multiple contact subtypes

[elisseck]

Overview

Issue is described at https://lab.civicrm.org/dev/core/issues/544

Before

Reports do not support filtering on multiple contact subtypes. Examples:

In the Contact Summary report, filtering on contact subtype 'is one of' 'Student' will bring up contacts with subtype 'Student', but not contacts with multiple subtypes including 'Student'.

Similarly, filtering on contact subtype 'is one of' 'Student', 'Volunteer' will bring up contacts with subtype 'Student', but not with multiple subtypes including 'Student', and not with the exact subtypes 'Student', 'Volunteer'.

After

Filtering on contact subtype 'is one of' with contacts that have multiple subtypes, or using multiple filters works as expected e.g. 'Student', 'Volunteer' will bring up contacts with subtypes 'Student', 'Volunteer', and 'StudentVolunteer' etc.

Technical Details

This is kind of a bandaid PR for this functionality rather than changing how it's stored. I've added whereSubtypeClause() to the report form and called it if we've got the contact_subtype field to get the desired results.

[civibot]

(Standard links)

• If this is your first pull-request for CiviCRM, please browse CONTRIBUTING.md for information about the development and testing processes.
• If you are reviewing this pull-request, you may wish to consult the test sites and the Review Standards (long template, short template).

[seamuslee001]

@eileenmcnaughton will the OR here cause performance issues possibly?

[eileenmcnaughton]

I'll give it a whirl - but my first instinct here is that we are defining the field metadata incorrectly in some way or this wouldn't be needed

[elisseck]

@eileenmcnaughton I agree as commented in the issue I was surprised to see how this was stored. From my point of view using this function is kind of like "this is the result we're shooting for in terms of output, and if we can get there in another way without causing too much ruckus elsewhere... even better"

@seamuslee001 I didn't notice delay or anything but I also didn't do any real stress test and my dev environment has access to lots of memory, so that's a good point.

[eileenmcnaughton]

OK - I understand the issue now - but it makes me wonder - we also have custom fields that get stored this way - I wonder if this applies to them too....

[elisseck]

@eileenmcnaughton good question! I looked into this by creating a custom field that accepts multiple values (multi-select). Looks like instead of using IN (1, 2) or LIKE %1% OR LIKE %2% the report is using REGEXP '([[:cntrl:]]|^)1([[:cntrl:]]|$)|([[:cntrl:]]|^)2([[:cntrl:]]|$)' to get around the same 'single line with control characters' storage concept.

IANAE but I my instinct is that this is also slow if not slower unless you need to match some higher number of option values (each one adding an OR ... LIKE). I think i'll make some time to test that to see if this PR should use that strategy instead... but ultimately if we are concerned with performance I think we'd normalize these types of multi-option value fields. Honestly that's got my vote if it's possible but it's a much more daunting project.

[eileenmcnaughton]

@elisseck looks like the regex was added by @jitendrapurohit in conjunction with CRM-18803 - I would be interested in performance stats on it

[eileenmcnaughton]

' normalize these types of multi-option value field' - definitely too hard!

[elisseck]

@eileenmcnaughton understood about normalization I thought as much :).

I don't have access to fancy SQL performance tools at the moment... but I tried the queries with a custom field in the following manner on a local dev server:

• Created a custom multi-select field with 25 options

• Filled database with ~10000 contacts with quasi-random selections for the custom field

• Ran the entire summary report query in mysql as it gets generated in the developer tab for the REGEXP queries, and replacing the REGEXP with what would be generated in the LIKE %foo% OR... type of query:

  1. LIKE...OR searching for 'is one of' a single value

2. LIKE...OR searching for 'is one of' any of the 25 values

3. REGEXP searching for 'is one of' a single value

4. REGEXP searching for 'is one of' any of the 25 values

Seems to me for most use-cases they're going to be largely the same in terms of time... but in terms of what the optimizer thinks of as query cost or at scale with lots of values avoiding regexp seems to be the way to go.

[eileenmcnaughton]

I couldn't merge this without a test but I decided to write one for you in this case (which also gives a little cover to the grant detail write @agh1 did too.

I'll merge this (which does fix the issue) & push that up separately

[eileenmcnaughton]

related regression https://lab.civicrm.org/dev/report/issues/15