Enable single rule disjunction

[GiaJordan]

Changelog

This PR enables single rule disjunction for the existing validation rules through the IsNA rule modifier. It can be used with each rule except the recommended rule by including it in the specification for validation rules as another rule (ie with the :: operator).

When in use, Validation rules will raise an error or warning or no message as appropriate for each metadata entry that is provided, and will not raise any message for metadata entries that are omitted. The attribute should be marked as FALSE for the REQUIRED column in the data model to use this modifier.

A data model specific to these requirements has been created to monitor the status of each rule combined with this modifier and tests have been created as well.

---

Note for Reviewers:

The line endings for some files were changed and so the diffs can show entire file changes as part of this PR. Before you begin reviewing, under Files changed>Settings toggle Hide Whitespace to ON to get an accurate record of what actual code was changed.

[GiaJordan]

@andrewelamb thank you for your comments but this PR was not ready for review; it is still a work in progress. #1366 was what I presented in code review yesterday; this is the one I said I would tag you and @/mialy-defelice in when it was complete and ready to be reviewed

[andrewelamb]

Oops! Sorry about that, you are right.

[mialy-defelice]

This is minor but can you change this naming to something more descriptive? The way it reads is a bit confusing.
Maybe something like missing_value_model_DMGE

[mialy-defelice]

same comment as above but in this case something like: missing_value_model_MetadataModel

[mialy-defelice]

@GiaJordan
When running the following (also happens without -rr):

schematic model -c config.yml validate -mp tests/data/mock_manifests/Invalid_Missing_Value_Test_Manifest.csv -dt MockComponent -rr

With the model:
"tests/data/example.missing.value.model.csv"
I got the following error:

['NA', 'Check Regex List', "For attribute Check Regex List, the provided validation rules (['list strict error', 'regex match [a-f] error', 'IsNa']) .have too many entries. We currently only specify two rules ('list :: another_rule').",

The multiple types error handling should be updated to allow for a rule combo to be paired with isNA.

[mialy-defelice]

Also if you run:

schematic model -c config.yml validate -mp tests/data/mock_manifests/Invalid_Missing_Value_Test_Manifest.csv -dt MockComponent -rr

with the example.model.jsonld
It throws this error:

File "/Users/mialydefelice/Library/Caches/pypoetry/virtualenvs/schematicpy-rs9bwhSl-py3.9/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/mialydefelice/Library/Caches/pypoetry/virtualenvs/schematicpy-rs9bwhSl-py3.9/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/Users/mialydefelice/Library/Caches/pypoetry/virtualenvs/schematicpy-rs9bwhSl-py3.9/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/mialydefelice/Library/Caches/pypoetry/virtualenvs/schematicpy-rs9bwhSl-py3.9/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/mialydefelice/Library/Caches/pypoetry/virtualenvs/schematicpy-rs9bwhSl-py3.9/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/mialydefelice/Library/Caches/pypoetry/virtualenvs/schematicpy-rs9bwhSl-py3.9/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/mialydefelice/Library/Caches/pypoetry/virtualenvs/schematicpy-rs9bwhSl-py3.9/lib/python3.9/site-packages/click/decorators.py", line 45, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/Users/mialydefelice/Documents/schematic_e/schematic/schematic/models/commands.py", line 263, in validate_manifest
    errors, warnings = metadata_model.validateModelManifest(
  File "/Users/mialydefelice/Documents/schematic_e/schematic/schematic/models/metadata.py", line 275, in validateModelManifest
    errors, warnings, manifest = validate_all(
  File "/Users/mialydefelice/Documents/schematic_e/schematic/schematic/models/validate_manifest.py", line 317, in validate_all
    manifest, vmr_errors, vmr_warnings = vm.validate_manifest_rules(
  File "/Users/mialydefelice/Documents/schematic_e/schematic/schematic/models/validate_manifest.py", line 238, in validate_manifest_rules
    vr_errors, vr_warnings = validation_method(
  File "/Users/mialydefelice/Documents/schematic_e/schematic/schematic/models/validate_attribute.py", line 765, in type_validation
    if bool(value) and not isinstance(value, specified_type[val_rule]):
  File "missing.pyx", line 392, in pandas._libs.missing.NAType.__bool__
TypeError: boolean value of NA is ambiguous

Im guessing that in TestManifestValidation It should run in combo, testing the invalid_missing_manifest against the standard example model. In this case you can check for expected errors. Should run in other combos as well to ensure coverage.

[mialy-defelice]

For function clarity can you move this into its own function?

[mialy-defelice]

Will deal with these functions when this branch merges with the matchNone branch.

[mialy-defelice]

@GiaJordan why is ths not being merged into develop?

This will now be merged directly to dev.

[mialy-defelice]

Also if you run:

schematic model -c config.yml validate -mp tests/data/mock_manifests/Invalid_Missing_Value_Test_Manifest.csv -dt MockComponent -rr

...

Fixed this issue.

[mialy-defelice]

@GiaJordan When running the following (also happens without -rr):

schematic model -c config.yml validate -mp tests/data/mock_manifests/Invalid_Missing_Value_Test_Manifest.csv -dt MockComponent -rr

With the model: "tests/data/example.missing.value.model.csv" I got the following error:

['NA', 'Check Regex List', "For attribute Check Regex List, the provided validation rules (['list strict error', 'regex match [a-f] error', 'IsNa']) .have too many entries. We currently only specify two rules ('list :: another_rule').",

The multiple types error handling should be updated to allow for a rule combo to be paired with isNA.

Okay, looking through the tests it looks like this is expected, but Its unclear why... need to investigate further...

[mialy-defelice]

Okay, looking through the tests it looks like this is expected, but Its unclear why... need to investigate further...

Okay, I think this is bc the error was anticipated but decided that it was within the acceptance criteria. We are going to allow NA to be used with other rule combos, so for now will need to adjust the number of allowable rules, update testing to no longer allow this error to be considered passing.

[sonarqubecloud]

Quality Gate passed

Issues
75 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud