Adds DependsOnTags field to Rule

[gfs]

Fix #531.

After processing of rules DependsOnTags will be checked, at that time, all matches whose rule have depends_on_tags which are not all present in the total set of UniqueTags gathered during the full run will be removed in iterative batches until no more rules have missing depends on tags.

Adds tests for new functionality and rule verifier will check to make sure all tags which are depended on are present in the rule set, and that any overridden rules have at least all the depends_on_tags of their overrider, since overridden rules are not tracked, and thus cannot be restored during the depends checking step, resulting in a potentially confusing interaction.

SemVer changes. Public properties in the some objects have been changed to IList from various Array or List.

Sample rules with new syntax (pulled from the new test cases)

One Rule Depends on tags from another rule

[
    {
        "id": "SA000005",
        "name": "Testing.Rules.DependsOnTags.OneWay",
        "tags": [
            "Dependant"
        ],
        "depends_on_tags": ["Dependee"],
        "severity": "Critical",
        "description": "This rule finds windows 2000",
        "patterns": [
            {
                "pattern": "windows 2000",
                "type": "regex",
                "confidence": "High",
                "modifiers": [
                    "m"
                ],
                "scopes": [
                    "code"
                ]
            }
        ],
        "_comment": "This rule depends on the Dependee tag"
    },
    {
        "id": "SA000006",
        "name": "Testing.Rules.DependsOnTags.OneWay",
        "tags": [
            "Dependee"
        ],
        "severity": "Critical",
        "description": "This rule finds linux",
        "patterns": [
            {
                "pattern": "linux",
                "type": "regex",
                "confidence": "High",
                "modifiers": [
                    "m"
                ],
                "scopes": [
                    "code"
                ]
            }
        ],
        "_comment": "This rule sets the dependee tag"
    }
]

Two Rules Depend on each other

[
    {
        "id": "SA000005",
        "name": "Testing.Rules.DependsOnTags.TwoWay",
        "tags": [
            "RuleOne"
        ],
        "depends_on_tags": ["RuleTwo"],
        "severity": "Critical",
        "description": "This rule finds windows 2000",
        "patterns": [
            {
                "pattern": "windows 2000",
                "type": "regex",
                "confidence": "High",
                "modifiers": [
                    "m"
                ],
                "scopes": [
                    "code"
                ]
            }
        ],
        "_comment": "This rule depends on the RuleTwo tag"
    },
    {
        "id": "SA000006",
        "name": "Testing.Rules.DependsOnTags.TwoWay",
        "tags": [
            "RuleTwo"
        ],
        "depends_on_tags": ["RuleOne"],
        "severity": "Critical",
        "description": "This rule finds linux",
        "patterns": [
            {
                "pattern": "linux",
                "type": "regex",
                "confidence": "High",
                "modifiers": [
                    "m"
                ],
                "scopes": [
                    "code"
                ]
            }
        ],
        "_comment": "This rule depends on the RuleOne tag"
    }
]

You can also chain depends on

[
    {
        "id": "SA000001",
        "name": "Testing.Rules.DependsOnTags.Chain.A",
        "tags": [
            "Category.A"
        ],
        "severity": "Critical",
        "description": "This rule finds A",
        "patterns": [
            {
                "pattern": "A",
                "type": "regex",
                "confidence": "High",
                "modifiers": [
                    "m"
                ],
                "scopes": [
                    "code"
                ]
            }
        ],
        "_comment": ""
    },
    {
        "id": "SA000002",
        "name": "Testing.Rules.DependsOnTags.Chain.B",
        "tags": [
            "Category.B"
        ],
        "depends_on_tags": ["Category.A"],
        "severity": "Critical",
        "description": "This rule finds B",
        "patterns": [
            {
                "pattern": "B",
                "type": "regex",
                "confidence": "High",
                "modifiers": [
                    "m"
                ],
                "scopes": [
                    "code"
                ]
            }
        ],
        "_comment": ""
    },
    {
        "id": "SA000003",
        "name": "Testing.Rules.DependsOnTags.Chain.C",
        "tags": [
            "Category.C"
        ],
        "depends_on_tags": ["Category.B"],
        "severity": "Critical",
        "description": "This rule finds C",
        "patterns": [
            {
                "pattern": "C",
                "type": "regex",
                "confidence": "High",
                "modifiers": [
                    "m"
                ],
                "scopes": [
                    "code"
                ]
            }
        ],
        "_comment": ""
    }
]

[jacobmsft]

👍 Get Outlook for Android<https://aka.ms/AAb9ysg>

________________________________ From: Gabe Stocco ***@***.***> Sent: Friday, March 3, 2023 3:30:18 PM To: microsoft/ApplicationInspector ***@***.***> Cc: Suraj Jacob ***@***.***>; Mention ***@***.***> Subject: Re: [microsoft/ApplicationInspector] Adds DependsOnTags field to Rule (PR #533) @gfs commented on this pull request.

________________________________ In AppInspector/Commands/AnalyzeCommand.cs<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2FApplicationInspector%2Fpull%2F533%23discussion_r1125132757&data=05%7C01%7Csujacob%40microsoft.com%7Ca2bc5cec38ba4a34103108db1c3f3fe9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638134830227211180%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=64rLjF7iwq47VhjmdZJkAD%2FqgWBLSAXNeGreHZ7XMPY%3D&reserved=0>:

@@ -335,6 +335,17 @@ void VerifyFile(string filename)

} } + // Remove match records that don't have the matching depends on tags + List<MatchRecord> filteredMatchRecords = _metaDataHelper.Matches.Where(x => x.Rule?.DependsOnTags?.All(tag => _metaDataHelper.UniqueTags.ContainsKey(tag)) ?? true).ToList(); I did discover an edge case, but its actually from the Overrides not the ID. Overrides are removed on a per file basis as each file is scanned. DependsOnTags must be checked after all files are scanned, because it cares about tags in other files. Lets say Rule A matches and does not have a DependsOnTags. Rule B also matches and overrides Rule A, however it has a DependsOnTags for "TagC". At the end of the run, if TagC isn't present, Rule B will be removed from the matches set, but Rule A will not currently be restored. We do not keep track of the rules which were overridden and when, so this information is not available at the time we check DependsOnTags to remove rules with missing dependent tags. The easiest way to fix this might be in the verifier, to require that a rule which is targetted with an override must depend on at least all of the tags which the overrider depends on. — Reply to this email directly, view it on GitHub<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2FApplicationInspector%2Fpull%2F533%23discussion_r1125132757&data=05%7C01%7Csujacob%40microsoft.com%7Ca2bc5cec38ba4a34103108db1c3f3fe9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638134830227366979%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3eCwzJ9EySiXuzlda4DpKoPevgTF5KFKd3drVbu5wtk%3D&reserved=0>, or unsubscribe<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAG3WFL6UXNUXSYW6QOOQE5DW2J5IVANCNFSM6AAAAAAVMXCKIY&data=05%7C01%7Csujacob%40microsoft.com%7Ca2bc5cec38ba4a34103108db1c3f3fe9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638134830227366979%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=DRY2AaUmn7ksb0804fHrgBWf9vslmLKZx0u2xJxhQ9Q%3D&reserved=0>. You are receiving this because you were mentioned.Message ID: ***@***.***>