Implement regex for consumer_groups

[yzhan289]

What does this PR do?

This PR adds a new config option called consumer_groups_regex. This functions similarly to consumer_groups to allow configuration of consumer groups, topics, and partitions to monitor, however this allows regex strings to be used.

This will be merged to master after #13918 is merged.

Motivation

Feature request

Additional Notes

When monitor_unlisted_consumer_groups is set to True, then both consumer_groups and consumer_groups_regex will be ignored.

When consumer_groups and consumer_groups_regex are both used, then consumer groups from both config options will be collected.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• PR title must be written as a CHANGELOG entry (see why)
• Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
• PR must have changelog/ and integration/ labels attached
• If the PR doesn't need to be tested during QA, please add a qa/skip-qa label.

[yzhan289]

Current dependencies on/for this PR:

• master
  • PR Revamp Kafka consumer check #13918
    • PR Implement OAuth #14184
    • PR Add a tls e2e env and implement it #14137
      • PR Add a kerberos e2e env #14120
    • PR Implement OAuth config #14247
    • PR Remove some unused functions #14145
    • PR Update all the tests to use the kafka_instance instead of a custom one #14144
    • PR Refactor the confluent kafka client #14158
    • PR Implement regex for consumer_groups #14193 👈
    • PR Implement the request_metadata_update method #14152
    • PR Remove the get_dict methods from the clients #14149
    • PR Manually build confluent-kafka in the test env #14173

This comment was auto-generated by Graphite.

[codecov]

Codecov Report

Merging #14193 (5a02fa3) into AI-2904/kafka-consumer-revamp (558307c) will increase coverage by 0.75%.
The diff coverage is 97.50%.

❗ Current head 5a02fa3 differs from pull request most recent head 3d7ac7e. Consider uploading reports for the commit 3d7ac7e to get more accurate results

Flag	Coverage Δ
datadog_checks_dev	82.71% <ø> (+2.60%)	⬆️
kafka_consumer	95.74% <97.50%> (+2.35%)	⬆️
kubelet	90.80% <ø> (+0.11%)	⬆️
process	85.42% <ø> (+0.28%)	⬆️
temporal	?

Flags with carried forward coverage won't be shown. Click here to find out more.

[yzhan289]

Skip legacy implementation from regex tests

[rtrieu]

LGTM

[FlorentClarret]

👍 I left some comments

[github-actions]

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

[github-actions]

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

[github-actions]

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

[github-actions]

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

[yzhan289]

Merging the revamp PR messed up the Git history of this PR, will investigate...