Implement regex for consumer_groups

[yzhan289]

Remake of #14193

What does this PR do?

This PR adds a new config option called consumer_groups_regex. This functions similarly to consumer_groups to allow configuration of consumer groups, topics, and partitions to monitor, however this allows regex strings to be used.

This will be merged to master after #13918 is merged.

Motivation

Feature request

Additional Notes

When monitor_unlisted_consumer_groups is set to True, then both consumer_groups and consumer_groups_regex will be ignored.

When consumer_groups and consumer_groups_regex are both used, then consumer groups from both config options will be collected.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• PR title must be written as a CHANGELOG entry (see why)
• Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
• PR must have changelog/ and integration/ labels attached
• If the PR doesn't need to be tested during QA, please add a qa/skip-qa label.

[codecov]

Codecov Report

Merging #14382 (7987b33) into master (a41ad12) will increase coverage by 0.08%.
The diff coverage is 97.53%.

Flag	Coverage Δ
kafka_consumer	93.97% <97.53%> (+12.14%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

[mwarkentin]

I think it would be helpful to provide some examples of proper regexes here - particularly to provide examples of keys like this in yaml (do they need to be wrapped in " or can you drop a raw regex as a key)?

[yzhan289]

Hey @mwarkentin sorry just saw this. I've made a PR changing these examples to be more realistic: #14533

[mwarkentin]

Thanks @yzhan289 !

[mwarkentin]

I was also wondering if you know when this will be released? Seems like datadog-agent is still on the 2.16.x series for the kafka consumer integration?

[yzhan289]

I was also wondering if you know when this will be released? Seems like datadog-agent is still on the 2.16.x series for the kafka consumer integration?

The integration will be a part of Agent 7.45.0, which will be released in the upcoming few weeks.