Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for fetching consumer offsets stored in Kafka to monitor_unlisted_consumer_groups #3957

Merged
merged 5 commits into from
Oct 1, 2019
Merged

Add support for fetching consumer offsets stored in Kafka to monitor_unlisted_consumer_groups #3957

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b7913cb
Add Kafka support for monitor_unlisted_consumer_groups
jeffwidman Jun 20, 2019
42c91f7
Merge branch 'master' into pr/3957
ofek Oct 1, 2019
4b6e742
1.4.7 was released
ofek Oct 1, 2019
5ab5deb
Update agent_requirements.in
ofek Oct 1, 2019
d2bd8cf
for now..
ofek Oct 1, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion datadog_checks_base/datadog_checks/base/data/agent_requirements.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ in-toto==0.3.0
ipaddress==1.0.22; python_version < '3.0'
jaydebeapi==1.1.1
jpype1==0.7.0
kafka-python==1.4.4; sys_platform != 'win32'
kafka-python==1.4.7; sys_platform != 'win32'
kazoo==2.6.1; sys_platform != 'win32'
kubernetes==8.0.1
ldap3==2.5
Expand Down
27 changes: 18 additions & 9 deletions kafka_consumer/datadog_checks/kafka_consumer/data/conf.yaml.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,11 +47,13 @@ instances:
# kafka_client_api_version: "2.3.0"

## @param consumer_groups - object - optional
## When using Kafka to store consumer offsets each level is mandatory.
## When using zookeeper to store consumer offsets each level is optional.
## Any empty values are fetched from Zookeeper. You can have empty partitions (example: <CONSUMER_NAME_2>),
## topics (example: <CONSUMER_NAME_3>), and even consumer_groups. If you omit
## consumer_groups, you must set 'monitor_unlisted_consumer_groups': true.
## Each level is optional. Any empty values are fetched from the Kafka cluster.
## You can have empty partitions (example: <CONSUMER_NAME_2>), topics (example: <CONSUMER_NAME_3>),
## and even consumer_groups. If you omit consumer_groups, you must set `monitor_unlisted_consumer_groups` to true.
##
## Deprecation notice: Omitting various levels works for zookeeper-based consumers. However, all
## functionality related to fetching offsets from Zookeeper is deprecated and will be removed from this
## check in the future.
#
# consumer_groups:
# <CONSUMER_NAME_1>:
Expand All @@ -61,10 +63,17 @@ instances:
# <CONSUMER_NAME_3>: {}

## @param monitor_unlisted_consumer_groups - boolean - optional
## Setting monitor_unlisted_consumer_groups to `true` tells the check to
## discover and fetch all offsets for all consumer groups stored in zookeeper.
## While this is convenient, it can also put a lot of load on zookeeper.
## If this is not set to true, you must specify consumer_groups.
## Setting monitor_unlisted_consumer_groups to `true` tells the check to discover all consumer groups
## and fetch all their known offsets. If this is not set to true, you must specify consumer_groups.
##
## WARNING: This feature requires that your Kafka brokers be version >= 0.10.2. It is impossible to
## support this feature on older brokers because they do not provide a way to determine the mapping
## of consumer groups to topics. For details, see KIP-88. For older Kafka brokers, the consumer groups
## must be specified. This requirement only applies to the brokers, not the consumers--they can be any version.
##
## Deprecation notice: This feature works for Zookeeper-based consumers. However, all
## functionality related to fetching offsets from Zookeeper is deprecated and will be removed from this
## check in the future.
#
# monitor_unlisted_consumer_groups: false

Expand Down
Loading