Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable socket wakeup when sending requests #13221

Merged
merged 3 commits into from
Nov 18, 2022

Conversation

yzhan289
Copy link
Contributor

@yzhan289 yzhan289 commented Oct 27, 2022

What does this PR do?

This PR disables socket wakeup when sending requests to the kafka-python library we use for the kafka_consumer check. Since our check runs in a single thread, it doesn't need to do a socket wakeup. This was also causing issues for set ups with high number of brokers per cluster:

  File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/kafka_consumer/new_kafka_consumer.py", line 64, in check
    self._get_consumer_offsets()
  File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/kafka_consumer/new_kafka_consumer.py", line 363, in _get_consumer_offsets
    list_groups_future = self.kafka_client._list_consumer_groups_send_request(broker.nodeId)
  File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/kafka/admin/client.py", line 1116, in _list_consumer_groups_send_request
    return self._send_request_to_node(broker_id, request)
  File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/kafka/admin/client.py", line 368, in _send_request_to_node
    return self._client.send(node_id, request)
  File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/kafka/client_async.py", line 546, in send
    self.wakeup()
  File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/kafka/client_async.py", line 935, in wakeup
    raise Errors.KafkaTimeoutError()

Motivation

Additional Notes

dpkp/kafka-python#1761 added the ability to disable socket wakeups with the wakeup argument to fix the KafkaTimeoutError error, although the fix wasn't being applied to the kafka-python function _send_request_to_node() which we use, so we needed to reimplement it by hand.

Here is the original Github issue: dpkp/kafka-python#1760

I've also opened a Github PR on kafka-python to add the wakeup argument to the _send_request_to_node() function: dpkp/kafka-python#2335

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • PR title must be written as a CHANGELOG entry (see why)
  • Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
  • PR must have changelog/ and integration/ labels attached
  • If the PR doesn't need to be tested during QA, please add a qa/skip-qa label.

@codecov
Copy link

codecov bot commented Oct 27, 2022

Codecov Report

Merging #13221 (4d74a9f) into master (e07c46a) will decrease coverage by 0.00%.
The diff coverage is 53.84%.

Flag Coverage Δ
kafka_consumer 84.06% <53.84%> (-0.35%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

…13265)

* disable wakeup list groups and send

* add comment

* fix getting kafka version, call patched method

* clean up

* fix getting ListGroupsRequest by version
@iliakur
Copy link
Contributor

iliakur commented Nov 7, 2022

I wonder: since it's clearly hard to come up with tests for this, we could try to add a big comment that explains the rationale for the changes (link to support case etc). Just so that we know exactly which parts we shouldn't touch in the future, which is what the tests would tell us as well.

Copy link
Member

@FlorentClarret FlorentClarret left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me but left a nit comment

@yzhan289 yzhan289 merged commit e773851 into master Nov 18, 2022
@yzhan289 yzhan289 deleted the az/kafka-consumer-disable-wakeup branch November 18, 2022 20:55
iliakur pushed a commit that referenced this pull request Nov 23, 2022
* Disable socket wakeup when sending requests

* Disable socket wakeup when listing consumer groups and send request (#13265)

* disable wakeup list groups and send

* add comment

* fix getting kafka version, call patched method

* clean up

* fix getting ListGroupsRequest by version

* Add comment

Co-authored-by: Fanny Jiang <fanny.jiang@datadoghq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants