Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(api-abuse): Add duration_ms to individual queries in querylog #3432

Merged
merged 3 commits into from
Nov 22, 2022
Merged

fix(api-abuse): Add duration_ms to individual queries in querylog #3432

merged 3 commits into from
Nov 22, 2022

Conversation

enochtangg
Copy link
Member

Overview

The snuba querylog has a duration_ms which is the runtime of the entire snuba query. However, the runtime of the individual clickhouse query itself (time elapsed for Clickhouse to run a query on server) is not captured by querylog. This PR is responsible for adding this data to querylog.

Before State

Querylog always writes 0 in clickhouse_queries.duration_ms

After State

Querylog writes the elapsed time to run the clickhouse query in clickhouse_queries.duration_ms

Testing

Tested locally to ensure messages produced to querylog topics are correct and contain accurate fields.

@enochtangg enochtangg requested a review from a team as a code owner November 22, 2022 17:54
nikhars
nikhars requested changes Nov 22, 2022
View reviewed changes
snuba/datasets/processors/querylog_processor.py Outdated
@@ -93,6 +91,7 @@ def __extract_query_list(
groupby_columns.append(profile["groupby_cols"])
array_join_columns.append(profile["array_join_cols"])
bytes_scanned_columns.append(result_profile.get("bytes", 0))
duration_ms.append(result_profile.get("elapsed", 0))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The result returned from clickhouse is in seconds while the column being written to is in milliseconds. https://clickhouse-driver.readthedocs.io/en/latest/features.html?highlight=elapsed#query-execution-statistics. You should convert it to milliseconds before writing to querylog

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Does it make more sense to preserve the original nature of ClickhouseProfile and do the conversion in the querylog processor or before the message is produced to topic?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What you did seems fine to me.

@enochtangg enochtangg requested a review from nikhars November 22, 2022 19:52
@enochtangg enochtangg merged commit 6c0a3a5 into master Nov 22, 2022
@enochtangg enochtangg deleted the enoch/add_duration_ms_to_querylog branch November 22, 2022 20:30
enochtangg added a commit that referenced this pull request Nov 22, 2022
@enochtangg
Revert "fix(api-abuse): Add duration_ms to individual queries in quer…
48d159a 
…ylog (#3432)"

This reverts commit 6c0a3a5.
@enochtangg enochtangg mentioned this pull request Nov 22, 2022
Merged
enochtangg added a commit that referenced this pull request Nov 22, 2022
@enochtangg
Revert "fix(api-abuse): Add duration_ms to individual queries in quer…
779ddb8 
…ylog (#3432)" (#3443)

This reverts commit 6c0a3a5.
@enochtangg enochtangg mentioned this pull request Nov 23, 2022
Merged
onkar added a commit that referenced this pull request Nov 5, 2024
@dependabot @onkar
chore(deps): bump urllib3 from 2.2.2 to 2.2.3 (#6492)
b3e3a9c 
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.2.2 to 2.2.3.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/urllib3/urllib3/releases">urllib3's
releases</a>.</em></p>
<blockquote>
<h2>2.2.3</h2>
<h2>🚀 urllib3 is fundraising for HTTP/2 support</h2>
<p><a
href="https://sethmlarson.dev/urllib3-is-fundraising-for-http2-support">urllib3
is raising ~$40,000 USD</a> to release HTTP/2 support and ensure
long-term sustainable maintenance of the project after a sharp decline
in financial support for 2023. If your company or organization uses
Python and would benefit from HTTP/2 support in Requests, pip, cloud
SDKs, and thousands of other projects <a
href="https://opencollective.com/urllib3">please consider contributing
financially</a> to ensure HTTP/2 support is developed sustainably and
maintained for the long-haul.</p>
<p>Thank you for your support.</p>
<h2>Features</h2>
<ul>
<li>Added support for Python 3.13. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3473">#3473</a>)</li>
</ul>
<h2>Bugfixes</h2>
<ul>
<li>Fixed the default encoding of chunked request bodies to be UTF-8
instead of ISO-8859-1. All other methods of supplying a request body
already use UTF-8 starting in urllib3 v2.0. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3053">#3053</a>)</li>
<li>Fixed ResourceWarning on CONNECT with Python &lt; 3.11.4 by
backporting <a
href="https://redirect.github.com/python/cpython/issues/103472">python/cpython#103472</a>.
(`<a
href="https://redirect.github.com/urllib3/urllib3/issues/3252">#3252</a>)</li>
<li>Adjust tolerance for floating-point comparison on Windows to avoid
flakiness in CI (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3413">#3413</a>)</li>
<li>Fixed a crash where certain standard library hash functions were
absent in restricted environments. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3432">#3432</a>)</li>
<li>Fixed mypy error when adding to
<code>HTTPConnection.default_socket_options</code>. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3448">#3448</a>)</li>
</ul>
<h2>HTTP/2 (experimental)</h2>
<p>HTTP/2 support is still in early development.</p>
<ul>
<li>Excluded Transfer-Encoding: chunked from HTTP/2 request body (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3425">#3425</a>)</li>
<li>Added version checking for <code>h2</code> (<a
href="https://pypi.org/project/h2/">https://pypi.org/project/h2/</a>)
usage. Now only accepting supported h2 major version 4.x.x. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3290">#3290</a>)</li>
<li>Added a probing mechanism for determining whether a given target
origin supports HTTP/2 via ALPN. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3301">#3301</a>)</li>
<li>Add support for sending a request body with HTTP/2 (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3302">#3302</a>)</li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/urllib3/urllib3/compare/2.2.2...2.2.3">https://github.com/urllib3/urllib3/compare/2.2.2...2.2.3</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/urllib3/urllib3/blob/main/CHANGES.rst">urllib3's
changelog</a>.</em></p>
<blockquote>
<h1>2.2.3 (2024-09-12)</h1>
<h2>Features</h2>
<ul>
<li>Added support for Python 3.13.
(<code>[#3473](urllib3/urllib3#3473)
&lt;https://github.com/urllib3/urllib3/issues/3473&gt;</code>__)</li>
</ul>
<h2>Bugfixes</h2>
<ul>
<li>Fixed the default encoding of chunked request bodies to be UTF-8
instead of ISO-8859-1.
All other methods of supplying a request body already use UTF-8 starting
in urllib3 v2.0.
(<code>[#3053](urllib3/urllib3#3053)
&lt;https://github.com/urllib3/urllib3/issues/3053&gt;</code>__)</li>
<li>Fixed ResourceWarning on CONNECT with Python <!-- raw HTML omitted
-->`__)</li>
<li>Adjust tolerance for floating-point comparison on Windows to avoid
flakiness in CI
(<code>[#3413](urllib3/urllib3#3413)
&lt;https://github.com/urllib3/urllib3/issues/3413&gt;</code>__)</li>
<li>Fixed a crash where certain standard library hash functions were
absent in restricted environments.
(<code>[#3432](urllib3/urllib3#3432)
&lt;https://github.com/urllib3/urllib3/issues/3432&gt;</code>__)</li>
<li>Fixed mypy error when adding to
<code>HTTPConnection.default_socket_options</code>.
(<code>[#3448](urllib3/urllib3#3448)
&lt;https://github.com/urllib3/urllib3/issues/3448&gt;</code>__)</li>
</ul>
<h2>HTTP/2 (experimental)</h2>
<p>HTTP/2 support is still in early development.</p>
<ul>
<li>
<p>Excluded Transfer-Encoding: chunked from HTTP/2 request body
(<code>[#3425](urllib3/urllib3#3425)
&lt;https://github.com/urllib3/urllib3/issues/3425&gt;</code>__)</p>
</li>
<li>
<p>Added version checking for <code>h2</code> (<a
href="https://pypi.org/project/h2/">https://pypi.org/project/h2/</a>)
usage.</p>
<p>Now only accepting supported h2 major version 4.x.x.
(<code>[#3290](urllib3/urllib3#3290)
&lt;https://github.com/urllib3/urllib3/issues/3290&gt;</code>__)</p>
</li>
<li>
<p>Added a probing mechanism for determining whether a given target
origin
supports HTTP/2 via ALPN.
(<code>[#3301](urllib3/urllib3#3301)
&lt;https://github.com/urllib3/urllib3/issues/3301&gt;</code>__)</p>
</li>
<li>
<p>Add support for sending a request body with HTTP/2
(<code>[#3302](urllib3/urllib3#3302)
&lt;https://github.com/urllib3/urllib3/issues/3302&gt;</code>__)</p>
</li>
</ul>
<h2>Deprecations and Removals</h2>
<ul>
<li>Note for downstream distributors: the <code>_version.py</code> file
has been removed and is now created at build time by hatch-vcs.
(<code>[#3412](urllib3/urllib3#3412)
&lt;https://github.com/urllib3/urllib3/issues/3412&gt;</code>__)</li>
<li>Drop support for end-of-life PyPy3.8 and PyPy3.9.
(<code>[#3475](urllib3/urllib3#3475)
&lt;https://github.com/urllib3/urllib3/issues/3475&gt;</code>__)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/urllib3/urllib3/commit/2458bfcd3dacdf6c196e98d077fc6bb02a5fc1df"><code>2458bfc</code></a>
Release 2.2.3</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/9b25db6d00e43858d49303ae55c43bc4a9832668"><code>9b25db6</code></a>
Only attempt to publish for upstream</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/b9adeef8501180cd7d04cc3fb90bed4bbc34b1bb"><code>b9adeef</code></a>
Drop support for EOL PyPy3.8 and PyPy3.9</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/b1d4649d43375f11a3072b4d9b5d33425d123bae"><code>b1d4649</code></a>
Add explicit support for Python 3.13</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/cc42860721836febf3fb6ebb485ed27d7f80122d"><code>cc42860</code></a>
Bump cryptography from 42.0.4 to 43.0.1 (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3470">#3470</a>)</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/3dae2e9b30d2e39bf20daea2353aa7ef055640cf"><code>3dae2e9</code></a>
Bump pypa/gh-action-pypi-publish from 1.9.0 to 1.10.1 (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3469">#3469</a>)</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/1e94feb2a671bf28721114dfea1105a2c1f91788"><code>1e94feb</code></a>
Revert &quot;Add TLS settings for HTTP/2 (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3456">#3456</a>)&quot;
(<a
href="https://redirect.github.com/urllib3/urllib3/issues/3466">#3466</a>)</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/aa73abc7b22a4a67e0ee957f5a3031109f73d3d9"><code>aa73abc</code></a>
Bump actions/setup-python from 5.1.0 to 5.2.0 (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3468">#3468</a>)</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/abbfbcb1dd274fc54b4f0a7785fd04d59b634195"><code>abbfbcb</code></a>
Add 1.26.20 to changelog and make the publish workflow the same (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3464">#3464</a>)</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/d48061505e72271116c5a33b04dbca6273f2a737"><code>d480615</code></a>
Add TLS settings for HTTP/2 (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3456">#3456</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/urllib3/urllib3/compare/2.2.2...2.2.3">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=urllib3&package-manager=pip&previous-version=2.2.2&new-version=2.2.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Onkar Deshpande <onkar.deshpande@sentry.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nikhars nikhars nikhars approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@enochtangg @nikhars