Use simdjson for json_parse #7658
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for meta-velox canceled.
|
|
This pull request was exported from Phabricator. Differential Revision: D51469435 |
Yuhta
added a commit
to Yuhta/velox
that referenced
this pull request
Nov 20, 2023
Summary: In `json_parse` when the input is invalid, we throw exception and it's slow (both the creation and throwing). To avoid creating or throwing the exception, we switch the implementation to `simdjson` and set a pre-canned exception when the input is invalid. This reduces the CPU time in some queries (with JSON validity check in filter) by more than 20 times (from 2.34 days to 2.68 hours). Differential Revision: D51469435
e9e8663 to
14aa377
Compare
|
This pull request was exported from Phabricator. Differential Revision: D51469435 |
Yuhta
added a commit
to Yuhta/velox
that referenced
this pull request
Nov 20, 2023
Summary: In `json_parse` when the input is invalid, we throw exception and it's slow (both the creation and throwing). To avoid creating or throwing the exception, we switch the implementation to `simdjson` and set a pre-canned exception when the input is invalid. This reduces the CPU time in some queries (with JSON validity check in filter) by more than 20 times (from 2.34 days to 2.68 hours). Differential Revision: D51469435
14aa377 to
ae4ad91
Compare
|
This pull request was exported from Phabricator. Differential Revision: D51469435 |
mbasmanova
approved these changes
Nov 22, 2023
PHILO-HE
reviewed
Nov 22, 2023
| @@ -114,6 +139,9 @@ class JsonParseFunction : public exec::VectorFunction { | |||
| .argumentType("varchar") | |||
| .build()}; | |||
| } | |||
|
|
|||
| private: | |||
| mutable simdjson::dom::parser parser_; | |||
There was a problem hiding this comment.
Yes I tried ondemand parser first and find it is not able to invalidate some malformed JSON.
Summary: In `json_parse` when the input is invalid, we throw exception and it's slow (both the creation and throwing). To avoid creating or throwing the exception, we switch the implementation to `simdjson` and set a pre-canned exception when the input is invalid. This reduces the CPU time in some queries (with JSON validity check in filter) by more than 20 times (from 2.34 days to 2.68 hours). Reviewed By: mbasmanova Differential Revision: D51469435
ae4ad91 to
a323eec
Compare
Yuhta
added a commit
to Yuhta/velox
that referenced
this pull request
Nov 22, 2023
Summary: In `json_parse` when the input is invalid, we throw exception and it's slow (both the creation and throwing). To avoid creating or throwing the exception, we switch the implementation to `simdjson` and set a pre-canned exception when the input is invalid. This reduces the CPU time in some queries (with JSON validity check in filter) by more than 20 times (from 2.34 days to 2.68 hours). Reviewed By: mbasmanova Differential Revision: D51469435
|
This pull request was exported from Phabricator. Differential Revision: D51469435 |
a323eec to
75fc8fd
Compare
|
This pull request was exported from Phabricator. Differential Revision: D51469435 |
|
This pull request has been merged in 0460044. |
|
Conbench analyzed the 1 benchmark run on commit There were no benchmark performance regressions. 🎉 The full Conbench report has more details. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary: In
json_parsewhen the input is invalid, we throw exception and it's slow (both the creation and throwing). To avoid creating or throwing the exception, we switch the implementation tosimdjsonand set a pre-canned exception when the input is invalid. This reduces the CPU time in some queries (with JSON validity check in filter) by more than 20 times (from 2.34 days to 2.68 hours).Differential Revision: D51469435