fix: Fix the memory reclaim bytes for hash join #11642
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for meta-velox canceled.
|
|
This pull request was exported from Phabricator. Differential Revision: D66437719 |
tanjialiang
reviewed
Nov 25, 2024
tanjialiang
reviewed
Nov 25, 2024
xiaoxmeng
added a commit
to xiaoxmeng/velox
that referenced
this pull request
Nov 26, 2024
) Summary: Both hash join and probe does the coordinated spill so we shouldn't report the reclaimed bytes from a single node but shall report from the plan node. Also probe side spill might spill built table from join side and the memory is actually reclaimed from build side pool instead of probe side. This PR also removes the unused wait for spill state from hash build Differential Revision: D66437719
aa2fafd to
157188f
Compare
|
This pull request was exported from Phabricator. Differential Revision: D66437719 |
xiaoxmeng
added a commit
to xiaoxmeng/velox
that referenced
this pull request
Nov 26, 2024
) Summary: Both hash join and probe does the coordinated spill so we shouldn't report the reclaimed bytes from a single node but shall report from the plan node. Also probe side spill might spill built table from join side and the memory is actually reclaimed from build side pool instead of probe side. This PR also removes the unused wait for spill state from hash build Differential Revision: D66437719
157188f to
01f2833
Compare
|
This pull request was exported from Phabricator. Differential Revision: D66437719 |
xiaoxmeng
added a commit
to xiaoxmeng/velox
that referenced
this pull request
Nov 26, 2024
) Summary: Both hash join and probe does the coordinated spill so we shouldn't report the reclaimed bytes from a single node but shall report from the plan node. Also probe side spill might spill built table from join side and the memory is actually reclaimed from build side pool instead of probe side. This PR also removes the unused wait for spill state from hash build Differential Revision: D66437719
01f2833 to
66ddc22
Compare
|
This pull request was exported from Phabricator. Differential Revision: D66437719 |
xiaoxmeng
added a commit
to xiaoxmeng/velox
that referenced
this pull request
Nov 26, 2024
) Summary: Both hash join and probe does the coordinated spill so we shouldn't report the reclaimed bytes from a single node but shall report from the plan node. Also probe side spill might spill built table from join side and the memory is actually reclaimed from build side pool instead of probe side. This PR also removes the unused wait for spill state from hash build Differential Revision: D66437719
66ddc22 to
978be89
Compare
|
This pull request was exported from Phabricator. Differential Revision: D66437719 |
tanjialiang
approved these changes
Nov 26, 2024
bikramSingh91
approved these changes
Nov 26, 2024
978be89 to
a14fd2f
Compare
xiaoxmeng
added a commit
to xiaoxmeng/velox
that referenced
this pull request
Nov 27, 2024
) Summary: Both hash join and probe does the coordinated spill so we shouldn't report the reclaimed bytes from a single node but shall report from the plan node. Also probe side spill might spill built table from join side and the memory is actually reclaimed from build side pool instead of probe side. This PR also removes the unused wait for spill state from hash build Reviewed By: bikramSingh91, tanjialiang Differential Revision: D66437719
|
This pull request was exported from Phabricator. Differential Revision: D66437719 |
xiaoxmeng
added a commit
to xiaoxmeng/velox
that referenced
this pull request
Nov 27, 2024
) Summary: Both hash join and probe does the coordinated spill so we shouldn't report the reclaimed bytes from a single node but shall report from the plan node. Also probe side spill might spill built table from join side and the memory is actually reclaimed from build side pool instead of probe side. This PR also removes the unused wait for spill state from hash build Reviewed By: bikramSingh91, tanjialiang Differential Revision: D66437719
a14fd2f to
b739c19
Compare
|
This pull request was exported from Phabricator. Differential Revision: D66437719 |
xiaoxmeng
added a commit
to xiaoxmeng/velox
that referenced
this pull request
Nov 27, 2024
) Summary: Both hash join and probe does the coordinated spill so we shouldn't report the reclaimed bytes from a single node but shall report from the plan node. Also probe side spill might spill built table from join side and the memory is actually reclaimed from build side pool instead of probe side. This PR also removes the unused wait for spill state from hash build Reviewed By: bikramSingh91, tanjialiang Differential Revision: D66437719
b739c19 to
8ea38ee
Compare
|
This pull request was exported from Phabricator. Differential Revision: D66437719 |
xiaoxmeng
added a commit
to xiaoxmeng/velox
that referenced
this pull request
Nov 27, 2024
) Summary: Both hash join and probe does the coordinated spill so we shouldn't report the reclaimed bytes from a single node but shall report from the plan node. Also probe side spill might spill built table from join side and the memory is actually reclaimed from build side pool instead of probe side. This PR also removes the unused wait for spill state from hash build Reviewed By: bikramSingh91, tanjialiang Differential Revision: D66437719
8ea38ee to
0702c8c
Compare
|
This pull request was exported from Phabricator. Differential Revision: D66437719 |
) Summary: Both hash join and probe does the coordinated spill so we shouldn't report the reclaimed bytes from a single node but shall report from the plan node. Also probe side spill might spill built table from join side and the memory is actually reclaimed from build side pool instead of probe side. This PR also removes the unused wait for spill state from hash build Reviewed By: bikramSingh91, tanjialiang Differential Revision: D66437719
0702c8c to
fbfb1c1
Compare
|
This pull request was exported from Phabricator. Differential Revision: D66437719 |
|
This pull request has been merged in 1ce3c7a. |
TongWei1105
pushed a commit
to TongWei1105/velox
that referenced
this pull request
Dec 3, 2024
) Summary: Pull Request resolved: facebookincubator#11642 Both hash join and probe does the coordinated spill so we shouldn't report the reclaimed bytes from a single node but shall report from the plan node. Also probe side spill might spill built table from join side and the memory is actually reclaimed from build side pool instead of probe side. This PR also removes the unused wait for spill state from hash build Reviewed By: bikramSingh91, tanjialiang Differential Revision: D66437719 fbshipit-source-id: 4ac7bb9cf87b4346d234ef5f7c04ed64ee12d249
athmaja-n
pushed a commit
to athmaja-n/velox
that referenced
this pull request
Jan 10, 2025
) Summary: Pull Request resolved: facebookincubator#11642 Both hash join and probe does the coordinated spill so we shouldn't report the reclaimed bytes from a single node but shall report from the plan node. Also probe side spill might spill built table from join side and the memory is actually reclaimed from build side pool instead of probe side. This PR also removes the unused wait for spill state from hash build Reviewed By: bikramSingh91, tanjialiang Differential Revision: D66437719 fbshipit-source-id: 4ac7bb9cf87b4346d234ef5f7c04ed64ee12d249
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Both hash join and probe does the coordinated spill so we shouldn't report the reclaimed bytes from a single node
but shall report from the plan node. Also probe side spill might spill built table from join side and the memory is
actually reclaimed from build side pool instead of probe side.
This PR also removes the unused wait for spill state from hash build
Differential Revision: D66437719