Properly report lazy loaded inputBytes #11097
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for meta-velox canceled.
|
|
This pull request was exported from Phabricator. Differential Revision: D63414708 |
xiaoxmeng
approved these changes
Sep 26, 2024
velox/exec/tests/TableScanTest.cpp
Outdated
| auto task = assertQuery(op, {filePath}, "select c0 % 3 from tmp"); | ||
|
|
||
| // Ensure lazy stats are attributed to table scan. | ||
| auto stats = task->taskStats(); |
velox/vector/LazyVector.cpp
Outdated
| @@ -41,7 +45,9 @@ void VectorLoader::load( | |||
| vector_size_t resultSize, | |||
| VectorPtr* result) { | |||
| { | |||
| DeltaCpuWallTimer timer([&](auto& delta) { writeIOTiming(delta); }); | |||
| DeltaCpuWallTimer lazyIoRecorder([&](auto& cpuDelta) { | |||
There was a problem hiding this comment.
Can we have a new class for this as it now does delta cpu and io stats tracking. Maybe LazyIoStatsRecorder? We could just define in LazyVector.cpp? Thanks!
velox/exec/Driver.cpp
Outdated
| int64_t inputBytesDelta = 0; | ||
| it = lockStats->runtimeStats.find(LazyVector::kInputBytes); | ||
| if (it != lockStats->runtimeStats.end()) { | ||
| int64_t inputBytes = it->second.sum; |
There was a problem hiding this comment.
const int64_t inputBytes
|
This pull request was exported from Phabricator. Differential Revision: D63414708 |
59b440e to
41fa353
Compare
pedroerp
added a commit
to pedroerp/velox-1
that referenced
this pull request
Sep 26, 2024
Summary: Pull Request resolved: facebookincubator#11097 Whenever there is lazy loading, TableScan would come out with zero input and output bytes, and they would be attributed to the operator that in fact loaded the lazy vector. Using the existing mechanism to save it locally and periodically apply to the correct source operator. Reviewed By: xiaoxmeng Differential Revision: D63414708
xiaoxmeng
reviewed
Sep 27, 2024
velox/common/time/CpuWallTimer.h
Outdated
| } | ||
|
|
||
| private: | ||
| // NOTE: Put `wallTimeStart_` before `cpuTimeStart_`, so that wall-time starts | ||
| // counting earlier than cpu-time. | ||
| const std::chrono::steady_clock::time_point wallTimeStart_; | ||
| const uint64_t cpuTimeStart_; | ||
| }; | ||
|
|
||
| // Composes delta CpuWallTiming upon destruction and passes it to the user |
There was a problem hiding this comment.
/// for public comment. Thanks!
velox/common/time/CpuWallTimer.h
Outdated
| @@ -58,18 +58,13 @@ class CpuWallTimer { | |||
| }; | |||
|
|
|||
| // Keeps track of elapsed CPU and wall time from construction time. | |||
|
This pull request was exported from Phabricator. Differential Revision: D63414708 |
pedroerp
added a commit
to pedroerp/velox-1
that referenced
this pull request
Sep 27, 2024
Summary: Pull Request resolved: facebookincubator#11097 Whenever there is lazy loading, TableScan would come out with zero input and output bytes, and they would be attributed to the operator that in fact loaded the lazy vector. Using the existing mechanism to save it locally and periodically apply to the correct source operator. Reviewed By: xiaoxmeng Differential Revision: D63414708
41fa353 to
b9dd69f
Compare
|
This pull request was exported from Phabricator. Differential Revision: D63414708 |
pedroerp
added a commit
to pedroerp/velox-1
that referenced
this pull request
Sep 27, 2024
Summary: Pull Request resolved: facebookincubator#11097 Whenever there is lazy loading, TableScan would come out with zero input and output bytes, and they would be attributed to the operator that in fact loaded the lazy vector. Using the existing mechanism to save it locally and periodically apply to the correct source operator. Reviewed By: xiaoxmeng Differential Revision: D63414708
b9dd69f to
dfa0bc8
Compare
Summary: Pull Request resolved: facebookincubator#11097 Whenever there is lazy loading, TableScan would come out with zero input and output bytes, and they would be attributed to the operator that in fact loaded the lazy vector. Using the existing mechanism to save it locally and periodically apply to the correct source operator. Reviewed By: xiaoxmeng Differential Revision: D63414708
dfa0bc8 to
faaccef
Compare
|
This pull request was exported from Phabricator. Differential Revision: D63414708 |
|
This pull request has been merged in 42940c8. |
|
Conbench analyzed the 1 benchmark run on commit There were no benchmark performance regressions. 🎉 The full Conbench report has more details. |
athmaja-n
pushed a commit
to athmaja-n/velox
that referenced
this pull request
Jan 10, 2025
Summary: Pull Request resolved: facebookincubator#11097 Whenever there is lazy loading, TableScan would come out with zero input and output bytes, and they would be attributed to the operator that in fact loaded the lazy vector. Using the existing mechanism to save it locally and periodically apply to the correct source operator. Reviewed By: xiaoxmeng Differential Revision: D63414708 fbshipit-source-id: d1bbc64fc7ec1ed952593aaeece0b02daed60038
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Whenever there is lazy loading, TableScan would come out with zero
input and output bytes, and they would be attributed to the operator that in
fact loaded the lazy vector. Using the existing mechanism to save it locally
and periodically apply to the correct source operator.
Differential Revision: D63414708