Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize CPU usage when read blob #12153

Merged
merged 7 commits into from
Dec 2, 2024

Conversation

nshestakov
Copy link
Collaborator

@nshestakov nshestakov commented Nov 29, 2024

Changelog entry

Now we are giving all messages to the end of the blob in which they are stored to reduce the load when reading small messages.

Changelog category

  • Performance improvement

Additional information

...

@nshestakov nshestakov requested a review from a team as a code owner November 29, 2024 18:33
Copy link

github-actions bot commented Nov 29, 2024

2024-11-29 18:36:59 UTC Pre-commit check linux-x86_64-release-asan for 5041719 has started.
2024-11-29 18:37:11 UTC Artifacts will be uploaded here
2024-11-29 18:40:03 UTC ya make is running...
🟡 2024-11-29 19:41:07 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13243 13185 0 9 5 44

🟢 2024-11-29 19:42:10 UTC Build successful.
🟢 2024-11-29 19:42:36 UTC ydbd size 4.9 GiB changed* by -14.9 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 1b51ba9 merge: 5041719 diff diff %
ydbd size 5 281 843 344 Bytes 5 281 828 128 Bytes -14.9 KiB -0.000%
ydbd stripped size 1 361 475 568 Bytes 1 361 469 616 Bytes -5.8 KiB -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Nov 29, 2024

2024-11-29 18:37:20 UTC Pre-commit check linux-x86_64-relwithdebinfo for 5041719 has started.
2024-11-29 18:37:32 UTC Artifacts will be uploaded here
2024-11-29 18:40:36 UTC ya make is running...
🟡 2024-11-29 19:31:55 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
20928 19411 0 1 1407 109

2024-11-29 19:33:36 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-11-29 19:44:59 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
677 (only retried tests) 571 0 0 0 106

🟢 2024-11-29 19:45:08 UTC Build successful.
🟢 2024-11-29 19:45:27 UTC ydbd size 2.5 GiB changed* by +928 Bytes, which is < 100.0 KiB vs main: OK

ydbd size dash main: 1b51ba9 merge: 5041719 diff diff %
ydbd size 2 687 209 616 Bytes 2 687 210 544 Bytes +928 Bytes +0.000%
ydbd stripped size 482 315 024 Bytes 482 315 280 Bytes +256 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@nshestakov nshestakov changed the title WIP Optimize CPU usage when read blob Nov 29, 2024
Copy link

github-actions bot commented Nov 30, 2024

2024-11-30 09:02:54 UTC Pre-commit check linux-x86_64-relwithdebinfo for db7590c has started.
2024-11-30 09:03:05 UTC Artifacts will be uploaded here
2024-11-30 09:06:02 UTC ya make is running...
🟡 2024-11-30 09:56:58 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
20929 19418 0 5 1395 111

2024-11-30 09:58:39 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-11-30 10:10:16 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
722 (only retried tests) 612 0 3 0 107

2024-11-30 10:10:26 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-11-30 10:21:45 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
676 (only retried tests) 567 0 3 0 106

🟢 2024-11-30 10:21:54 UTC Build successful.
🟢 2024-11-30 10:22:15 UTC ydbd size 2.5 GiB changed* by +792 Bytes, which is < 100.0 KiB vs main: OK

ydbd size dash main: be00795 merge: db7590c diff diff %
ydbd size 2 687 210 080 Bytes 2 687 210 872 Bytes +792 Bytes +0.000%
ydbd stripped size 482 315 280 Bytes 482 315 408 Bytes +128 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Nov 30, 2024

2024-11-30 09:03:02 UTC Pre-commit check linux-x86_64-release-asan for db7590c has started.
2024-11-30 09:03:13 UTC Artifacts will be uploaded here
2024-11-30 09:06:08 UTC ya make is running...
🟡 2024-11-30 10:10:11 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13244 13170 0 18 9 47

🟢 2024-11-30 10:11:16 UTC Build successful.
🟢 2024-11-30 10:11:42 UTC ydbd size 4.9 GiB changed* by -14.8 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: be00795 merge: db7590c diff diff %
ydbd size 5 281 847 488 Bytes 5 281 832 288 Bytes -14.8 KiB -0.000%
ydbd stripped size 1 361 476 784 Bytes 1 361 470 704 Bytes -5.9 KiB -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Nov 30, 2024

2024-11-30 20:15:26 UTC Pre-commit check linux-x86_64-release-asan for 9f2f856 has started.
2024-11-30 20:16:48 UTC Artifacts will be uploaded here
2024-11-30 20:20:08 UTC ya make is running...
🟡 2024-11-30 21:22:57 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13244 13172 0 22 5 45

🟢 2024-11-30 21:24:02 UTC Build successful.
🟢 2024-11-30 21:24:29 UTC ydbd size 4.9 GiB changed* by -14.8 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 7466d62 merge: 9f2f856 diff diff %
ydbd size 5 289 692 488 Bytes 5 289 677 352 Bytes -14.8 KiB -0.000%
ydbd stripped size 1 364 389 456 Bytes 1 364 383 440 Bytes -5.9 KiB -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Nov 30, 2024

2024-11-30 20:16:51 UTC Pre-commit check linux-x86_64-relwithdebinfo for 9f2f856 has started.
2024-11-30 20:17:02 UTC Artifacts will be uploaded here
2024-11-30 20:20:06 UTC ya make is running...
🟡 2024-11-30 21:11:14 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
20929 19404 0 13 1396 116

2024-11-30 21:12:56 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-11-30 21:24:57 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
809 (only retried tests) 697 0 1 1 110

2024-11-30 21:25:08 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-11-30 21:37:04 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
727 (only retried tests) 617 0 1 0 109

🟢 2024-11-30 21:37:14 UTC Build successful.
🟢 2024-11-30 21:37:35 UTC ydbd size 2.5 GiB changed* by +800 Bytes, which is < 100.0 KiB vs main: OK

ydbd size dash main: 7466d62 merge: 9f2f856 diff diff %
ydbd size 2 691 029 656 Bytes 2 691 030 456 Bytes +800 Bytes +0.000%
ydbd stripped size 483 252 592 Bytes 483 252 720 Bytes +128 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Dec 2, 2024

2024-12-02 11:28:35 UTC Pre-commit check linux-x86_64-relwithdebinfo for d7509fd has started.
2024-12-02 11:28:46 UTC Artifacts will be uploaded here
2024-12-02 11:31:40 UTC ya make is running...
🟡 2024-12-02 12:18:03 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
20930 19419 0 1 1397 113

2024-12-02 12:19:52 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-12-02 12:20:55 UTC ydbd size 2.5 GiB changed* by +784 Bytes, which is < 100.0 KiB vs main: OK

ydbd size dash main: d3a36a5 merge: d7509fd diff diff %
ydbd size 2 692 171 576 Bytes 2 692 172 360 Bytes +784 Bytes +0.000%
ydbd stripped size 483 439 056 Bytes 483 439 184 Bytes +128 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation
2024-12-02 12:20:57 UTC Check cancelled

Copy link

github-actions bot commented Dec 2, 2024

2024-12-02 11:28:48 UTC Pre-commit check linux-x86_64-release-asan for d7509fd has started.
2024-12-02 11:28:59 UTC Artifacts will be uploaded here
2024-12-02 11:31:52 UTC ya make is running...
2024-12-02 12:20:46 UTC Check cancelled

Copy link

github-actions bot commented Dec 2, 2024

2024-12-02 12:22:35 UTC Pre-commit check linux-x86_64-release-asan for 48d2ad7 has started.
2024-12-02 12:22:45 UTC Artifacts will be uploaded here
2024-12-02 12:25:40 UTC ya make is running...
2024-12-02 12:54:55 UTC Check cancelled

Copy link

github-actions bot commented Dec 2, 2024

2024-12-02 12:23:05 UTC Pre-commit check linux-x86_64-relwithdebinfo for 48d2ad7 has started.
2024-12-02 12:23:16 UTC Artifacts will be uploaded here
2024-12-02 12:26:12 UTC ya make is running...
2024-12-02 12:53:47 UTC Check cancelled

Copy link

github-actions bot commented Dec 2, 2024

2024-12-02 12:59:14 UTC Pre-commit check linux-x86_64-release-asan for bf52bfc has started.
2024-12-02 12:59:53 UTC Artifacts will be uploaded here
2024-12-02 13:03:17 UTC ya make is running...
🟡 2024-12-02 14:22:23 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13247 13177 0 13 10 47

🟢 2024-12-02 14:23:28 UTC Build successful.
🟢 2024-12-02 14:23:55 UTC ydbd size 4.9 GiB changed* by -14.8 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 6fc18f8 merge: bf52bfc diff diff %
ydbd size 5 292 109 432 Bytes 5 292 094 296 Bytes -14.8 KiB -0.000%
ydbd stripped size 1 365 100 752 Bytes 1 365 094 736 Bytes -5.9 KiB -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Dec 2, 2024

2024-12-02 12:59:23 UTC Pre-commit check linux-x86_64-relwithdebinfo for bf52bfc has started.
2024-12-02 12:59:34 UTC Artifacts will be uploaded here
2024-12-02 13:02:28 UTC ya make is running...
🟡 2024-12-02 13:53:46 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
20960 19447 0 1 1396 116

2024-12-02 13:55:28 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-12-02 14:07:11 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
733 (only retried tests) 625 0 0 0 108

🟢 2024-12-02 14:07:20 UTC Build successful.
🟢 2024-12-02 14:07:40 UTC ydbd size 2.5 GiB changed* by +65.9 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 1cf8c98 merge: bf52bfc diff diff %
ydbd size 2 692 171 624 Bytes 2 692 239 088 Bytes +65.9 KiB +0.003%
ydbd stripped size 483 439 056 Bytes 483 446 416 Bytes +7.2 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@nshestakov nshestakov merged commit 7cae2d2 into ydb-platform:main Dec 2, 2024
11 checks passed
@nshestakov nshestakov deleted the Codecs-optimization branch December 2, 2024 14:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants