Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop wide combiner state from growing unlimited #10997

Conversation

vladl2802
Copy link
Collaborator

@vladl2802 vladl2802 commented Oct 28, 2024

Changelog entry

Stop wide combiner state from growing unlimited and spill it if spilling is enabled

Changelog category

  • Improvement

Additional information

Until that point TState in WideCombiner could grow indefinitely. Without spilling it will just crash when allocator fails to allocate more memory for hashtable. With spilling it will stop its growth only when allocator reaches yellow zone (after that point spilling can begin). But with large enough hashtable TState can grow from green zone up to memory limit in one go (so spilling will not have a chance to begin).
So after this PR TState now catches exception when trying to grow hashtable and if it fails, state will stop its grow.
Using that we can add another condition for enabling spilling: if state can't grow it will be spilled.

Copy link

github-actions bot commented Oct 28, 2024

2024-10-28 12:19:05 UTC Pre-commit check linux-x86_64-relwithdebinfo for e17b551 has started.
2024-10-28 12:19:42 UTC Artifacts will be uploaded here
2024-10-28 12:23:16 UTC ya make is running...
2024-10-28 12:40:33 UTC Check cancelled

Copy link

github-actions bot commented Oct 28, 2024

2024-10-28 12:19:22 UTC Pre-commit check linux-x86_64-release-asan for e17b551 has started.
2024-10-28 12:19:33 UTC Artifacts will be uploaded here
2024-10-28 12:22:37 UTC ya make is running...
2024-10-28 12:40:32 UTC Check cancelled

Copy link

github-actions bot commented Oct 28, 2024

2024-10-28 12:42:23 UTC Pre-commit check linux-x86_64-relwithdebinfo for b8db0fc has started.
2024-10-28 12:42:34 UTC Artifacts will be uploaded here
2024-10-28 12:45:35 UTC ya make is running...
🟢 2024-10-28 14:12:07 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
75893 61840 0 0 13953 100

🟢 2024-10-28 14:19:16 UTC Build successful.
🟡 2024-10-28 14:19:34 UTC ydbd size 2.8 GiB changed* by +242.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: f752bc1 merge: b8db0fc diff diff %
ydbd size 3 033 679 416 Bytes 3 033 928 128 Bytes +242.9 KiB +0.008%
ydbd stripped size 480 648 664 Bytes 480 683 288 Bytes +33.8 KiB +0.007%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Oct 28, 2024

2024-10-28 12:42:45 UTC Pre-commit check linux-x86_64-release-asan for b8db0fc has started.
2024-10-28 12:42:55 UTC Artifacts will be uploaded here
2024-10-28 12:45:59 UTC ya make is running...
🟡 2024-10-28 14:29:44 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
12134 11939 0 46 49 100

🟢 2024-10-28 14:30:57 UTC Build successful.
🟢 2024-10-28 14:31:31 UTC ydbd size 5.7 GiB changed* by +1.9 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 3a074dd merge: b8db0fc diff diff %
ydbd size 6 139 738 960 Bytes 6 139 740 928 Bytes +1.9 KiB +0.000%
ydbd stripped size 1 532 103 536 Bytes 1 532 104 336 Bytes +800 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Oct 29, 2024

2024-10-29 10:45:47 UTC Pre-commit check linux-x86_64-relwithdebinfo for 8f11e0b has started.
2024-10-29 10:45:58 UTC Artifacts will be uploaded here
2024-10-29 10:49:10 UTC ya make is running...
🟡 2024-10-29 11:51:12 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
75905 61848 0 4 13954 99

2024-10-29 11:58:33 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-10-29 12:10:26 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
104 (only retried tests) 8 0 0 1 95

🟢 2024-10-29 12:10:32 UTC Build successful.
🟡 2024-10-29 12:10:54 UTC ydbd size 2.8 GiB changed* by +638.2 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 8acfed4 merge: 8f11e0b diff diff %
ydbd size 3 033 991 864 Bytes 3 034 645 392 Bytes +638.2 KiB +0.022%
ydbd stripped size 480 683 032 Bytes 480 723 608 Bytes +39.6 KiB +0.008%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Oct 29, 2024

2024-10-29 10:46:12 UTC Pre-commit check linux-x86_64-release-asan for 8f11e0b has started.
2024-10-29 10:46:23 UTC Artifacts will be uploaded here
2024-10-29 10:49:35 UTC ya make is running...
🟡 2024-10-29 12:09:14 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11734 11646 0 35 21 32

🟢 2024-10-29 12:10:15 UTC Build successful.
🟡 2024-10-29 12:10:50 UTC ydbd size 5.7 GiB changed* by +969.7 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 8acfed4 merge: 8f11e0b diff diff %
ydbd size 6 139 835 928 Bytes 6 140 828 872 Bytes +969.7 KiB +0.016%
ydbd stripped size 1 532 105 648 Bytes 1 532 228 112 Bytes +119.6 KiB +0.008%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Oct 29, 2024

2024-10-29 14:58:40 UTC Pre-commit check linux-x86_64-release-asan for 82e2ad7 has started.
2024-10-29 14:58:51 UTC Artifacts will be uploaded here
2024-10-29 15:02:00 UTC ya make is running...
2024-10-29 15:24:10 UTC Check cancelled

Copy link

github-actions bot commented Oct 29, 2024

2024-10-29 15:00:07 UTC Pre-commit check linux-x86_64-relwithdebinfo for 82e2ad7 has started.
2024-10-29 15:00:18 UTC Artifacts will be uploaded here
2024-10-29 15:03:21 UTC ya make is running...
2024-10-29 15:24:06 UTC Check cancelled

@vladl2802 vladl2802 marked this pull request as ready for review October 29, 2024 15:01
@vladl2802 vladl2802 requested a review from a team as a code owner October 29, 2024 15:01
Copy link

github-actions bot commented Oct 29, 2024

2024-10-29 15:26:07 UTC Pre-commit check linux-x86_64-release-asan for 51ff5e2 has started.
2024-10-29 15:26:17 UTC Artifacts will be uploaded here
2024-10-29 15:29:21 UTC ya make is running...
🟡 2024-10-29 16:42:53 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11729 11627 0 34 35 33

🟢 2024-10-29 16:43:53 UTC Build successful.
🟡 2024-10-29 16:44:23 UTC ydbd size 5.7 GiB changed* by +458.8 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: eee456c merge: 51ff5e2 diff diff %
ydbd size 6 147 101 648 Bytes 6 147 571 456 Bytes +458.8 KiB +0.008%
ydbd stripped size 1 533 991 632 Bytes 1 533 996 880 Bytes +5.1 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Oct 29, 2024

2024-10-29 15:26:09 UTC Pre-commit check linux-x86_64-relwithdebinfo for 51ff5e2 has started.
2024-10-29 15:26:19 UTC Artifacts will be uploaded here
2024-10-29 15:29:26 UTC ya make is running...
🟡 2024-10-29 16:31:07 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
75906 61844 0 11 13953 98

2024-10-29 16:38:15 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-10-29 16:49:41 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
109 (only retried tests) 18 0 0 0 91

🟢 2024-10-29 16:49:48 UTC Build successful.
🟡 2024-10-29 16:50:07 UTC ydbd size 2.8 GiB changed* by +444.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: eee456c merge: 51ff5e2 diff diff %
ydbd size 3 037 838 856 Bytes 3 038 294 440 Bytes +444.9 KiB +0.015%
ydbd stripped size 481 185 272 Bytes 481 185 912 Bytes +640 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

lll-phill-lll
lll-phill-lll previously approved these changes Oct 29, 2024
Copy link

github-actions bot commented Oct 30, 2024

2024-10-30 16:13:33 UTC Pre-commit check linux-x86_64-relwithdebinfo for 7e2546e has started.
2024-10-30 16:13:37 UTC Artifacts will be uploaded here
2024-10-30 16:16:37 UTC ya make is running...
🟡 2024-10-30 17:20:11 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
76029 61957 0 1 13973 98

2024-10-30 17:27:14 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-10-30 17:38:50 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
99 (only retried tests) 6 0 0 1 92

🟢 2024-10-30 17:38:57 UTC Build successful.
🟢 2024-10-30 17:39:17 UTC ydbd size 2.8 GiB changed* by +2.4 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 2dbf832 merge: 7e2546e diff diff %
ydbd size 3 038 379 008 Bytes 3 038 381 488 Bytes +2.4 KiB +0.000%
ydbd stripped size 481 282 744 Bytes 481 283 128 Bytes +384 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Oct 30, 2024

2024-10-30 16:14:16 UTC Pre-commit check linux-x86_64-release-asan for 7e2546e has started.
2024-10-30 16:14:27 UTC Artifacts will be uploaded here
2024-10-30 16:17:38 UTC ya make is running...
🟡 2024-10-30 17:30:28 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11727 11652 0 25 20 30

🟢 2024-10-30 17:31:26 UTC Build successful.
🟢 2024-10-30 17:31:58 UTC ydbd size 5.7 GiB changed* by +5.2 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 2dbf832 merge: 7e2546e diff diff %
ydbd size 6 148 051 872 Bytes 6 148 057 152 Bytes +5.2 KiB +0.000%
ydbd stripped size 1 534 231 600 Bytes 1 534 233 168 Bytes +1.5 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Oct 31, 2024

2024-10-31 08:19:25 UTC Pre-commit check linux-x86_64-relwithdebinfo for 600ef97 has started.
2024-10-31 08:19:37 UTC Artifacts will be uploaded here
2024-10-31 08:22:41 UTC ya make is running...
🟡 2024-10-31 09:24:41 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
76013 61941 0 3 13973 96

2024-10-31 09:31:50 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-10-31 09:43:18 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
96 (only retried tests) 5 0 1 0 90

2024-10-31 09:43:26 UTC ya make is running... (failed tests rerun, try 3)
🟢 2024-10-31 09:55:03 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
91 (only retried tests) 1 0 0 0 90

🟢 2024-10-31 09:55:09 UTC Build successful.
🟢 2024-10-31 09:55:29 UTC ydbd size 2.8 GiB changed* by +2.4 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 21fa904 merge: 600ef97 diff diff %
ydbd size 3 039 248 776 Bytes 3 039 251 208 Bytes +2.4 KiB +0.000%
ydbd stripped size 481 327 064 Bytes 481 327 448 Bytes +384 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Oct 31, 2024

2024-10-31 08:19:51 UTC Pre-commit check linux-x86_64-release-asan for 600ef97 has started.
2024-10-31 08:20:01 UTC Artifacts will be uploaded here
2024-10-31 08:23:04 UTC ya make is running...
🟡 2024-10-31 09:36:42 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11726 11641 0 37 19 29

🟢 2024-10-31 09:37:42 UTC Build successful.
🟢 2024-10-31 09:38:14 UTC ydbd size 5.7 GiB changed* by +5.0 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 21fa904 merge: 600ef97 diff diff %
ydbd size 6 149 695 528 Bytes 6 149 700 624 Bytes +5.0 KiB +0.000%
ydbd stripped size 1 534 413 616 Bytes 1 534 415 056 Bytes +1.4 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@vladl2802 vladl2802 requested a review from vitstn October 31, 2024 10:18
Copy link

github-actions bot commented Oct 31, 2024

2024-10-31 11:13:03 UTC Pre-commit check linux-x86_64-relwithdebinfo for 8dcaeea has started.
2024-10-31 11:13:14 UTC Artifacts will be uploaded here
2024-10-31 11:16:26 UTC ya make is running...
🟡 2024-10-31 12:18:37 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
76031 61959 0 5 13973 94

2024-10-31 12:25:43 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-10-31 12:37:18 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
98 (only retried tests) 8 0 0 0 90

🟢 2024-10-31 12:37:25 UTC Build successful.
🟢 2024-10-31 12:37:45 UTC ydbd size 2.8 GiB changed* by +2.4 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 55a88ca merge: 8dcaeea diff diff %
ydbd size 3 039 299 072 Bytes 3 039 301 504 Bytes +2.4 KiB +0.000%
ydbd stripped size 481 329 016 Bytes 481 329 400 Bytes +384 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Oct 31, 2024

2024-10-31 11:13:17 UTC Pre-commit check linux-x86_64-release-asan for 8dcaeea has started.
2024-10-31 11:13:28 UTC Artifacts will be uploaded here
2024-10-31 11:16:34 UTC ya make is running...
🟡 2024-10-31 12:34:38 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11707 11602 0 42 30 33

🟢 2024-10-31 12:35:42 UTC Build successful.
🟢 2024-10-31 12:36:15 UTC ydbd size 5.7 GiB changed* by +5.0 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 55a88ca merge: 8dcaeea diff diff %
ydbd size 6 149 801 808 Bytes 6 149 806 912 Bytes +5.0 KiB +0.000%
ydbd stripped size 1 534 429 616 Bytes 1 534 431 056 Bytes +1.4 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@lll-phill-lll lll-phill-lll linked an issue Oct 31, 2024 that may be closed by this pull request
@lll-phill-lll lll-phill-lll merged commit 5dad85d into ydb-platform:main Nov 1, 2024
10 checks passed
robot-piglet pushed a commit that referenced this pull request Nov 12, 2024
Apply GH: Stop wide combiner state from growing unlimited (#10997)

Apply GH: Reconnect session has been supported (#9862)

Apply GH: Handle unexpected future exception (#11091)
commit_hash:c2597abeae3a153692a80cc13446423769765ae1
maximyurchuk pushed a commit that referenced this pull request Nov 12, 2024
Apply GH: Stop wide combiner state from growing unlimited (#10997)

Apply GH: Reconnect session has been supported (#9862)

Apply GH: Handle unexpected future exception (#11091)
commit_hash:c2597abeae3a153692a80cc13446423769765ae1
lll-phill-lll pushed a commit to lll-phill-lll/ydb that referenced this pull request Nov 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Try to handle no memory exception during RH hash relocation
4 participants