batch portions on write #13182

ivanmorozov333 · 2025-01-06T09:42:49Z

New feature
Improvement
Performance improvement

github-actions · 2025-01-06T09:46:36Z

⚪ 2025-01-06 09:46:36 UTC Pre-commit check linux-x86_64-release-asan for a38321e has started.
⚪ 2025-01-06 09:46:47 UTC Artifacts will be uploaded here
⚪ 2025-01-06 09:49:46 UTC ya make is running...
🟡 2025-01-06 10:51:10 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
13244	13187	0	23	7	27

⚪ 2025-01-06 10:52:18 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-06 11:04:25 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
97 (only retried tests)	63	0	8	1	25

⚪ 2025-01-06 11:04:33 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-01-06 11:06:36 UTC ydbd size 3.6 GiB changed* by +848.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `dcb70d0`	merge: `a38321e`	diff	diff %
ydbd size	3 872 739 328 Bytes	3 873 608 224 Bytes	+848.5 KiB	+0.022%
ydbd stripped size	1 350 441 520 Bytes	1 350 610 896 Bytes	+165.4 KiB	+0.013%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}
⚫ 2025-01-06 11:06:38 UTC Check cancelled

github-actions · 2025-01-06T09:46:43Z

⚪ 2025-01-06 09:46:42 UTC Pre-commit check linux-x86_64-relwithdebinfo for a38321e has started.
⚪ 2025-01-06 09:46:55 UTC Artifacts will be uploaded here
⚪ 2025-01-06 09:50:07 UTC ya make is running...
🟡 2025-01-06 10:51:02 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
27160	24648	0	4	2383	125

⚪ 2025-01-06 10:53:16 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-06 11:05:22 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
192 (only retried tests)	70	0	1	0	121

⚪ 2025-01-06 11:05:30 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-01-06 11:06:33 UTC ydbd size 2.1 GiB changed* by +536.8 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `dcb70d0`	merge: `a38321e`	diff	diff %
ydbd size	2 243 627 536 Bytes	2 244 177 200 Bytes	+536.8 KiB	+0.024%
ydbd stripped size	472 255 312 Bytes	472 314 384 Bytes	+57.7 KiB	+0.013%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}
⚫ 2025-01-06 11:06:34 UTC Check cancelled

github-actions · 2025-01-06T11:10:36Z

⚪ 2025-01-06 11:10:35 UTC Pre-commit check linux-x86_64-relwithdebinfo for ebd6dce has started.
⚪ 2025-01-06 11:10:47 UTC Artifacts will be uploaded here
⚪ 2025-01-06 11:13:46 UTC ya make is running...
🟡 2025-01-06 12:05:00 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
27160	24644	0	6	2383	127

⚪ 2025-01-06 12:07:11 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-01-06 12:19:15 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
196 (only retried tests)	73	0	0	0	123

🟢 2025-01-06 12:19:22 UTC Build successful.
🟡 2025-01-06 12:19:41 UTC ydbd size 2.1 GiB changed* by +536.8 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `dcb70d0`	merge: `ebd6dce`	diff	diff %
ydbd size	2 243 627 536 Bytes	2 244 177 264 Bytes	+536.8 KiB	+0.025%
ydbd stripped size	472 255 312 Bytes	472 314 384 Bytes	+57.7 KiB	+0.013%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-01-06T11:10:36Z

⚪ 2025-01-06 11:10:36 UTC Pre-commit check linux-x86_64-release-asan for ebd6dce has started.
⚪ 2025-01-06 11:10:49 UTC Artifacts will be uploaded here
⚪ 2025-01-06 11:13:51 UTC ya make is running...
🟡 2025-01-06 12:12:58 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
13244	13183	0	21	11	29

⚪ 2025-01-06 12:14:04 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-06 12:26:46 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
99 (only retried tests)	66	0	5	2	26

⚪ 2025-01-06 12:26:54 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-01-06 12:38:31 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
57 (only retried tests)	26	0	4	2	25

🟢 2025-01-06 12:38:38 UTC Build successful.
🟡 2025-01-06 12:39:05 UTC ydbd size 3.6 GiB changed* by +848.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `dcb70d0`	merge: `ebd6dce`	diff	diff %
ydbd size	3 872 739 328 Bytes	3 873 608 240 Bytes	+848.5 KiB	+0.022%
ydbd stripped size	1 350 441 520 Bytes	1 350 610 832 Bytes	+165.3 KiB	+0.013%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-01-06T12:46:44Z

⚪ 2025-01-06 12:46:44 UTC Pre-commit check linux-x86_64-relwithdebinfo for b5d0d1c has started.
⚪ 2025-01-06 12:47:09 UTC Artifacts will be uploaded here
⚪ 2025-01-06 12:50:39 UTC ya make is running...
⚫ 2025-01-06 13:01:42 UTC Check cancelled

github-actions · 2025-01-06T12:47:59Z

⚪ 2025-01-06 12:47:58 UTC Pre-commit check linux-x86_64-release-asan for b5d0d1c has started.
⚪ 2025-01-06 12:48:21 UTC Artifacts will be uploaded here
⚪ 2025-01-06 12:51:18 UTC ya make is running...
⚫ 2025-01-06 13:01:44 UTC Check cancelled

github-actions · 2025-01-06T13:03:19Z

⚪ 2025-01-06 13:03:18 UTC Pre-commit check linux-x86_64-relwithdebinfo for f79277a has started.
⚪ 2025-01-06 13:05:06 UTC Artifacts will be uploaded here
⚪ 2025-01-06 13:08:38 UTC ya make is running...
⚫ 2025-01-06 13:13:42 UTC Check cancelled

github-actions · 2025-01-06T13:05:34Z

⚪ 2025-01-06 13:05:33 UTC Pre-commit check linux-x86_64-release-asan for f79277a has started.
⚪ 2025-01-06 13:05:45 UTC Artifacts will be uploaded here
⚪ 2025-01-06 13:08:48 UTC ya make is running...
⚫ 2025-01-06 13:13:50 UTC Check cancelled

github-actions · 2025-01-06T13:16:42Z

⚪ 2025-01-06 13:16:41 UTC Pre-commit check linux-x86_64-release-asan for a261529 has started.
⚪ 2025-01-06 13:17:11 UTC Artifacts will be uploaded here
⚪ 2025-01-06 13:20:35 UTC ya make is running...
🟡 2025-01-06 14:57:26 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
13455	13393	0	22	10	30

⚪ 2025-01-06 14:58:34 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-06 15:09:54 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
102 (only retried tests)	72	0	5	0	25

⚪ 2025-01-06 15:10:02 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-01-06 15:14:16 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
58 (only retried tests)	32	0	1	0	25

🟢 2025-01-06 15:14:24 UTC Build successful.
🟡 2025-01-06 15:14:52 UTC ydbd size 3.6 GiB changed* by +871.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `dcb70d0`	merge: `a261529`	diff	diff %
ydbd size	3 872 739 328 Bytes	3 873 631 600 Bytes	+871.4 KiB	+0.023%
ydbd stripped size	1 350 441 520 Bytes	1 350 620 240 Bytes	+174.5 KiB	+0.013%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-01-06T13:17:45Z

⚪ 2025-01-06 13:17:45 UTC Pre-commit check linux-x86_64-relwithdebinfo for a261529 has started.
⚪ 2025-01-06 13:17:56 UTC Artifacts will be uploaded here
⚪ 2025-01-06 13:21:22 UTC ya make is running...
🟡 2025-01-06 14:44:48 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
27454	24942	0	4	2384	124

⚪ 2025-01-06 14:47:00 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-01-06 14:58:50 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
189 (only retried tests)	69	0	0	0	120

🟢 2025-01-06 14:58:57 UTC Build successful.
🟡 2025-01-06 14:59:18 UTC ydbd size 2.1 GiB changed* by +551.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `dcb70d0`	merge: `a261529`	diff	diff %
ydbd size	2 243 627 536 Bytes	2 244 192 152 Bytes	+551.4 KiB	+0.025%
ydbd stripped size	472 255 312 Bytes	472 320 528 Bytes	+63.7 KiB	+0.014%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

ydb/core/formats/arrow/reader/position.h

zverevgeny · 2025-01-06T14:54:37Z

ydb/core/kqp/ut/olap/helpers/typed_local.h

@@ -37,6 +37,44 @@ class TTypedLocalHelper: public Tests::NCS::THelper {
        SetShardingMethod("HASH_FUNCTION_CONSISTENCY_64");
    }

+    class TWritingGuard {


Непонятно, почему это гард. Я бы назвал MultiWriteHelper

Conflicts: ydb/core/tx/columnshard/blobs_action/transaction/tx_blobs_written.cpp

ivanmorozov333 added 3 commits January 5, 2025 16:02

batch portions on write

1152529

fixes

7dd58f7

corrections

ac45565

github-actions bot added the new-feature label Jan 6, 2025

fix

3bbac70

fixes and add test

1c331cb

correction

7c18538

correction

4a66d66

zverevgeny reviewed Jan 6, 2025

View reviewed changes

ydb/core/formats/arrow/reader/position.h Show resolved Hide resolved

zverevgeny approved these changes Jan 6, 2025

View reviewed changes

ivanmorozov333 marked this pull request as ready for review January 6, 2025 15:46

ivanmorozov333 requested a review from a team as a code owner January 6, 2025 15:46

ivanmorozov333 merged commit 63b0998 into ydb-platform:main Jan 6, 2025
12 checks passed

ivanmorozov333 deleted the batch_portions_on_write branch January 6, 2025 15:46

zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Jan 21, 2025

batch portions on write (ydb-platform#13182)

3b32afc

Conflicts: ydb/core/tx/columnshard/blobs_action/transaction/tx_blobs_written.cpp

zverevgeny mentioned this pull request Jan 21, 2025

Integrate changes from main #13202

Closed

azevaykin pushed a commit to azevaykin/ydb that referenced this pull request Feb 3, 2025

batch portions on write (ydb-platform#13182)

5cd3574

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

batch portions on write #13182

batch portions on write #13182

ivanmorozov333 commented Jan 6, 2025

github-actions bot commented Jan 6, 2025 •

edited

Loading

github-actions bot commented Jan 6, 2025 •

edited

Loading

github-actions bot commented Jan 6, 2025 •

edited

Loading

github-actions bot commented Jan 6, 2025 •

edited

Loading

github-actions bot commented Jan 6, 2025 •

edited

Loading

github-actions bot commented Jan 6, 2025 •

edited

Loading

github-actions bot commented Jan 6, 2025 •

edited

Loading

github-actions bot commented Jan 6, 2025 •

edited

Loading

github-actions bot commented Jan 6, 2025 •

edited

Loading

github-actions bot commented Jan 6, 2025 •

edited

Loading

zverevgeny Jan 6, 2025

batch portions on write #13182

batch portions on write #13182

Conversation

ivanmorozov333 commented Jan 6, 2025

github-actions bot commented Jan 6, 2025 • edited Loading

github-actions bot commented Jan 6, 2025 • edited Loading

github-actions bot commented Jan 6, 2025 • edited Loading

github-actions bot commented Jan 6, 2025 • edited Loading

github-actions bot commented Jan 6, 2025 • edited Loading

github-actions bot commented Jan 6, 2025 • edited Loading

github-actions bot commented Jan 6, 2025 • edited Loading

github-actions bot commented Jan 6, 2025 • edited Loading

github-actions bot commented Jan 6, 2025 • edited Loading

github-actions bot commented Jan 6, 2025 • edited Loading

zverevgeny Jan 6, 2025

Choose a reason for hiding this comment

github-actions bot commented Jan 6, 2025 •

edited

Loading

github-actions bot commented Jan 6, 2025 •

edited

Loading

github-actions bot commented Jan 6, 2025 •

edited

Loading

github-actions bot commented Jan 6, 2025 •

edited

Loading

github-actions bot commented Jan 6, 2025 •

edited

Loading

github-actions bot commented Jan 6, 2025 •

edited

Loading

github-actions bot commented Jan 6, 2025 •

edited

Loading

github-actions bot commented Jan 6, 2025 •

edited

Loading

github-actions bot commented Jan 6, 2025 •

edited

Loading

github-actions bot commented Jan 6, 2025 •

edited

Loading