Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Default compression setting via CS config #13203

Merged

Conversation

vlad-gogov
Copy link
Collaborator

Changelog entry

Default compression setting via CS config and its validation

Changelog category

  • New feature

@vlad-gogov vlad-gogov self-assigned this Jan 9, 2025
@vlad-gogov vlad-gogov requested a review from a team as a code owner January 9, 2025 04:55
Copy link

github-actions bot commented Jan 9, 2025

2025-01-09 04:59:29 UTC Pre-commit check linux-x86_64-relwithdebinfo for a6bfc84 has started.
2025-01-09 04:59:41 UTC Artifacts will be uploaded here
2025-01-09 05:03:01 UTC ya make is running...
🟡 2025-01-09 06:29:24 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
27468 24948 0 8 2383 129

2025-01-09 06:31:42 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-01-09 06:43:12 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
198 (only retried tests) 73 0 0 0 125

🟢 2025-01-09 06:43:20 UTC Build successful.
🟢 2025-01-09 06:43:40 UTC ydbd size 2.1 GiB changed* by +19.4 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 9f31130 merge: a6bfc84 diff diff %
ydbd size 2 244 317 464 Bytes 2 244 337 352 Bytes +19.4 KiB +0.001%
ydbd stripped size 472 336 560 Bytes 472 338 224 Bytes +1.6 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jan 9, 2025

2025-01-09 04:59:41 UTC Pre-commit check linux-x86_64-release-asan for a6bfc84 has started.
2025-01-09 04:59:53 UTC Artifacts will be uploaded here
2025-01-09 05:03:04 UTC ya make is running...
🟡 2025-01-09 06:29:53 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13467 13399 0 30 7 31

2025-01-09 06:31:00 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-09 06:43:14 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
112 (only retried tests) 81 0 4 0 27

2025-01-09 06:43:22 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-01-09 06:49:13 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
58 (only retried tests) 31 0 1 0 26

🟢 2025-01-09 06:49:19 UTC Build successful.
🟢 2025-01-09 06:49:49 UTC ydbd size 3.6 GiB changed* by +30.3 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 9f31130 merge: a6bfc84 diff diff %
ydbd size 3 873 834 368 Bytes 3 873 865 376 Bytes +30.3 KiB +0.001%
ydbd stripped size 1 350 684 240 Bytes 1 350 689 136 Bytes +4.8 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jan 9, 2025

2025-01-09 10:22:30 UTC Pre-commit check linux-x86_64-release-asan for c3b875d has started.
2025-01-09 10:22:37 UTC Artifacts will be uploaded here
2025-01-09 10:25:53 UTC ya make is running...
🟡 2025-01-09 11:53:04 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13468 13413 0 22 6 27

2025-01-09 11:54:45 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-09 12:06:33 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
93 (only retried tests) 64 0 4 2 23

2025-01-09 12:06:41 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-01-09 12:18:18 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
52 (only retried tests) 26 0 2 0 24

🟢 2025-01-09 12:18:27 UTC Build successful.
🟢 2025-01-09 12:18:55 UTC ydbd size 3.6 GiB changed* by +34.5 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 0859ec3 merge: c3b875d diff diff %
ydbd size 3 874 438 760 Bytes 3 874 474 072 Bytes +34.5 KiB +0.001%
ydbd stripped size 1 350 831 216 Bytes 1 350 840 336 Bytes +8.9 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jan 9, 2025

2025-01-09 10:24:34 UTC Pre-commit check linux-x86_64-relwithdebinfo for c3b875d has started.
2025-01-09 10:24:46 UTC Artifacts will be uploaded here
2025-01-09 10:27:58 UTC ya make is running...
🟡 2025-01-09 11:47:48 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
27470 24947 0 5 2383 135

2025-01-09 11:50:03 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-01-09 12:01:54 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
198 (only retried tests) 78 0 0 0 120

🟢 2025-01-09 12:02:01 UTC Build successful.
🟢 2025-01-09 12:02:19 UTC ydbd size 2.1 GiB changed* by +19.6 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 592bc99 merge: c3b875d diff diff %
ydbd size 2 244 699 432 Bytes 2 244 719 536 Bytes +19.6 KiB +0.001%
ydbd stripped size 472 406 672 Bytes 472 408 400 Bytes +1.7 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@vlad-gogov vlad-gogov force-pushed the validate_compression_from_cs_config branch from 459a77f to 2bb005a Compare January 10, 2025 12:35
Copy link

github-actions bot commented Jan 10, 2025

2025-01-10 12:37:19 UTC Pre-commit check linux-x86_64-relwithdebinfo for e93ff3d has started.
2025-01-10 12:37:30 UTC Artifacts will be uploaded here
2025-01-10 12:40:50 UTC ya make is running...
🟡 2025-01-10 14:06:18 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
27480 24974 0 1 2380 125

🟢 2025-01-10 14:08:41 UTC ydbd size 2.1 GiB changed* by +27.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: f5f4a12 merge: e93ff3d diff diff %
ydbd size 2 244 922 968 Bytes 2 244 951 312 Bytes +27.7 KiB +0.001%
ydbd stripped size 472 441 872 Bytes 472 451 760 Bytes +9.7 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jan 10, 2025

2025-01-10 12:40:02 UTC Pre-commit check linux-x86_64-release-asan for e93ff3d has started.
2025-01-10 12:40:14 UTC Artifacts will be uploaded here
2025-01-10 12:43:23 UTC ya make is running...
🟡 2025-01-10 14:12:26 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13477 13417 0 23 3 34

🟢 2025-01-10 14:13:08 UTC ydbd size 3.6 GiB changed* by +34.6 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: f5f4a12 merge: e93ff3d diff diff %
ydbd size 3 874 879 552 Bytes 3 874 914 984 Bytes +34.6 KiB +0.001%
ydbd stripped size 1 350 984 496 Bytes 1 350 993 584 Bytes +8.9 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link
Collaborator

@swalrus1 swalrus1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

В основном, переиспользовать уже реализованную логику валидации

}
std::optional<arrow::Compression::type> codec = NArrow::CompressionFromProto(columnShardConfig.GetDefaultCompression());
if (!codec.has_value()) {
msg.push_back("ColumnShardConfig: Unknown compression");
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Зачем собирать сообщения об ошибках в вектор? Не будет лучше подходить TConclusionStatus: возвращается либо провал валидации и сообщение, либо успех?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Общая функция валидации конфигурации принимает вектор, в который записываются сообщения об ошибках. Писал по аналогии с https://github.com/ydb-platform/ydb/blob/main/ydb/core/config/validation/auth_config_validator.cpp

Copy link
Collaborator

@swalrus1 swalrus1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

В основном, переиспользовать логику валидации

Copy link

github-actions bot commented Jan 13, 2025

2025-01-13 04:48:32 UTC Pre-commit check linux-x86_64-relwithdebinfo for 6db9496 has started.
2025-01-13 04:48:36 UTC Artifacts will be uploaded here
2025-01-13 04:51:48 UTC ya make is running...
🟡 2025-01-13 06:14:20 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
27484 24973 0 1 2382 128

2025-01-13 06:16:38 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-01-13 06:29:01 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
190 (only retried tests) 69 0 0 0 121

🟢 2025-01-13 06:29:10 UTC Build successful.
🟢 2025-01-13 06:29:29 UTC ydbd size 2.1 GiB changed* by +19.6 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 0ae852e merge: 6db9496 diff diff %
ydbd size 2 245 751 616 Bytes 2 245 771 720 Bytes +19.6 KiB +0.001%
ydbd stripped size 472 620 080 Bytes 472 621 744 Bytes +1.6 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jan 13, 2025

2025-01-13 04:50:22 UTC Pre-commit check linux-x86_64-release-asan for 6db9496 has started.
2025-01-13 04:50:34 UTC Artifacts will be uploaded here
2025-01-13 04:53:54 UTC ya make is running...
🟡 2025-01-13 06:24:39 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13477 13424 0 23 4 26

2025-01-13 06:25:47 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-01-13 06:37:48 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
87 (only retried tests) 60 0 0 1 26

🟢 2025-01-13 06:37:58 UTC Build successful.
🟡 2025-01-13 06:38:26 UTC ydbd size 3.6 GiB changed* by +120.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 039db66 merge: 6db9496 diff diff %
ydbd size 3 893 372 472 Bytes 3 893 495 912 Bytes +120.5 KiB +0.003%
ydbd stripped size 1 360 148 528 Bytes 1 360 179 120 Bytes +29.9 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@vlad-gogov vlad-gogov requested a review from swalrus1 January 13, 2025 06:54
@vlad-gogov vlad-gogov linked an issue Jan 13, 2025 that may be closed by this pull request
Copy link

github-actions bot commented Jan 13, 2025

2025-01-13 08:19:04 UTC Pre-commit check linux-x86_64-relwithdebinfo for df0a032 has started.
2025-01-13 08:19:28 UTC Artifacts will be uploaded here
2025-01-13 08:23:07 UTC ya make is running...
🟡 2025-01-13 09:16:43 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
27484 24964 0 6 2382 132

2025-01-13 09:18:54 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-13 09:30:27 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
207 (only retried tests) 83 0 2 0 122

2025-01-13 09:30:35 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-01-13 09:42:37 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
180 (only retried tests) 58 0 1 0 121

🟢 2025-01-13 09:42:44 UTC Build successful.
🟢 2025-01-13 09:43:08 UTC ydbd size 2.1 GiB changed* by +19.8 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: c2439a5 merge: df0a032 diff diff %
ydbd size 2 245 751 888 Bytes 2 245 772 200 Bytes +19.8 KiB +0.001%
ydbd stripped size 472 620 080 Bytes 472 621 744 Bytes +1.6 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jan 13, 2025

2025-01-13 08:19:47 UTC Pre-commit check linux-x86_64-release-asan for df0a032 has started.
2025-01-13 08:19:58 UTC Artifacts will be uploaded here
2025-01-13 08:23:07 UTC ya make is running...
🟡 2025-01-13 09:21:44 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13477 13421 0 27 2 27

2025-01-13 09:22:59 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-13 09:34:43 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
92 (only retried tests) 66 0 1 1 24

2025-01-13 09:34:51 UTC ya make is running... (failed tests rerun, try 3)
🟢 2025-01-13 09:46:28 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
50 (only retried tests) 25 0 0 0 25

🟢 2025-01-13 09:46:35 UTC Build successful.
🟢 2025-01-13 09:47:02 UTC ydbd size 3.6 GiB changed* by +30.5 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: c2439a5 merge: df0a032 diff diff %
ydbd size 3 893 469 264 Bytes 3 893 500 504 Bytes +30.5 KiB +0.001%
ydbd stripped size 1 360 178 416 Bytes 1 360 183 216 Bytes +4.7 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jan 13, 2025

2025-01-13 09:53:05 UTC Pre-commit check linux-x86_64-relwithdebinfo for 375c455 has started.
2025-01-13 09:53:17 UTC Artifacts will be uploaded here
2025-01-13 09:56:34 UTC ya make is running...
🟡 2025-01-13 11:02:34 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
27484 24976 0 2 2380 126

2025-01-13 11:05:00 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-01-13 11:17:04 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
185 (only retried tests) 63 0 0 0 122

🟢 2025-01-13 11:17:11 UTC Build successful.
🟢 2025-01-13 11:17:31 UTC ydbd size 2.1 GiB changed* by +19.8 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: d61f688 merge: 375c455 diff diff %
ydbd size 2 245 779 304 Bytes 2 245 799 600 Bytes +19.8 KiB +0.001%
ydbd stripped size 472 620 976 Bytes 472 622 640 Bytes +1.6 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jan 13, 2025

2025-01-13 09:53:31 UTC Pre-commit check linux-x86_64-release-asan for 375c455 has started.
2025-01-13 09:53:49 UTC Artifacts will be uploaded here
2025-01-13 09:57:10 UTC ya make is running...
🟡 2025-01-13 11:07:56 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13477 13426 0 19 3 29

2025-01-13 11:09:06 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-13 11:21:03 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
84 (only retried tests) 55 0 2 3 24

2025-01-13 11:21:12 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-01-13 11:32:23 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
53 (only retried tests) 28 0 1 0 24

🟢 2025-01-13 11:32:34 UTC Build successful.
🟢 2025-01-13 11:33:04 UTC ydbd size 3.6 GiB changed* by +30.5 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: d61f688 merge: 375c455 diff diff %
ydbd size 3 893 504 640 Bytes 3 893 535 880 Bytes +30.5 KiB +0.001%
ydbd stripped size 1 360 179 376 Bytes 1 360 184 176 Bytes +4.7 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@vlad-gogov vlad-gogov merged commit fef7e41 into ydb-platform:main Jan 13, 2025
17 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Default compression setting via CS config
3 participants