support folders in S3 uri in CS tiers #13337

swalrus1 · 2025-01-13T14:53:15Z

Changelog entry

Support folders in S3 URIs for CS tiers

Changelog category

Improvement

Additional information

...

github-actions · 2025-01-13T14:54:42Z

⚪ 2025-01-13 14:54:42 UTC Pre-commit check linux-x86_64-relwithdebinfo for 336f81b has started.
⚪ 2025-01-13 14:54:46 UTC Artifacts will be uploaded here
⚪ 2025-01-13 14:57:45 UTC ya make is running...
🟡 2025-01-13 15:45:50 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
18731	17421	0	9	1174	127

⚪ 2025-01-13 15:47:51 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-13 16:00:28 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
197 (only retried tests)	76	0	1	0	120

⚪ 2025-01-13 16:00:40 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-01-13 16:05:11 UTC ydbd size 2.1 GiB changed* by +160.8 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `c0dd9da`	merge: `336f81b`	diff	diff %
ydbd size	2 245 790 952 Bytes	2 245 955 576 Bytes	+160.8 KiB	+0.007%
ydbd stripped size	472 639 696 Bytes	472 643 856 Bytes	+4.1 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}
⚫ 2025-01-13 16:05:13 UTC Check cancelled

github-actions · 2025-01-13T14:57:03Z

⚪ 2025-01-13 14:57:03 UTC Pre-commit check linux-x86_64-release-asan for 336f81b has started.
⚪ 2025-01-13 14:57:14 UTC Artifacts will be uploaded here
⚪ 2025-01-13 15:00:06 UTC ya make is running...
🟡 2025-01-13 15:58:54 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11603	11555	0	18	2	28

⚪ 2025-01-13 15:59:58 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-13 16:05:12 UTC ydbd size 3.6 GiB changed* by +259.8 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `c0dd9da`	merge: `336f81b`	diff	diff %
ydbd size	3 893 424 088 Bytes	3 893 690 080 Bytes	+259.8 KiB	+0.007%
ydbd stripped size	1 360 223 984 Bytes	1 360 242 128 Bytes	+17.7 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}
⚫ 2025-01-13 16:05:14 UTC Check cancelled

github-actions · 2025-01-13T16:08:36Z

⚪ 2025-01-13 16:08:36 UTC Pre-commit check linux-x86_64-relwithdebinfo for 602c46b has started.
⚪ 2025-01-13 16:09:02 UTC Artifacts will be uploaded here
⚪ 2025-01-13 16:12:33 UTC ya make is running...
🟢 2025-01-13 16:58:59 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
18733	17443	0	0	1169	121

🟢 2025-01-13 17:01:02 UTC Build successful.
🟡 2025-01-13 17:01:18 UTC ydbd size 2.1 GiB changed* by +160.8 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `1402e4e`	merge: `602c46b`	diff	diff %
ydbd size	2 245 809 440 Bytes	2 245 974 064 Bytes	+160.8 KiB	+0.007%
ydbd stripped size	472 640 144 Bytes	472 644 304 Bytes	+4.1 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-01-13T16:09:32Z

⚪ 2025-01-13 16:09:31 UTC Pre-commit check linux-x86_64-release-asan for 602c46b has started.
⚪ 2025-01-13 16:09:43 UTC Artifacts will be uploaded here
⚪ 2025-01-13 16:12:34 UTC ya make is running...
🟡 2025-01-13 17:13:30 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11605	11534	0	42	2	27

⚪ 2025-01-13 17:14:28 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-13 17:26:37 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
112 (only retried tests)	85	0	1	0	26

⚪ 2025-01-13 17:26:45 UTC ya make is running... (failed tests rerun, try 3)
🟢 2025-01-13 17:38:13 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
49 (only retried tests)	24	0	0	1	24

🟢 2025-01-13 17:38:20 UTC Build successful.
🟡 2025-01-13 17:38:49 UTC ydbd size 3.6 GiB changed* by +259.8 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `1402e4e`	merge: `602c46b`	diff	diff %
ydbd size	3 893 453 432 Bytes	3 893 719 504 Bytes	+259.8 KiB	+0.007%
ydbd stripped size	1 360 226 992 Bytes	1 360 245 200 Bytes	+17.8 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

zverevgeny · 2025-01-14T07:21:21Z

ydb/core/tx/tiering/tier/s3_uri.h

+
+class TS3Uri {
+private:
+    YDB_READONLY_DEF(TString, Host);


На мой взгляд, лучше перечислять в порядке их вхождения в url

github-actions · 2025-01-14T08:53:36Z

⚪ 2025-01-14 08:53:35 UTC Pre-commit check linux-x86_64-relwithdebinfo for a621cf3 has started.
⚪ 2025-01-14 08:54:18 UTC Artifacts will be uploaded here
⚪ 2025-01-14 08:57:37 UTC ya make is running...
🟡 2025-01-14 09:39:45 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
18733	17438	0	3	1169	123

⚪ 2025-01-14 09:41:20 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-01-14 09:52:52 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
183 (only retried tests)	66	0	0	0	117

🟢 2025-01-14 09:53:03 UTC Build successful.
🟡 2025-01-14 09:53:25 UTC ydbd size 2.1 GiB changed* by +160.8 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `90166bc`	merge: `a621cf3`	diff	diff %
ydbd size	2 245 824 976 Bytes	2 245 989 648 Bytes	+160.8 KiB	+0.007%
ydbd stripped size	472 643 472 Bytes	472 647 632 Bytes	+4.1 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-01-14T08:54:26Z

⚪ 2025-01-14 08:54:26 UTC Pre-commit check linux-x86_64-release-asan for a621cf3 has started.
⚪ 2025-01-14 08:54:57 UTC Artifacts will be uploaded here
⚪ 2025-01-14 08:58:35 UTC ya make is running...
🟡 2025-01-14 09:52:54 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11605	11547	0	21	9	28

⚪ 2025-01-14 09:53:54 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-14 10:05:57 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
93 (only retried tests)	62	0	1	4	26

⚪ 2025-01-14 10:06:09 UTC ya make is running... (failed tests rerun, try 3)
🟢 2025-01-14 10:17:51 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
53 (only retried tests)	28	0	0	0	25

🟢 2025-01-14 10:18:03 UTC Build successful.
🟡 2025-01-14 10:18:41 UTC ydbd size 3.6 GiB changed* by +264.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `90166bc`	merge: `a621cf3`	diff	diff %
ydbd size	3 893 526 408 Bytes	3 893 797 288 Bytes	+264.5 KiB	+0.007%
ydbd stripped size	1 360 250 608 Bytes	1 360 273 104 Bytes	+22.0 KiB	+0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

zverevgeny · 2025-01-14T09:21:28Z

ydb/core/tx/tiering/tier/s3_uri.h

+
+    static std::optional<TUriStyle> DeduceUriStyle(const NUri::TUri& uri) {
+        {
+            static const std::regex StrictEndpointPattern = std::regex{ "^(.+\\.)?s3[.-].*" };


Перепиши, плиз, без регулярки. Тут не сложный формат

github-actions · 2025-01-14T12:43:51Z

⚪ 2025-01-14 12:43:51 UTC Pre-commit check linux-x86_64-relwithdebinfo for 0e0ab0b has started.
⚪ 2025-01-14 12:44:02 UTC Artifacts will be uploaded here
⚪ 2025-01-14 12:46:55 UTC ya make is running...
⚫ 2025-01-14 12:58:47 UTC Check cancelled

github-actions · 2025-01-14T12:44:57Z

⚪ 2025-01-14 12:44:57 UTC Pre-commit check linux-x86_64-release-asan for 0e0ab0b has started.
⚪ 2025-01-14 12:45:36 UTC Artifacts will be uploaded here
⚪ 2025-01-14 12:49:08 UTC ya make is running...
⚫ 2025-01-14 12:58:27 UTC Check cancelled

github-actions · 2025-01-14T13:02:03Z

⚪ 2025-01-14 13:02:02 UTC Pre-commit check linux-x86_64-release-asan for 975e174 has started.
⚪ 2025-01-14 13:02:14 UTC Artifacts will be uploaded here
⚪ 2025-01-14 13:05:06 UTC ya make is running...
🟡 2025-01-14 14:00:07 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11607	11559	0	17	7	24

⚪ 2025-01-14 14:01:39 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-14 14:13:36 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
76 (only retried tests)	46	0	2	5	23

⚪ 2025-01-14 14:13:51 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-01-14 14:17:12 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
53 (only retried tests)	30	0	1	0	22

🟢 2025-01-14 14:18:01 UTC Build successful.
🟢 2025-01-14 14:23:28 UTC ydbd size 3.6 GiB changed* by +67.6 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `f9605fd`	merge: `975e174`	diff	diff %
ydbd size	3 895 410 248 Bytes	3 895 479 448 Bytes	+67.6 KiB	+0.002%
ydbd stripped size	1 360 963 408 Bytes	1 360 988 784 Bytes	+24.8 KiB	+0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-01-14T13:02:48Z

⚪ 2025-01-14 13:02:47 UTC Pre-commit check linux-x86_64-relwithdebinfo for 975e174 has started.
⚪ 2025-01-14 13:02:59 UTC Artifacts will be uploaded here
⚪ 2025-01-14 13:05:52 UTC ya make is running...
🟡 2025-01-14 13:51:34 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
18735	17444	0	1	1169	121

⚪ 2025-01-14 13:53:07 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-01-14 14:04:52 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
175 (only retried tests)	58	0	0	0	117

🟢 2025-01-14 14:05:18 UTC Build successful.
🟢 2025-01-14 14:05:37 UTC ydbd size 2.1 GiB changed* by +40.2 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `7be46e5`	merge: `975e174`	diff	diff %
ydbd size	2 247 484 640 Bytes	2 247 525 808 Bytes	+40.2 KiB	+0.002%
ydbd stripped size	472 871 664 Bytes	472 876 272 Bytes	+4.5 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

zverevgeny · 2025-01-15T11:07:30Z

ydb/core/tx/tiering/tier/s3_uri.h

+        }
+
+        TUriStyle uriStyle;
+        if (const auto deducedStyle = DeduceUriStyle(uri)) {


DeduceUriStyle(uri).value_or(PATH_STYLE)

github-actions · 2025-01-15T12:15:25Z

⚪ 2025-01-15 12:15:25 UTC Pre-commit check linux-x86_64-release-asan for 8fd3ba8 has started.
⚪ 2025-01-15 12:15:36 UTC Artifacts will be uploaded here
⚪ 2025-01-15 12:18:28 UTC ya make is running...
🟡 2025-01-15 13:13:48 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11615	11564	0	23	3	25

⚪ 2025-01-15 13:14:51 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-01-15 13:34:27 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
88 (only retried tests)	60	0	1	1	26

⚪ 2025-01-15 13:34:36 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-01-15 13:45:53 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
49 (only retried tests)	24	0	1	0	24

🟢 2025-01-15 13:46:02 UTC Build successful.
🟢 2025-01-15 13:46:31 UTC ydbd size 3.6 GiB changed* by +72.5 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `4959d3e`	merge: `8fd3ba8`	diff	diff %
ydbd size	3 895 426 488 Bytes	3 895 500 744 Bytes	+72.5 KiB	+0.002%
ydbd stripped size	1 360 971 952 Bytes	1 360 992 848 Bytes	+20.4 KiB	+0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-01-15T12:16:57Z

⚪ 2025-01-15 12:16:56 UTC Pre-commit check linux-x86_64-relwithdebinfo for 8fd3ba8 has started.
⚪ 2025-01-15 12:17:01 UTC Artifacts will be uploaded here
⚪ 2025-01-15 12:19:52 UTC ya make is running...
🟡 2025-01-15 13:07:57 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
18745	17449	0	2	1169	125

⚪ 2025-01-15 13:09:31 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-01-15 13:21:46 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
184 (only retried tests)	66	0	0	0	118

🟢 2025-01-15 13:21:53 UTC Build successful.
🟢 2025-01-15 13:22:13 UTC ydbd size 2.1 GiB changed* by +42.9 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `4959d3e`	merge: `8fd3ba8`	diff	diff %
ydbd size	2 247 494 784 Bytes	2 247 538 752 Bytes	+42.9 KiB	+0.002%
ydbd stripped size	472 865 744 Bytes	472 871 312 Bytes	+5.4 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

extend s3 uri parsing for CS tiers

0c04c91

github-actions bot added the improvement label Jan 13, 2025

swalrus1 changed the title ~~extend s3 uri parsing for CS tiers~~ support folders in S3 uri in CS tiers Jan 13, 2025

github-actions bot added improvement and removed improvement labels Jan 13, 2025

fix test

8a74ee6

zverevgeny reviewed Jan 14, 2025

View reviewed changes

Update s3_uri.h

6cd42ba

zverevgeny reviewed Jan 14, 2025

View reviewed changes

swalrus1 added 2 commits January 14, 2025 15:36

remove regex

443a8ca

Merge branch 'parse-s3-uri' of github.com:swalrus1/ydb into parse-s3-uri

e5d8540

fix

eade249

swalrus1 marked this pull request as ready for review January 14, 2025 16:25

swalrus1 linked an issue Jan 15, 2025 that may be closed by this pull request

Support S3 buckets with folders in CS tiering #13378

Closed

zverevgeny reviewed Jan 15, 2025

View reviewed changes

zverevgeny approved these changes Jan 15, 2025

View reviewed changes

swalrus1 added the rebase-and-check Rebase PR with the current base branch and check label Jan 15, 2025

github-actions bot removed the rebase-and-check Rebase PR with the current base branch and check label Jan 15, 2025

swalrus1 merged commit 6a230d5 into ydb-platform:main Jan 15, 2025
17 of 18 checks passed

swalrus1 deleted the parse-s3-uri branch January 15, 2025 15:13

zverevgeny pushed a commit to zverevgeny/ydb that referenced this pull request Jan 21, 2025

support folders in S3 uri in CS tiers (ydb-platform#13337)

83f4579

zverevgeny mentioned this pull request Jan 21, 2025

Integrate changes from main #13202

Closed

azevaykin pushed a commit to azevaykin/ydb that referenced this pull request Feb 3, 2025

support folders in S3 uri in CS tiers (ydb-platform#13337)

c479667

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support folders in S3 uri in CS tiers #13337

support folders in S3 uri in CS tiers #13337

swalrus1 commented Jan 13, 2025

github-actions bot commented Jan 13, 2025 •

edited

Loading

github-actions bot commented Jan 13, 2025 •

edited

Loading

github-actions bot commented Jan 13, 2025 •

edited

Loading

github-actions bot commented Jan 13, 2025 •

edited

Loading

zverevgeny Jan 14, 2025

github-actions bot commented Jan 14, 2025 •

edited

Loading

github-actions bot commented Jan 14, 2025 •

edited

Loading

zverevgeny Jan 14, 2025

github-actions bot commented Jan 14, 2025 •

edited

Loading

github-actions bot commented Jan 14, 2025 •

edited

Loading

github-actions bot commented Jan 14, 2025 •

edited

Loading

github-actions bot commented Jan 14, 2025 •

edited

Loading

zverevgeny Jan 15, 2025

github-actions bot commented Jan 15, 2025 •

edited

Loading

github-actions bot commented Jan 15, 2025 •

edited

Loading

support folders in S3 uri in CS tiers #13337

support folders in S3 uri in CS tiers #13337

Conversation

swalrus1 commented Jan 13, 2025

Changelog entry

Changelog category

Additional information

github-actions bot commented Jan 13, 2025 • edited Loading

github-actions bot commented Jan 13, 2025 • edited Loading

github-actions bot commented Jan 13, 2025 • edited Loading

github-actions bot commented Jan 13, 2025 • edited Loading

zverevgeny Jan 14, 2025

Choose a reason for hiding this comment

github-actions bot commented Jan 14, 2025 • edited Loading

github-actions bot commented Jan 14, 2025 • edited Loading

zverevgeny Jan 14, 2025

Choose a reason for hiding this comment

github-actions bot commented Jan 14, 2025 • edited Loading

github-actions bot commented Jan 14, 2025 • edited Loading

github-actions bot commented Jan 14, 2025 • edited Loading

github-actions bot commented Jan 14, 2025 • edited Loading

zverevgeny Jan 15, 2025

Choose a reason for hiding this comment

github-actions bot commented Jan 15, 2025 • edited Loading

github-actions bot commented Jan 15, 2025 • edited Loading

github-actions bot commented Jan 13, 2025 •

edited

Loading

github-actions bot commented Jan 13, 2025 •

edited

Loading

github-actions bot commented Jan 13, 2025 •

edited

Loading

github-actions bot commented Jan 13, 2025 •

edited

Loading

github-actions bot commented Jan 14, 2025 •

edited

Loading

github-actions bot commented Jan 14, 2025 •

edited

Loading

github-actions bot commented Jan 14, 2025 •

edited

Loading

github-actions bot commented Jan 14, 2025 •

edited

Loading

github-actions bot commented Jan 14, 2025 •

edited

Loading

github-actions bot commented Jan 14, 2025 •

edited

Loading

github-actions bot commented Jan 15, 2025 •

edited

Loading

github-actions bot commented Jan 15, 2025 •

edited

Loading