Test deleting from s3 by ttl #13533

zverevgeny · 2025-01-19T14:13:43Z

Changelog category

Not for changelog (changelog entry is not required)

github-actions · 2025-01-19T14:15:48Z

⚪ 2025-01-19 14:15:47 UTC Pre-commit check linux-x86_64-release-asan for 2e031bd has started.
⚪ 2025-01-19 14:16:18 UTC Artifacts will be uploaded here
⚪ 2025-01-19 14:18:40 UTC ya make is running...
🟢 2025-01-19 14:18:45 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
0	0	0	0	0	0

🟢 2025-01-19 14:18:50 UTC Build successful.

github-actions · 2025-01-19T14:17:16Z

⚪ 2025-01-19 14:17:16 UTC Pre-commit check linux-x86_64-relwithdebinfo for 2e031bd has started.
⚪ 2025-01-19 14:17:27 UTC Artifacts will be uploaded here
⚪ 2025-01-19 14:19:55 UTC ya make is running...
🟢 2025-01-19 14:20:00 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
0	0	0	0	0	0

🟢 2025-01-19 14:20:05 UTC Build successful.

github-actions · 2025-01-20T13:57:29Z

⚪ 2025-01-20 13:57:29 UTC Pre-commit check linux-x86_64-relwithdebinfo for d5df818 has started.
⚪ 2025-01-20 13:57:41 UTC Artifacts will be uploaded here
⚪ 2025-01-20 14:00:16 UTC ya make is running...
🟢 2025-01-20 14:00:22 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
0	0	0	0	0	0

🟢 2025-01-20 14:00:28 UTC Build successful.

github-actions · 2025-01-20T13:57:29Z

⚪ 2025-01-20 13:57:28 UTC Pre-commit check linux-x86_64-release-asan for d5df818 has started.
⚪ 2025-01-20 13:57:41 UTC Artifacts will be uploaded here
⚪ 2025-01-20 14:00:03 UTC ya make is running...
🟢 2025-01-20 14:00:14 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
0	0	0	0	0	0

🟢 2025-01-20 14:00:21 UTC Build successful.

github-actions · 2025-01-20T14:19:35Z

⚪ 2025-01-20 14:19:35 UTC Pre-commit check linux-x86_64-release-asan for b06e31f has started.
⚪ 2025-01-20 14:19:41 UTC Artifacts will be uploaded here
⚪ 2025-01-20 14:22:05 UTC ya make is running...
🟢 2025-01-20 14:22:11 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
0	0	0	0	0	0

🟢 2025-01-20 14:22:17 UTC Build successful.

github-actions · 2025-01-20T14:19:44Z

⚪ 2025-01-20 14:19:44 UTC Pre-commit check linux-x86_64-relwithdebinfo for b06e31f has started.
⚪ 2025-01-20 14:19:55 UTC Artifacts will be uploaded here
⚪ 2025-01-20 14:22:26 UTC ya make is running...
🟢 2025-01-20 14:22:32 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
0	0	0	0	0	0

🟢 2025-01-20 14:22:37 UTC Build successful.

github-actions · 2025-01-20T14:26:09Z

⚪ 2025-01-20 14:26:09 UTC Pre-commit check linux-x86_64-relwithdebinfo for 5607227 has started.
⚪ 2025-01-20 14:26:20 UTC Artifacts will be uploaded here
⚪ 2025-01-20 14:28:50 UTC ya make is running...
🟢 2025-01-20 14:34:22 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
7	7	0	0	0	0

🟢 2025-01-20 14:34:29 UTC Build successful.
🟢 2025-01-20 14:34:38 UTC ydbd size 2.1 GiB changed* by +34.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `04f4156`	merge: `5607227`	diff	diff %
ydbd size	2 216 627 368 Bytes	2 216 662 904 Bytes	+34.7 KiB	+0.002%
ydbd stripped size	468 629 104 Bytes	468 635 152 Bytes	+5.9 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-01-20T14:26:10Z

⚪ 2025-01-20 14:26:09 UTC Pre-commit check linux-x86_64-release-asan for 5607227 has started.
⚪ 2025-01-20 14:26:22 UTC Artifacts will be uploaded here
⚪ 2025-01-20 14:28:46 UTC ya make is running...
🟢 2025-01-20 14:38:37 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
5	5	0	0	0	0

🟢 2025-01-20 14:38:44 UTC Build successful.
🟢 2025-01-20 14:38:56 UTC ydbd size 3.6 GiB changed* by +72.8 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `04f4156`	merge: `5607227`	diff	diff %
ydbd size	3 852 980 728 Bytes	3 853 055 320 Bytes	+72.8 KiB	+0.002%
ydbd stripped size	1 346 195 888 Bytes	1 346 223 856 Bytes	+27.3 KiB	+0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

maximyurchuk · 2025-01-20T16:34:45Z

ydb/tests/olap/ttl_tiering/base.py

+        return {row["TierName"]: {"Portions": row["Portions"], "BlobSize": row["BlobSize"], "BlobCount": row["BlobCount"]} for result_set in results for row in result_set.rows}
+
+    def get_stat(self):
+        pass


Недописано?

удалил

maximyurchuk · 2025-01-20T16:35:40Z

ydb/tests/olap/ttl_tiering/base.py

+    def teardown_class(cls):
+        cls.ydb_client.stop()
+        cls.cluster.stop()
+        recipes_common.stop_daemon(cls.s3_pid)


Меня смущает что порядок инициализации-деинициализации не зеркальный

поправил

maximyurchuk · 2025-01-20T16:37:07Z

ydb/tests/olap/ttl_tiering/base.py

+
+    @classmethod
+    def _setup_ydb(cls):
+        ydb_path = yatest.common.build_path(os.environ.get("YDB_DRIVER_BINARY", "ydb/apps/ydbd/ydbd"))


Вот тут не имеет смысла дефолтное значение как будто

Потому что env задается прям на уровне ya.make

Ну либо я задумки ен понял

поправил

maximyurchuk · 2025-01-20T16:37:53Z

ydb/tests/olap/ttl_tiering/base.py

+        ydb_path = yatest.common.build_path(os.environ.get("YDB_DRIVER_BINARY", "ydb/apps/ydbd/ydbd"))
+        logger.info(yatest.common.execute([ydb_path, "-V"], wait=True).stdout.decode("utf-8"))
+        config = KikimrConfigGenerator(
+            erasure=Erasure.MIRROR_3_DC,


Вот тут аккуратнее: нужен ли тебе прям многонодный кластер? Если достаточно однонодного, то лучше однонодный, потому что много ресурсов на ci оно жрет

оставил дефолт

maximyurchuk · 2025-01-20T16:39:44Z

ydb/tests/olap/ttl_tiering/ttl_delete_s3.py

+    cold_bucket = "cold"
+    frozen_bucket = "frozen"
+    days_to_cool = 1000
+    days_to_froze = 3000


days_to_freeze все таки наверное

поправил

maximyurchuk · 2025-01-20T16:40:19Z

ydb/tests/olap/ttl_tiering/ttl_delete_s3.py

+logger = logging.getLogger(__name__)
+
+
+class TestDeleteS3Ttl(TllTieringTestBase):


Я вот не уверен до конца что имеет смысл разбивать на два классе TllTieringTestBase/TestDeleteS3Ttl но смотри сам

предполагается переиспользование другими тестами тиринга

maximyurchuk · 2025-01-20T16:41:14Z

ydb/tests/olap/ttl_tiering/ttl_delete_s3.py

+    def get_row_count_by_date(self, table_path: str, past_days: int) -> int:
+        return self.ydb_client.query(f"SELECT count(*) as Rows from `{table_path}` WHERE ts < CurrentUtcTimestamp() - DateTime::IntervalFromDays({past_days})")[0].rows[0]["Rows"]
+
+    def test(self):


Может стоит написать комментарий про то что тест делает? Типа чтобы весь код не читать

В докстринге к классу есть ссылка на описание

github-actions · 2025-01-20T17:44:43Z

⚪ 2025-01-20 17:44:43 UTC Pre-commit check linux-x86_64-release-asan for cbc40d5 has started.
⚪ 2025-01-20 17:44:54 UTC Artifacts will be uploaded here
⚪ 2025-01-20 17:47:18 UTC ya make is running...
🟢 2025-01-20 18:14:57 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
5	5	0	0	0	0

🟢 2025-01-20 18:15:08 UTC Build successful.
🟡 2025-01-20 18:15:20 UTC ydbd size 3.6 GiB changed* by +313.7 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `dbe9d9b`	merge: `cbc40d5`	diff	diff %
ydbd size	3 853 083 840 Bytes	3 853 405 104 Bytes	+313.7 KiB	+0.008%
ydbd stripped size	1 346 233 456 Bytes	1 346 322 160 Bytes	+86.6 KiB	+0.007%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-01-20T17:46:03Z

⚪ 2025-01-20 17:46:03 UTC Pre-commit check linux-x86_64-relwithdebinfo for cbc40d5 has started.
⚪ 2025-01-20 17:46:07 UTC Artifacts will be uploaded here
⚪ 2025-01-20 17:48:30 UTC ya make is running...
🟢 2025-01-20 17:55:20 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
7	7	0	0	0	0

🟢 2025-01-20 17:55:26 UTC Build successful.
🟡 2025-01-20 17:55:35 UTC ydbd size 2.1 GiB changed* by +208.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `dbe9d9b`	merge: `cbc40d5`	diff	diff %
ydbd size	2 216 675 360 Bytes	2 216 889 232 Bytes	+208.9 KiB	+0.010%
ydbd stripped size	468 634 832 Bytes	468 664 560 Bytes	+29.0 KiB	+0.006%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

Test deleting from s3 by ttl

f4b5d16

github-actions bot added the not-for-changelog label Jan 19, 2025

Move to proper place, convert to yatest

cdff865

zverevgeny added 2 commits January 20, 2025 16:57

fix rename

6c25cb9

fix bugs

a410ecb

zverevgeny marked this pull request as ready for review January 20, 2025 14:19

zverevgeny requested a review from maximyurchuk January 20, 2025 14:23

add recurse

be06b15

maximyurchuk reviewed Jan 20, 2025

View reviewed changes

maximyurchuk previously approved these changes Jan 20, 2025

View reviewed changes

minor fixes

310bbfc

zverevgeny dismissed maximyurchuk’s stale review via 310bbfc January 20, 2025 17:42

iddqdex approved these changes Jan 20, 2025

View reviewed changes

zverevgeny merged commit fdde655 into ydb-platform:main Jan 20, 2025
12 checks passed

zverevgeny linked an issue Jan 20, 2025 that may be closed by this pull request

Test tiering. Functional. When configuring DELETE tier, data evaporates from S3 #13467

Open

the-ancient-1 pushed a commit to the-ancient-1/ydb that referenced this pull request Jan 21, 2025

Test deleting from s3 by ttl (ydb-platform#13533)

1f1201e

zverevgeny mentioned this pull request Jan 21, 2025

Integrate changes from main #13202

Closed

azevaykin pushed a commit to azevaykin/ydb that referenced this pull request Feb 3, 2025

Test deleting from s3 by ttl (ydb-platform#13533)

3dad3c0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test deleting from s3 by ttl #13533

Test deleting from s3 by ttl #13533

zverevgeny commented Jan 19, 2025

github-actions bot commented Jan 19, 2025 •

edited

Loading

github-actions bot commented Jan 19, 2025 •

edited

Loading

github-actions bot commented Jan 20, 2025 •

edited

Loading

github-actions bot commented Jan 20, 2025 •

edited

Loading

github-actions bot commented Jan 20, 2025 •

edited

Loading

github-actions bot commented Jan 20, 2025 •

edited

Loading

github-actions bot commented Jan 20, 2025 •

edited

Loading

github-actions bot commented Jan 20, 2025 •

edited

Loading

maximyurchuk Jan 20, 2025

zverevgeny Jan 20, 2025

maximyurchuk Jan 20, 2025

zverevgeny Jan 20, 2025

maximyurchuk Jan 20, 2025

zverevgeny Jan 20, 2025

maximyurchuk Jan 20, 2025

zverevgeny Jan 20, 2025

maximyurchuk Jan 20, 2025

zverevgeny Jan 20, 2025

maximyurchuk Jan 20, 2025

zverevgeny Jan 20, 2025

maximyurchuk Jan 20, 2025

zverevgeny Jan 20, 2025

github-actions bot commented Jan 20, 2025 •

edited

Loading

github-actions bot commented Jan 20, 2025 •

edited

Loading

		logger = logging.getLogger(__name__)


		class TestDeleteS3Ttl(TllTieringTestBase):

Test deleting from s3 by ttl #13533

Test deleting from s3 by ttl #13533

Conversation

zverevgeny commented Jan 19, 2025

Changelog category

github-actions bot commented Jan 19, 2025 • edited Loading

github-actions bot commented Jan 19, 2025 • edited Loading

github-actions bot commented Jan 20, 2025 • edited Loading

github-actions bot commented Jan 20, 2025 • edited Loading

github-actions bot commented Jan 20, 2025 • edited Loading

github-actions bot commented Jan 20, 2025 • edited Loading

github-actions bot commented Jan 20, 2025 • edited Loading

github-actions bot commented Jan 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jan 20, 2025 • edited Loading

github-actions bot commented Jan 20, 2025 • edited Loading

github-actions bot commented Jan 19, 2025 •

edited

Loading

github-actions bot commented Jan 19, 2025 •

edited

Loading

github-actions bot commented Jan 20, 2025 •

edited

Loading

github-actions bot commented Jan 20, 2025 •

edited

Loading

github-actions bot commented Jan 20, 2025 •

edited

Loading

github-actions bot commented Jan 20, 2025 •

edited

Loading

github-actions bot commented Jan 20, 2025 •

edited

Loading

github-actions bot commented Jan 20, 2025 •

edited

Loading

github-actions bot commented Jan 20, 2025 •

edited

Loading

github-actions bot commented Jan 20, 2025 •

edited

Loading