Index build: do not lose the requested partitioning info of indexImplTables in case of SchemeShard reboots

[jepett0]

Ticket:

• index build: persist IndexImplTables' descriptions #6743

Context

Index building is a long-running operation that is not a single transaction, but many. YDB does persist the state of the individual transactions (i.e. SchemeShard's proposals) and is able to continue them from the last state even if the SchemeShard reboots during the execution of the transaction. However, index build operations require their own persistent description for the SchemeShard to continue them if it reboots.

Description

PersistCreateBuildIndex is a function that is responsible for the index build operation ability to recover from the SchemeShard's reboots. As you can see, it didn't persist the ImplTableDescriptions (they store the partitioning info about the indexImplTables to be created) as well as the Specialized Index Description (i.e. specific vector index description). Previously, if the SchemeShard had rebooted before the indexImplTable was created, but after the index build operation had started, then the partitioning info of the indexImplTables would have been lost. A unit test Is added to reproduce the scenario.

Note

The main commit is:

• a0b4306

The others are supplementary (they add unit tests).

[github-actions]

⚪ 2024-10-18 16:45:48 UTC Pre-commit check linux-x86_64-relwithdebinfo for 5dea000 has started.
⚪ 2024-10-18 16:45:58 UTC Artifacts will be uploaded here
⚪ 2024-10-18 16:48:58 UTC ya make is running...
🟡 2024-10-18 18:10:02 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
40319	34555	0	4	5727	33

⚪ 2024-10-18 18:13:27 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-10-18 18:24:57 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
269 (only retried tests)	171	0	0	7	91

🟢 2024-10-18 18:25:05 UTC Build successful.
🟢 2024-10-18 18:25:24 UTC ydbd size 2.8 GiB changed* by +72.9 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: f5d549e	merge: 5dea000	diff	diff %
ydbd size	3 013 067 224 Bytes	3 013 141 896 Bytes	+72.9 KiB	+0.002%
ydbd stripped size	477 721 976 Bytes	477 730 072 Bytes	+7.9 KiB	+0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

[github-actions]

⚪ 2024-10-18 16:46:25 UTC Pre-commit check linux-x86_64-release-asan for 5dea000 has started.
⚪ 2024-10-18 16:46:37 UTC Artifacts will be uploaded here
⚪ 2024-10-18 16:49:35 UTC ya make is running...
🟡 2024-10-18 18:39:44 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
10580	10337	0	77	59	107

🟢 2024-10-18 18:40:40 UTC Build successful.
🟡 2024-10-18 18:41:14 UTC ydbd size 5.7 GiB changed* by +120.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: f5d549e	merge: 5dea000	diff	diff %
ydbd size	6 104 513 032 Bytes	6 104 636 824 Bytes	+120.9 KiB	+0.002%
ydbd stripped size	1 521 867 664 Bytes	1 521 882 192 Bytes	+14.2 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}