Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

grace join: add bloom filters #6951

Merged
merged 9 commits into from
Aug 1, 2024

Conversation

yumkam
Copy link
Collaborator

@yumkam yumkam commented Jul 22, 2024

Changelog entry

...

Changelog category

  • Performance improvement
  • Not for changelog (changelog entry is not required)

Additional information

Closes #6991

Copy link

github-actions bot commented Jul 22, 2024

2024-07-22 14:26:06 UTC Pre-commit check for 8ccfee0 has started.
2024-07-22 14:29:25 UTC Build+Tests linux-x86_64-release-asan is running...
🟢 2024-07-22 16:36:51 UTC Build successful.
🔴 2024-07-22 16:40:47 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11966 11359 0 96 341 170

🟢 2024-07-22 16:41:44 UTC ydbd size 5.2 GiB changed* by +70.5 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 427c5e6 merge: 8ccfee0 diff diff %
ydbd size 5 557 031 536 Bytes 5 557 103 736 Bytes +70.5 KiB +0.001%
ydbd stripped size 1 196 983 568 Bytes 1 197 003 536 Bytes +19.5 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 22, 2024

2024-07-22 14:27:16 UTC Pre-commit check for 8ccfee0 has started.
2024-07-22 14:30:24 UTC Build+Tests linux-x86_64-release-clang14 is running...
🟢 2024-07-22 14:35:20 UTC Build successful.

Copy link

github-actions bot commented Jul 22, 2024

2024-07-22 14:37:08 UTC Pre-commit check for 8ccfee0 has started.
2024-07-22 14:39:51 UTC Build+Tests linux-x86_64-relwithdebinfo is running...
🟢 2024-07-22 16:44:46 UTC Build successful.
🔴 2024-07-22 16:52:52 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
73047 59592 0 5 13437 13

🟢 2024-07-22 16:53:34 UTC ydbd size 8.1 GiB changed* by +68.4 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 427c5e6 merge: 8ccfee0 diff diff %
ydbd size 8 659 284 416 Bytes 8 659 354 408 Bytes +68.4 KiB +0.001%
ydbd stripped size 471 148 704 Bytes 471 155 808 Bytes +6.9 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

yumkam added 8 commits July 29, 2024 13:04
keys with NULL are not equal to anything else and should be considered unique
when we definitely know row on one side won't match any row on other
side (e.g. NULL in keys), we can skip processing and ignore or return
row with NULL added

any side of INNER, LEFT SEMI, RIGHT SEMI, right side of LEFT, LEFT ONLY, left side of RIGHT, RIGHT ONLY
-> skip/ignore

left side of LEFT, LEFT ONLY, right side of RIGHT, RIGHT ONLY, any side of FULL, EXCLUSION:
-> add NULL to other side and return

kind       | left side          | right side
-----------+--------------------+-----------
INNER      | drop               | drop
LEFT SEMI  | drop               | drop
RIGHT SEMI | drop               | drop
LEFT       | return row + NULL  | drop
LEFT ONLY  | return row + NULL  | drop
RIGHT      | drop               | return NULL + row
RIGHT ONLY | drop               | return NULL + row
FULL       | return row + NULL  | return NULL + row
EXCLUSION  | return row + NULL  | return NULL + row
try to use it once, if it turned out to be inefficient -- just drop it
@yumkam yumkam force-pushed the grace-join-bloom-filters branch from 8e3da99 to 21f6246 Compare July 29, 2024 11:24
Copy link

github-actions bot commented Jul 29, 2024

2024-07-29 11:28:28 UTC Pre-commit check for 530ed1a has started.
2024-07-29 11:31:24 UTC Check linux-x86_64-relwithdebinfo is running...
🟢 2024-07-29 13:40:55 UTC Tests successful.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
73391 59830 0 0 13548 13

🟢 2024-07-29 13:48:38 UTC Build successful.
🟢 2024-07-29 13:49:24 UTC ydbd size 8.1 GiB changed* by +69.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: bd147a9 merge: 530ed1a diff diff %
ydbd size 8 676 405 096 Bytes 8 676 476 424 Bytes +69.7 KiB +0.001%
ydbd stripped size 471 815 968 Bytes 471 823 200 Bytes +7.1 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 29, 2024

2024-07-29 11:28:54 UTC Pre-commit check for 530ed1a has started.
2024-07-29 11:32:08 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-07-29 11:38:36 UTC Build successful.

Copy link

github-actions bot commented Jul 29, 2024

2024-07-29 11:31:21 UTC Pre-commit check for 530ed1a has started.
2024-07-29 11:34:43 UTC Check linux-x86_64-release-asan is running...
🔴 2024-07-29 13:46:55 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
12010 11421 0 84 335 170

🟢 2024-07-29 13:48:11 UTC Build successful.
🟢 2024-07-29 13:49:02 UTC ydbd size 5.2 GiB changed* by +80.4 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: bd147a9 merge: 530ed1a diff diff %
ydbd size 5 569 932 912 Bytes 5 570 015 224 Bytes +80.4 KiB +0.001%
ydbd stripped size 1 199 041 552 Bytes 1 199 066 192 Bytes +24.1 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@yumkam yumkam changed the title [WIP] grace join: add bloom filters grace join: add bloom filters Jul 29, 2024
Copy link

github-actions bot commented Jul 29, 2024

2024-07-29 23:11:16 UTC Pre-commit check for 46c05d3 has started.
2024-07-29 23:14:18 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-07-30 01:21:31 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
73413 59848 0 2 13551 12

2024-07-30 01:29:14 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟢 2024-07-30 01:37:18 UTC Tests successful.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
12 (only retried tests) 4 0 0 0 8

🟢 2024-07-30 01:44:15 UTC Build successful.
🟢 2024-07-30 01:44:50 UTC ydbd size 8.1 GiB changed* by +70.4 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 0bc0896 merge: 46c05d3 diff diff %
ydbd size 8 676 853 056 Bytes 8 676 925 128 Bytes +70.4 KiB +0.001%
ydbd stripped size 471 825 824 Bytes 471 833 056 Bytes +7.1 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Jul 29, 2024

2024-07-29 23:11:40 UTC Pre-commit check for 46c05d3 has started.
2024-07-29 23:14:41 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-07-29 23:28:40 UTC Build successful.

Copy link

github-actions bot commented Jul 29, 2024

2024-07-29 23:11:56 UTC Pre-commit check for 46c05d3 has started.
2024-07-29 23:14:51 UTC Check linux-x86_64-release-asan is running...
🔴 2024-07-30 01:29:10 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
12021 11435 0 80 334 172

🟢 2024-07-30 01:30:19 UTC Build successful.
🟢 2024-07-30 01:31:08 UTC ydbd size 5.2 GiB changed* by +82.2 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 0bc0896 merge: 46c05d3 diff diff %
ydbd size 5 570 228 600 Bytes 5 570 312 816 Bytes +82.2 KiB +0.002%
ydbd stripped size 1 199 072 560 Bytes 1 199 097 328 Bytes +24.2 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@yumkam yumkam marked this pull request as ready for review July 30, 2024 13:56
@yumkam yumkam requested a review from a team as a code owner July 30, 2024 13:56
@yumkam yumkam requested a review from lll-phill-lll July 30, 2024 14:16
@yumkam yumkam marked this pull request as draft July 30, 2024 17:39
@yumkam yumkam requested a review from Tony-Romanov August 1, 2024 11:05
@yumkam yumkam marked this pull request as ready for review August 1, 2024 12:01
@yumkam yumkam requested a review from aakulaga-ydb August 1, 2024 12:01
@aakulaga-ydb aakulaga-ydb merged commit 774401f into ydb-platform:main Aug 1, 2024
10 of 12 checks passed
yumkam added a commit to yumkam/ydb that referenced this pull request Oct 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Динамические bloom фильтры в join
2 participants