Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce computation node for BlockMapJoinCore (LeftOnly and LeftSemi) #7947

Conversation

igormunkin
Copy link
Collaborator

@igormunkin igormunkin commented Aug 16, 2024

This patchset is the first part of the series implementing BlockMapJoinCore MKQL callable. This part implements BlockMapJoinCore computation node for the join types, that don't require "dict" payload (i.e. LeftSeim and LeftOnly), so MKQL version is bumped either. The support for other MapJoinCore types are implemented in scope of the next PRs.

By the way the new flag is added to ValidateBlock*Type helpers to control whether the resulting item types are block or "raw" (i.e. unwrapped).

Changelog category

  • New feature

Copy link

github-actions bot commented Aug 16, 2024

2024-08-16 18:10:40 UTC Pre-commit check for 992cb98 has started.
2024-08-16 18:13:45 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-16 20:36:25 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
15480 15396 0 17 33 34

🟢 2024-08-16 20:37:41 UTC Build successful.
🔴 2024-08-16 20:38:07 UTC ydbd size 5.5 GiB changed* by +4.3 MiB, which is >= 2.0 MiB vs main: Alert

ydbd size dash main: 8dc96f7 merge: 992cb98 diff diff %
ydbd size 5 877 295 352 Bytes 5 881 804 264 Bytes +4.3 MiB +0.077%
ydbd stripped size 1 476 632 880 Bytes 1 477 109 808 Bytes +465.8 KiB +0.032%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Aug 16, 2024

2024-08-16 18:15:24 UTC Pre-commit check for 992cb98 has started.
2024-08-16 18:18:36 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-08-16 19:53:43 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
77142 63500 0 5 13599 38

2024-08-16 20:01:08 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟡 2024-08-16 20:09:53 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
51 (only retried tests) 22 0 1 0 28

2024-08-16 20:10:01 UTC Failed tests rerun (try 3) linux-x86_64-relwithdebinfo is running...
🟢 2024-08-16 20:17:08 UTC Tests successful.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
44 (only retried tests) 17 0 0 0 27

🟢 2024-08-16 20:17:15 UTC Build successful.
🔴 2024-08-16 20:17:49 UTC ydbd size 8.1 GiB changed* by +6.1 MiB, which is >= 2.0 MiB vs main: Alert

ydbd size dash main: 20635ca merge: 992cb98 diff diff %
ydbd size 8 732 615 232 Bytes 8 738 982 488 Bytes +6.1 MiB +0.073%
ydbd stripped size 475 287 976 Bytes 475 410 568 Bytes +119.7 KiB +0.026%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Aug 16, 2024

2024-08-16 18:23:23 UTC Pre-commit check for 992cb98 has started.
2024-08-16 18:26:38 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-16 18:46:14 UTC Build successful.

@igormunkin igormunkin force-pushed the YQL-18498-block-map-join-comp-node-wo-right branch from 9030f7d to b1366fc Compare August 19, 2024 06:36
Copy link

github-actions bot commented Aug 19, 2024

2024-08-19 06:40:03 UTC Pre-commit check for df7fd5d has started.
2024-08-19 06:43:08 UTC Check linux-x86_64-release-clang14 is running...
2024-08-19 06:57:29 UTC Check cancelled

Copy link

github-actions bot commented Aug 19, 2024

2024-08-19 06:40:19 UTC Pre-commit check for df7fd5d has started.
2024-08-19 06:43:19 UTC Check linux-x86_64-release-asan is running...
2024-08-19 06:57:29 UTC Check cancelled

Copy link

github-actions bot commented Aug 19, 2024

2024-08-19 06:40:23 UTC Pre-commit check for df7fd5d has started.
2024-08-19 06:43:21 UTC Check linux-x86_64-relwithdebinfo is running...
2024-08-19 06:57:29 UTC Check cancelled

@igormunkin igormunkin marked this pull request as ready for review August 19, 2024 06:56
@igormunkin igormunkin requested a review from a team as a code owner August 19, 2024 06:56
@igormunkin igormunkin requested a review from vitstn August 19, 2024 06:56
@igormunkin igormunkin added the area/yql YQL query language issues label Aug 19, 2024
@igormunkin igormunkin self-assigned this Aug 19, 2024
Copy link

github-actions bot commented Aug 19, 2024

2024-08-19 06:58:55 UTC Pre-commit check for df7fd5d has started.
2024-08-19 07:02:13 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-19 07:16:24 UTC Build successful.

Copy link

github-actions bot commented Aug 19, 2024

2024-08-19 06:59:32 UTC Pre-commit check for df7fd5d has started.
2024-08-19 07:02:37 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-08-19 08:39:19 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
77177 63532 0 2 13604 39

2024-08-19 08:46:35 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟢 2024-08-19 08:55:20 UTC Tests successful.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
50 (only retried tests) 21 0 0 0 29

🟢 2024-08-19 08:55:28 UTC Build successful.
🔴 2024-08-19 08:56:02 UTC ydbd size 8.1 GiB changed* by +3.5 MiB, which is >= 2.0 MiB vs main: Alert

ydbd size dash main: 5522a96 merge: df7fd5d diff diff %
ydbd size 8 736 888 376 Bytes 8 740 605 312 Bytes +3.5 MiB +0.043%
ydbd stripped size 475 560 200 Bytes 475 604 104 Bytes +42.9 KiB +0.009%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Aug 19, 2024

2024-08-19 07:00:49 UTC Pre-commit check for df7fd5d has started.
2024-08-19 07:03:47 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-19 09:24:41 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
15476 15397 0 16 32 31

🟢 2024-08-19 09:26:01 UTC Build successful.
🔴 2024-08-19 09:26:28 UTC ydbd size 5.5 GiB changed* by +2.5 MiB, which is >= 2.0 MiB vs main: Alert

ydbd size dash main: 274bfdd merge: df7fd5d diff diff %
ydbd size 5 880 316 984 Bytes 5 882 928 408 Bytes +2.5 MiB +0.044%
ydbd stripped size 1 477 405 904 Bytes 1 477 534 416 Bytes +125.5 KiB +0.009%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

}

const auto dictList = pgmBuilder.NewList(dictStructType, dictListItems);
return pgmBuilder.ToSortedDict(dictList, false,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

по умолчанию лучше использовать hashed dict

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@igormunkin igormunkin force-pushed the YQL-18498-block-map-join-comp-node-wo-right branch from b1366fc to 887c78b Compare August 19, 2024 14:44
Copy link

github-actions bot commented Aug 19, 2024

2024-08-19 14:48:05 UTC Pre-commit check for c32b410 has started.
2024-08-19 14:51:15 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-19 15:19:37 UTC Build successful.

Copy link

github-actions bot commented Aug 19, 2024

2024-08-19 14:48:08 UTC Pre-commit check for c32b410 has started.
2024-08-19 14:52:26 UTC Check linux-x86_64-release-asan is running...
2024-08-19 15:21:27 UTC Check cancelled

Copy link

github-actions bot commented Aug 19, 2024

2024-08-19 14:48:43 UTC Pre-commit check for c32b410 has started.
2024-08-19 14:52:36 UTC Check linux-x86_64-relwithdebinfo is running...
2024-08-19 15:21:24 UTC Check cancelled

Copy link

github-actions bot commented Aug 19, 2024

2024-08-19 15:22:19 UTC Pre-commit check for ee0c706 has started.
2024-08-19 15:25:57 UTC Check linux-x86_64-release-asan is running...
2024-08-19 16:45:25 UTC Check cancelled

Copy link

github-actions bot commented Aug 19, 2024

2024-08-19 15:24:48 UTC Pre-commit check for ee0c706 has started.
2024-08-19 15:27:52 UTC Check linux-x86_64-relwithdebinfo is running...
2024-08-19 16:45:21 UTC Check cancelled

Copy link

github-actions bot commented Aug 19, 2024

2024-08-19 15:24:59 UTC Pre-commit check for ee0c706 has started.
2024-08-19 15:27:54 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-19 15:33:43 UTC Build successful.

@igormunkin igormunkin requested a review from vitstn August 19, 2024 15:37
Copy link

github-actions bot commented Aug 19, 2024

2024-08-19 16:48:20 UTC Pre-commit check for ba0329e has started.
2024-08-19 16:51:17 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-19 17:00:30 UTC Build successful.

Copy link

github-actions bot commented Aug 19, 2024

2024-08-19 16:48:23 UTC Pre-commit check for ba0329e has started.
2024-08-19 16:51:18 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-19 19:17:00 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
15545 15464 0 12 33 36

🟢 2024-08-19 19:18:27 UTC Build successful.
🔴 2024-08-19 19:18:54 UTC ydbd size 5.5 GiB changed* by +2.5 MiB, which is >= 2.0 MiB vs main: Alert

ydbd size dash main: 9f309c4 merge: ba0329e diff diff %
ydbd size 5 883 817 464 Bytes 5 886 488 760 Bytes +2.5 MiB +0.045%
ydbd stripped size 1 478 040 816 Bytes 1 478 180 368 Bytes +136.3 KiB +0.009%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Aug 19, 2024

2024-08-19 16:50:19 UTC Pre-commit check for ba0329e has started.
2024-08-19 16:53:26 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-08-19 18:34:19 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
77239 63590 0 6 13605 38

2024-08-19 18:41:48 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟢 2024-08-19 18:49:06 UTC Tests successful.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
54 (only retried tests) 27 0 0 0 27

🟢 2024-08-19 18:49:14 UTC Build successful.
🔴 2024-08-19 18:49:47 UTC ydbd size 8.1 GiB changed* by +3.5 MiB, which is >= 2.0 MiB vs main: Alert

ydbd size dash main: 9f309c4 merge: ba0329e diff diff %
ydbd size 8 743 498 016 Bytes 8 747 171 360 Bytes +3.5 MiB +0.042%
ydbd stripped size 475 706 408 Bytes 475 745 768 Bytes +38.4 KiB +0.008%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@vitstn vitstn requested a review from nepal August 19, 2024 16:51
@igormunkin igormunkin merged commit b0abac7 into ydb-platform:main Aug 20, 2024
11 of 13 checks passed
@igormunkin igormunkin deleted the YQL-18498-block-map-join-comp-node-wo-right branch August 21, 2024 10:41
stanislav-shchetinin pushed a commit to stanislav-shchetinin/ydb that referenced this pull request Aug 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/yql YQL query language issues new-feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants