Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Correctly validate joins between windowed and non-windowed sources #4390

Merged
merged 4 commits into from
Jan 29, 2020
Merged

Correctly validate joins between windowed and non-windowed sources #4390

merged 4 commits into from
Jan 29, 2020

Conversation

big-andy-coates
Copy link
Contributor

@big-andy-coates big-andy-coates commented Jan 27, 2020 •
edited
Loading

Description

Fixes: #4389

Previously, KSQL did nothing to validate a join between two sources had keys with compatible windowing.

  • it did not check that both sources were either windowed or non-windowed. Joining a non-windowed source to a windowed source will never result in any output as the binary keys can never match.
  • it did not check that both sources had compatible window types. Session windows include both window start and end times in the serialized key, where as hopping and tumbling windows only serialize the window start. For this reason, session windowed sources can only be joined with other session windowed sources. Hopping and tumbling windowed sources can only be joined with other hopping and tumbling windowed sources.

This commit adds these checks and suitable QTT tests to prove them.

Testing done

test cases added to QTT

Reviewer checklist

  • Ensure docs are updated if necessary. (eg. if a user visible feature is being added or changed).
  • Ensure relevant issues are linked (description should include text like "Fixes #")

@big-andy-coates
chore: fail on joins with incompatible window/non-windowed keys
76e6499 
Previously, KSQL did nothing to validate a join between two sources had keys with compatible windowing.

* it did not check that both sources were either windowed or non-windowed.  Joining a non-windowed source to a windowed source will never result in any output as the binary keys can never match.
* it did not check that both sources had compatible window types. Session windows include both window start and end times in the serialized key, where as hopping and tumbling windows only serialize the window start. For this reason, session windowed sources can only be joined with other session windowed sources. Hopping and tumbling windowed sources can only be joined with other hopping and tumbling windowed sources.

This commit adds these checks and suitable QTT tests to prove them.
@big-andy-coates
chore: fix test
234f733
@big-andy-coates big-andy-coates requested a review from a team as a code owner January 27, 2020 19:26
vpapavas
vpapavas approved these changes Jan 29, 2020
View reviewed changes
Copy link
Member

@vpapavas vpapavas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@big-andy-coates big-andy-coates merged commit c1faef9 into confluentinc:master Jan 29, 2020
@big-andy-coates big-andy-coates deleted the windowed_joins branch January 29, 2020 11:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vpapavas vpapavas vpapavas approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Ensure joins for windows keys work
2 participants
@big-andy-coates @vpapavas