Partial support for arbitrary key column names. #4701
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Partial fix for: confluentinc#3536 First part of supporting key column names other than `ROWKEY`. With this initial pass you can now name your key columns anything you want in your `CREATE TABLE` and `CREATE STREAM` statements, e.g. ```sql CREATE STREAM S (ID INT KEY, NAME STRING) WITH (...); ``` Any GROUP BY, PARTITION BY or JOIN on the key column results any created data source having a key column with a matching name, e.g. ```sql -- schema of T: ID INT KEY, COUNT BIGINT CREATE TABLE T AS SELECT COUNT() AS COUNT FROM S GROUP BY ID; ``` Pull and push queries work as expected and quoted identifiers work too. However, this functionality is not complete yet. Hence it is guarded by the `ksql.any.key.name.enabled` feature flag, which defaults to off. The following big ticket items are remaining: * PARTITION BY a single value column should result in a stream with the key column that matches the value column name. * GROUP BY a single value column should result in a table with the key column that matches the value column name. * JOIN on a single value column should result in a stream/table with the key column that matches the value column name. This additional work will be tracked under the same ticket, e.g. confluentinc#3536
purplefox
approved these changes
Mar 4, 2020
agavra
approved these changes
Mar 4, 2020
| @@ -123,4 +129,19 @@ public KeyField getKeyField() { | |||
| getTimestampColumn() | |||
| ); | |||
| } | |||
|
|
|||
| private void validate() { | |||
ksql-functional-tests/src/test/resources/query-validation-tests/elements.json
Show resolved
Hide resolved
|
Merging to avoid merge hell. @agavra : reach out if you still think we need any changes and I'll pick them up in the next PR. Thanks for the reviews! |
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Partial fix for: #3536
First part of supporting key column names other than
ROWKEY.With this initial pass you can now name your key columns anything you want in your
CREATE TABLEandCREATE STREAMstatements, e.g.CREATE STREAM S (ID INT KEY, NAME STRING) WITH (...);Any GROUP BY, PARTITION BY or JOIN on the key column results any created data source having a key column with a matching name, e.g.
Pull and push queries work as expected and quoted identifiers work too.
However, this functionality is not complete yet. Hence it is guarded by the
ksql.any.key.name.enabledfeature flag, which defaults to off. The following big ticket items are remaining:This additional work will be tracked under the same ticket, e.g. #3536
Reviewing notes:
Documentation changes will be done in a separate PR: #4686 (WIP)
Some changes are just renaming vars/funcs so that they no longer contain 'rowkey' in their names. This may seem unnecessary. However, use of
ROWKEYis deeply ingrained in the code base. I'm having to search for 'rowkey' as part of this work. So these renames are mainly just so that they don't show up in my next search.PR is broken down into the following commits to help with reviewing:
ROWKEYand a new version using a custom key column name. The oldROWKEYversion will be removed once this feature is complete.Testing done
Extensive (R)QTT testing
Reviewer checklist