Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

P1 antlr grammar - update lambda & join #874

Merged
merged 15 commits into from
Nov 5, 2024

Conversation

YANG-DB
Copy link
Member

@YANG-DB YANG-DB commented Nov 5, 2024

Description

update ANTLR with lambda & join grammar changes

Check List

  • Updated documentation (docs/ppl-lang/README.md)
  • Implemented unit tests
  • Implemented tests for combination with other commands
  • New added source code should include a copyright header
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

YANG-DB and others added 15 commits October 31, 2024 11:52
Signed-off-by: YANGDB <yang.db.dev@gmail.com>
Signed-off-by: YANGDB <yang.db.dev@gmail.com>
Signed-off-by: YANGDB <yang.db.dev@gmail.com>
Signed-off-by: YANGDB <yang.db.dev@gmail.com>
* The flatten command implemented

Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com>

* The flatten command integration tests were extended with additional checks for logical plans.

Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com>

* flatten, added more tests related to plan translation and integration tests

Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com>

* Flatten command added to command names list.

Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com>

---------

Signed-off-by: Lukasz Soszynski <lukasz.soszynski@eliatra.com>
* add sourceTables to MV index metadata properties

Signed-off-by: Sean Kao <seankao@amazon.com>

* parse source tables from mv query

Signed-off-by: Sean Kao <seankao@amazon.com>

* test cases for parse source tables from mv query

Signed-off-by: Sean Kao <seankao@amazon.com>

* use constant for metadata cache version

Signed-off-by: Sean Kao <seankao@amazon.com>

* write source tables to metadata cache

Signed-off-by: Sean Kao <seankao@amazon.com>

* address comment

Signed-off-by: Sean Kao <seankao@amazon.com>

* generate source tables for old mv without new prop

Signed-off-by: Sean Kao <seankao@amazon.com>

* syntax fix

Signed-off-by: Sean Kao <seankao@amazon.com>

---------

Signed-off-by: Sean Kao <seankao@amazon.com>
…-project#850)

* Fallback to internal scheduler when index creation failed

Signed-off-by: Louis Chu <clingzhi@amazon.com>

* Fix IT

Signed-off-by: Louis Chu <clingzhi@amazon.com>

* Fix IOException

Signed-off-by: Louis Chu <clingzhi@amazon.com>

---------

Signed-off-by: Louis Chu <clingzhi@amazon.com>
* WIP trendline command

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* wip

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* trendline supports sorting

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* run scalafmtAll

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* return null when there are too few data points

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* sbt scalafmtAll

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* Remove WMA references

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* trendline - sortByField as Optional<Field>

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* introduce TrendlineStrategy

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* keywordsCanBeId -> replace SMA with trendlineType

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* handle trendline alias as qualifiedName instead of fieldExpression

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>

* Add docs

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Make alias optional

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Adapt tests for optional alias

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Adden logical plan unittests

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Add missing license headers

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Fix docs

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* numberOfDataPoints must be 1 or greater

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Rename TrendlineStrategy to  TrendlineCatalystUtils

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Validate TrendlineType early and pass around enum type

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Add trendline chaining test

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Fix compile errors

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Fix imports

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

* Fix imports

Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>

---------

Signed-off-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>
Signed-off-by: Hendrik Saly <hendrik.saly@eliatra.com>
Co-authored-by: Kacper Trochimiak <kacper.trochimiak@eliatra.com>
Signed-off-by: YANGDB <yang.db.dev@gmail.com>
# Conflicts:
#	ppl-spark-integration/src/main/antlr4/OpenSearchPPLLexer.g4
#	ppl-spark-integration/src/main/antlr4/OpenSearchPPLParser.g4
Signed-off-by: YANGDB <yang.db.dev@gmail.com>
…t#865)

* update logical tests and docs

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* update scala fmt style

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* fix type error

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

---------

Signed-off-by: YANGDB <yang.db.dev@gmail.com>
* json function enhancement

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Add JavaToScalaTransformer

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Apply scalafmtAll

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Address comments

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Add IT and change to use the same function name as spark

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Address comments

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Add document and separate lambda functions from json functions

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Add lambda functions transform and reduce

Signed-off-by: Heng Qian <qianheng@amazon.com>

* polish lambda function document

Signed-off-by: Heng Qian <qianheng@amazon.com>

* polish lambda function document

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Minor fix

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Minor change to polish the documents

Signed-off-by: Heng Qian <qianheng@amazon.com>

---------

Signed-off-by: Heng Qian <qianheng@amazon.com>
# Conflicts:
#	ppl-spark-integration/src/main/antlr4/OpenSearchPPLParser.g4
…grammar

# Conflicts:
#	docs/ppl-lang/PPL-Example-Commands.md
#	ppl-spark-integration/src/main/antlr4/OpenSearchPPLParser.g4
#	ppl-spark-integration/src/main/java/org/opensearch/sql/ppl/CatalystExpressionVisitor.java
//JSON_ARRAY_MAP: 'JSON_ARRAY_MAP';
//JSON_ARRAY_REDUCE: 'JSON_ARRAY_REDUCE';

// COLLECTION FUNCTIONS
ARRAY: 'ARRAY';

// LAMBDA FUNCTIONS
//EXISTS: 'EXISTS';
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this commented out?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not supported yet by the lambda command

Copy link
Member

@LantaoJin LantaoJin Nov 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commented EXISTS out because the it is an existed keyword. Just add it here as a placeholder. I think this line can be deleted to reduce confusion.

@YANG-DB YANG-DB merged commit f8c7c80 into opensearch-project:p1-antlr-grammar Nov 5, 2024
3 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Lang:PPL Pipe Processing Language support
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants