Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate SQL/PPL to OpenSearch namespace #16

Merged
merged 8 commits into from
Apr 23, 2021
Merged

Migrate SQL/PPL to OpenSearch namespace #16

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
4fe84ec
Move SQL settings to opensearch namespace
dai-chen Apr 21, 2021
3f819f2
Move PPL settings to opensearch namespace
dai-chen Apr 21, 2021
15668d2
Merge branch 'develop' into sql-ppl-settings-migration-for-opensearch
dai-chen Apr 22, 2021
ac58511
Migrate SQL settings in legacy API IT
dai-chen Apr 22, 2021
5e85a4b
Migrate expression language name to opensearch
dai-chen Apr 22, 2021
599832a
Migrate ANLTR grammar name to opensearch
dai-chen Apr 22, 2021
07f120d
Rename IT base class
dai-chen Apr 22, 2021
2856cba
Merge branch 'develop' into sql-ppl-settings-migration-for-opensearch
dai-chen Apr 23, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions common/src/main/java/com/amazon/opendistroforelasticsearch/sql/common/setting/Settings.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,14 +40,14 @@ public enum Key {
/**
* PPL Setting.
*/
PPL_QUERY_MEMORY_LIMIT("opendistro.ppl.query.memory_limit"),
PPL_QUERY_MEMORY_LIMIT("opensearch.ppl.query.memory_limit"),

PPL_ENABLED("opendistro.ppl.enabled"),
PPL_ENABLED("opensearch.ppl.enabled"),

/**
* Common Setting for SQL and PPL.
*/
QUERY_SIZE_LIMIT("opendistro.query.size_limit");
QUERY_SIZE_LIMIT("opensearch.query.size_limit");

@Getter
private final String keyValue;
Expand Down
4 changes: 2 additions & 2 deletions docs/dev/Architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ The OpenSearch SQL (OD-SQL) project is developed based on NLPChina project (http

In the high level, the OD-SQL Engine could be divided into four major sub-module.

* *Parser*: Currently, there are two Lex&Parser coexists. The Druid Lex&Parser is the original one from NLPChina. The input AST of Core Engine is from the Druid Lex&Parser. The [ANTLR](https://github.com/opensearch-project/sql/blob/master/src/main/antlr/OpenDistroSqlParser.g4) Lex&Parser is added by us to customized the verification and exception handling.
* *Parser*: Currently, there are two Lex&Parser coexists. The Druid Lex&Parser is the original one from NLPChina. The input AST of Core Engine is from the Druid Lex&Parser. The [ANTLR](https://github.com/opensearch-project/sql/blob/main/legacy/src/main/antlr/OpenSearchLegacySqlLexer.g4) Lex&Parser is added by us to customized the verification and exception handling.
* *Analyzer*: The analyzer module take the output from ANTLR Lex&Parser then perform syntax and semantic analyze.
* *Core Engine*: The QueryAction take the output from Druid Lex&Parser and translate to the OpenSearch DSL if possible. This is an NLPChina original module. The QueryPlanner Builder is added by us to support the JOIN and Post-processing logic. The QueryPlanner will take the take the output from Druid Lex&Parser and build the PhysicalPlan
* *Execution*: The execution module execute QueryAction or QueryPlanner and return the response to the client. Different from the Frontend, Analyzer and Core Engine which running on the Transport Thread and can’t do any blocking operation. The Execution module running on the client threadpool and can perform the blocking operation.
Expand All @@ -30,7 +30,7 @@ The following diagram take a sample query and explain how the query flow within

![Architecture Journey](img/architecture-journey.png)

1. The ANTRL parser based on grammar file (https://github.com/opensearch-project/sql/blob/master/src/main/antlr/OpenDistroSqlParser.g4) to auto generate the AST.
1. The ANTRL parser based on grammar file (https://github.com/opensearch-project/sql/blob/main/legacy/src/main/antlr/OpenSearchLegacySqlParser.g4) to auto generate the AST.
2. The Syntax and Semantic Analyzer will walk through the AST and verify whether the query is follow the grammar and supported by the OD-SQL. e.g. *SELECT * FROM semantics WHERE LOG(age, city) = 1, *will throw exception with message* Function [LOG] cannot work with [INTEGER, KEYWORD]. *and sample usage message* Usage: LOG(NUMBER T) → DOUBLE.
3. The Druid Lex&Parser takes the input query and generate the druid AST which is different from the AST generated by the ANTRL. This module is the open source library (https://github.com/alibaba/druid) used by NLPChina originally.
4. The QueryPlanner Builder take the AST as input and generate the LogicalPlan from it. Then it optimize the LogicalPlan to PhysicalPlan.(In current implementation, only rule-based model is implemented). The major part of PhysicalPlan generation use NLPChina’s original logic to translate the SQL expression in AST to OpenSearch DSL.
Expand Down
18 changes: 9 additions & 9 deletions docs/dev/Pagination.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -439,7 +439,7 @@ When OpenSearch bootstraps, SQL plugin will register a few settings in OpenSearc
Most of the settings are able to change dynamically so you can control the behavior of SQL plugin without need to bounce your cluster.
For cursors we will be exposing the following settings:

#### opendistro.sql.cursor.enabled
#### opensearch.sql.cursor.enabled

You can disable cursor for all SQL queries which support pagination.

Expand All @@ -453,7 +453,7 @@ Example:
```
>> curl -H 'Content-Type: application/json' -X PUT localhost:9200/_cluster/settings -d '{
"transient" : {
"opendistro.sql.cursor.enabled" : "false"
"opensearch.sql.cursor.enabled" : "false"
}
}'
```
Expand All @@ -465,7 +465,7 @@ Response:
"acknowledged" : true,
"persistent" : { },
"transient" : {
"opendistro" : {
"opensearch" : {
"sql" : {
"cursor" : {
"enabled" : "false"
Expand All @@ -477,7 +477,7 @@ Response:

```

#### opendistro.sql.cursor.fetch_size
#### opensearch.sql.cursor.fetch_size

This setting controls the default page size for all cursor requests.

Expand All @@ -493,7 +493,7 @@ Example:
```
>> curl -H 'Content-Type: application/json' -X PUT localhost:9200/_cluster/settings -d '{
"persistent" : {
"opendistro.sql.cursor.fetch_size" : "100"
"opensearch.sql.cursor.fetch_size" : "100"
}
}'
```
Expand All @@ -505,7 +505,7 @@ Response:
"acknowledged" : true,
"transient" : { },
"persistent" : {
"opendistro" : {
"opensearch" : {
"sql" : {
"cursor" : {
"fetch_size" : "100"
Expand All @@ -517,7 +517,7 @@ Response:

```

#### opendistro.sql.cursor.keep_alive
#### opensearch.sql.cursor.keep_alive

This setting controls the how long the cursor context is open for all cursor requests.
You can five the time in human readable time format like `5h` (5 hours) or `20s` (20 seconds) etc.
Expand All @@ -532,7 +532,7 @@ Example:
```
>> curl -H 'Content-Type: application/json' -X PUT localhost:9200/_cluster/settings -d '{
"transient" : {
"opendistro.sql.cursor.keep_alive" : "200s"
"opensearch.sql.cursor.keep_alive" : "200s"
}
}'
```
Expand All @@ -544,7 +544,7 @@ Response:
"acknowledged" : true,
"persistent" : { },
"transient" : {
"opendistro" : {
"opensearch" : {
"sql" : {
"cursor" : {
"keep_alive" : "200s"
Expand Down
22 changes: 11 additions & 11 deletions docs/experiment/ppl/admin/settings.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Introduction

When OpenSearch bootstraps, PPL plugin will register a few settings in OpenSearch cluster settings. Most of the settings are able to change dynamically so you can control the behavior of PPL plugin without need to bounce your cluster.

opendistro.ppl.enabled
opensearch.ppl.enabled
======================

Description
Expand All @@ -39,12 +39,12 @@ PPL query::

sh$ curl -sS -H 'Content-Type: application/json' \
... -X PUT localhost:9200/_cluster/settings \
... -d '{"transient" : {"opendistro.ppl.enabled" : "false"}}'
... -d '{"transient" : {"opensearch.ppl.enabled" : "false"}}'
{
"acknowledged": true,
"persistent": {},
"transient": {
"opendistro": {
"opensearch": {
"ppl": {
"enabled": "false"
}
Expand All @@ -64,7 +64,7 @@ PPL query::
{
"error": {
"reason": "Invalid Query",
"details": "Either opendistro.ppl.enabled or rest.action.multi.allow_explicit_index setting is false",
"details": "Either opensearch.ppl.enabled or rest.action.multi.allow_explicit_index setting is false",
"type": "IllegalAccessException"
},
"status": 400
Expand All @@ -79,14 +79,14 @@ PPL query::

sh$ curl -sS -H 'Content-Type: application/json' \
... -X PUT localhost:9200/_cluster/settings \
... -d '{"transient" : {"opendistro.ppl.enabled" : null}}'
... -d '{"transient" : {"opensearch.ppl.enabled" : null}}'
{
"acknowledged": true,
"persistent": {},
"transient": {}
}

opendistro.ppl.query.memory_limit
opensearch.ppl.query.memory_limit
=================================

Description
Expand All @@ -101,11 +101,11 @@ PPL query::

sh$ curl -sS -H 'Content-Type: application/json' \
... -X PUT localhost:9200/_cluster/settings \
... -d '{"persistent" : {"opendistro.ppl.query.memory_limit" : "80%"}}'
... -d '{"persistent" : {"opensearch.ppl.query.memory_limit" : "80%"}}'
{
"acknowledged": true,
"persistent": {
"opendistro": {
"opensearch": {
"ppl": {
"query": {
"memory_limit": "80%"
Expand Down Expand Up @@ -133,11 +133,11 @@ Change the size_limit to 1000::

sh$ curl -sS -H 'Content-Type: application/json' \
... -X PUT localhost:9200/_cluster/settings \
... -d '{"persistent" : {"opendistro.query.size_limit" : "1000"}}'
... -d '{"persistent" : {"opensearch.query.size_limit" : "1000"}}'
{
"acknowledged": true,
"persistent": {
"opendistro": {
"opensearch": {
"query": {
"size_limit": "1000"
}
Expand All @@ -150,7 +150,7 @@ Rollback to default value::

sh$ curl -sS -H 'Content-Type: application/json' \
... -X PUT localhost:9200/_cluster/settings \
... -d '{"persistent" : {"opendistro.query.size_limit" : null}}'
... -d '{"persistent" : {"opensearch.query.size_limit" : null}}'
{
"acknowledged": true,
"persistent": {},
Expand Down
Loading