Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: PPL - json_set function #3271

Open
wants to merge 97 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
97 commits
Select commit Hold shift + click to select a range
70152eb
added implementation
14yapkc1 Jan 3, 2025
76c3995
added doctest, integ-tests, and unit tests
14yapkc1 Jan 6, 2025
ce2c551
addressed PR comments
kenrickyap Jan 6, 2025
ad1bde3
fixed unit tests
kenrickyap Jan 7, 2025
ccf47a2
addressed pr comments
kenrickyap Jan 7, 2025
acc76a0
addressed PR comments
kenrickyap Jan 7, 2025
519c6f2
removed unused dependencies
kenrickyap Jan 7, 2025
2e319fe
linting
kenrickyap Jan 7, 2025
ee0820d
addressed pr comment and rolling back disabled test case
kenrickyap Jan 8, 2025
d44fc5a
Merge branch 'main' into feature/json-valid
kenrickyap Jan 8, 2025
3407d4a
removed disabled import
kenrickyap Jan 9, 2025
7ef6cc9
Update docs/user/ppl/functions/json.rst
kenrickyap Jan 9, 2025
e5e90ac
Update integ-test/src/test/java/org/opensearch/sql/ppl/JsonFunctionIT…
kenrickyap Jan 9, 2025
2187a5a
nit
kenrickyap Jan 9, 2025
5e1e488
Merge branch 'feature/json-valid' of https://github.com/Bit-Quill/ope…
kenrickyap Jan 9, 2025
3512b33
fixed integ test
kenrickyap Jan 9, 2025
9fea606
change text type to keyword
kenrickyap Jan 9, 2025
fbc54bc
addressed PR comments
kenrickyap Jan 10, 2025
31ad2a4
fix doc-test
kenrickyap Jan 11, 2025
2b2a8f3
added null test
kenrickyap Jan 14, 2025
dc96563
Merge branch 'main' into feature/json-valid
acarbonetto Jan 15, 2025
1913bfe
SQL: adding error case unit tests for json_valid
acarbonetto Jan 15, 2025
67d979d
json_valid: null and missing should return false
acarbonetto Jan 15, 2025
aa6b723
PPL: Add json and cast to json functions
acarbonetto Jan 8, 2025
4c99235
PPL: Update json cast for review
acarbonetto Jan 8, 2025
9ccde7f
Fix testes
acarbonetto Jan 9, 2025
4306bf3
spotless
acarbonetto Jan 9, 2025
613137b
Fix tests
acarbonetto Jan 14, 2025
ab28872
SPOTLESS
acarbonetto Jan 14, 2025
3ec16e0
Clean up for merge
acarbonetto Jan 15, 2025
6dbf37b
added implementation
14yapkc1 Jan 3, 2025
b8c6d68
added doctest, integ-tests, and unit tests
14yapkc1 Jan 6, 2025
afb668c
addressed pr comments
kenrickyap Jan 7, 2025
54ef183
addressed PR comments
kenrickyap Jan 7, 2025
d841394
removed unused dependencies
kenrickyap Jan 7, 2025
25fb527
linting
kenrickyap Jan 7, 2025
4a20d08
addressed pr comment and rolling back disabled test case
kenrickyap Jan 8, 2025
fdc4729
removed disabled import
kenrickyap Jan 9, 2025
707a0b9
nit
kenrickyap Jan 9, 2025
4f28211
Update integ-test/src/test/java/org/opensearch/sql/ppl/JsonFunctionIT…
kenrickyap Jan 9, 2025
9ec6335
fixed integ test
kenrickyap Jan 9, 2025
3324e66
SQL: adding error case unit tests for json_valid
acarbonetto Jan 15, 2025
7123c35
json_valid: null and missing should return false
acarbonetto Jan 15, 2025
dbca991
PPL: Add json and cast to json functions
acarbonetto Jan 8, 2025
7df87cb
PPL: Update json cast for review
acarbonetto Jan 8, 2025
cd45fcc
Fix testes
acarbonetto Jan 9, 2025
6f5dc07
spotless
acarbonetto Jan 9, 2025
0aae36e
Fix tests
acarbonetto Jan 14, 2025
b225f28
SPOTLESS
acarbonetto Jan 14, 2025
78af4f8
Clean up for merge
acarbonetto Jan 15, 2025
b84282a
clean up unit tests
acarbonetto Jan 15, 2025
1e23286
Add casting from undefined
acarbonetto Jan 15, 2025
343f5a2
Add cast to scalar from undefined expression
acarbonetto Jan 16, 2025
e8b6df3
Add test for missing/null
acarbonetto Jan 16, 2025
ab9be75
Clean up merge conflicts
acarbonetto Jan 17, 2025
788be9d
Fix jacoco coverage
acarbonetto Jan 17, 2025
a9721bf
Move to Switch by json type
acarbonetto Jan 17, 2025
daa95ff
Merge branch 'main' into feature/acarbo_json_cast_ppl
acarbonetto Jan 20, 2025
018e462
functionality implemented
kenrickyap Jan 20, 2025
c6c6cc1
Remove conflicted files
acarbonetto Jan 21, 2025
a5652ea
Add doctext row
acarbonetto Jan 21, 2025
2cd10a2
added integ-test and doc test
kenrickyap Jan 22, 2025
cd78ddd
fixed integ tests
kenrickyap Jan 22, 2025
afb385f
unit tests
kenrickyap Jan 23, 2025
0e91b2e
Merge branch 'main' into feature/json-extract
kenrickyap Jan 23, 2025
794db8a
finnished unit tests
kenrickyap Jan 23, 2025
0f0b8d4
update doctest
kenrickyap Jan 23, 2025
f030057
addessed comments
kenrickyap Jan 27, 2025
2b08007
added addition edge cases for unit tests
kenrickyap Jan 28, 2025
be52786
Merge branch 'feature/acarbo_json_cast_ppl' into feature/json-extract
kenrickyap Jan 28, 2025
6bd2f40
Merge branch 'feature/acarbo_json_cast_ppl' into feature/json-extract
kenrickyap Jan 28, 2025
0b9e9e4
Merge branch 'feature/json-extract' of https://github.com/Bit-Quill/o…
kenrickyap Jan 28, 2025
6678be4
addressed PR comments
kenrickyap Jan 29, 2025
e57fa21
fix code coverage
14yapkc1 Jan 30, 2025
112be65
Update core/src/test/java/org/opensearch/sql/expression/json/JsonFunc…
kenrickyap Jan 30, 2025
306ac97
address comments
kenrickyap Jan 30, 2025
75e9cc3
fix build error
kenrickyap Jan 30, 2025
77827bb
Merge branch 'main' into feature/json-extract
kenrickyap Jan 31, 2025
80f44e2
add header
kenrickyap Jan 31, 2025
0d1cc28
addressing PR comments
kenrickyap Feb 12, 2025
b6ae5ba
added multi path use case
kenrickyap Feb 12, 2025
adde88d
Merge branch 'main' into feature/json-extract
kenrickyap Feb 12, 2025
aa8b81e
linting
kenrickyap Feb 12, 2025
ec6ff5e
fixing doc tests
kenrickyap Feb 12, 2025
95e996b
Update core/src/main/java/org/opensearch/sql/utils/JsonUtils.java
kenrickyap Feb 13, 2025
527415e
Update jsonSet
andy-k-improving Jan 20, 2025
aefa9cb
Provide primitive support
andy-k-improving Jan 21, 2025
3fabe17
Manual test
andy-k-improving Jan 27, 2025
46183d9
IT tests
andy-k-improving Jan 27, 2025
afd7a31
IT test
andy-k-improving Jan 28, 2025
6daca82
Code review
andy-k-improving Jan 31, 2025
b1cb3aa
Fix doc test
andy-k-improving Jan 31, 2025
741322a
Update supported types
andy-k-improving Feb 4, 2025
8d26f6b
Code comments
andy-k-improving Feb 5, 2025
c5b4da6
Update the test-case
andy-k-improving Feb 7, 2025
b44d1a1
Rebase
andy-k-improving Feb 14, 2025
85c23f9
Code style
andy-k-improving Feb 14, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
addressed PR comments
Signed-off-by: Kenrick Yap <kenrick.yap@improving.com>
  • Loading branch information
kenrickyap committed Jan 7, 2025
commit acc76a027792e7f66d6a6dd2c3c8771080fb01ff
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
import org.opensearch.sql.expression.function.BuiltinFunctionName;
import org.opensearch.sql.expression.function.BuiltinFunctionRepository;
import org.opensearch.sql.expression.function.DefaultFunctionResolver;
import org.opensearch.sql.utils.JsonUtils;

import static org.opensearch.sql.data.type.ExprCoreType.BOOLEAN;
import static org.opensearch.sql.data.type.ExprCoreType.STRING;
Expand All @@ -30,22 +31,6 @@ public void register(BuiltinFunctionRepository repository) {
private DefaultFunctionResolver jsonValid() {
return define(
BuiltinFunctionName.JSON_VALID.getName(),
impl(nullMissingHandling(JsonFunctions::isValidJson), BOOLEAN, STRING));
}

/**
* Checks if given JSON string can be parsed as valid JSON.
*
* @param jsonExprValue JSON string (e.g. "198.51.100.14" or "2001:0db8::ff00:42:8329").
* @return true if the string can be parsed as valid JSON, else false.
*/
private ExprValue isValidJson(ExprValue jsonExprValue) {
ObjectMapper objectMapper = new ObjectMapper();
try {
objectMapper.readTree(jsonExprValue.stringValue());
return ExprValueUtils.LITERAL_TRUE;
} catch (JsonProcessingException e) {
return ExprValueUtils.LITERAL_FALSE;
}
impl(nullMissingHandling(JsonUtils::isValidJson), BOOLEAN, STRING));
}
}
26 changes: 26 additions & 0 deletions core/src/main/java/org/opensearch/sql/utils/JsonUtils.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
package org.opensearch.sql.utils;

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.experimental.UtilityClass;
import org.opensearch.sql.data.model.ExprValue;
import org.opensearch.sql.data.model.ExprValueUtils;

@UtilityClass
public class JsonUtils {
/**
* Checks if given JSON string can be parsed as valid JSON.
*
* @param jsonExprValue JSON string (e.g. "198.51.100.14" or "2001:0db8::ff00:42:8329").
* @return true if the string can be parsed as valid JSON, else false.
*/
public static ExprValue isValidJson(ExprValue jsonExprValue) {
ObjectMapper objectMapper = new ObjectMapper();
try {
objectMapper.readTree(jsonExprValue.stringValue());
return ExprValueUtils.LITERAL_TRUE;
} catch (JsonProcessingException e) {
return ExprValueUtils.LITERAL_FALSE;
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
import java.util.List;
import java.util.stream.Stream;
import lombok.AllArgsConstructor;
import org.junit.jupiter.api.Disabled;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.Arguments;
Expand Down Expand Up @@ -1228,9 +1229,8 @@ public void testWeekFormats(
expectedInteger);
}

// subtracting 1 as a temporary fix for year 2024.
// Issue: https://github.com/opensearch-project/sql/issues/2477
@Test
@Disabled("Test is disabled because of issue https://github.com/opensearch-project/sql/issues/2477")
public void testWeekOfYearWithTimeType() {
assertAll(
() ->
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@

import java.time.LocalDate;
import java.util.stream.Stream;

import org.junit.jupiter.api.Disabled;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.Arguments;
Expand Down Expand Up @@ -91,6 +93,7 @@ private void datePartWithTimeArgQuery(String part, String time, long expected) {
}

@Test
@Disabled("Test is disabled because of issue https://github.com/opensearch-project/sql/issues/2477")
public void testExtractDatePartWithTimeType() {
datePartWithTimeArgQuery(
"DAY", timeInput, LocalDate.now(functionProperties.getQueryStartClock()).getDayOfMonth());
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@

import java.time.LocalDate;
import java.util.stream.Stream;

import org.junit.jupiter.api.Disabled;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.Arguments;
Expand Down Expand Up @@ -97,9 +99,8 @@ public void testYearweekWithoutMode() {
assertEquals(eval(expression), eval(expressionWithoutMode));
}

// subtracting 1 as a temporary fix for year 2024.
// Issue: https://github.com/opensearch-project/sql/issues/2477
@Test
@Disabled("Test is disabled because of issue https://github.com/opensearch-project/sql/issues/2477")
public void testYearweekWithTimeType() {
int week = LocalDate.now(functionProperties.getQueryStartClock()).get(ALIGNED_WEEK_OF_YEAR) - 1;
int year = LocalDate.now(functionProperties.getQueryStartClock()).getYear();
Expand Down