Adding CONVERT_TZ and DATETIME functions to SQL and PPL (#848)

* Added `CONVERT_TZ` to the PPL lexer/parser and the SQL lexer/parser.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added `convert_tz` to the BuiltinFunctionName.java.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added convert_tz to DateTimeFunction.java register, private FunctionResolver convert_tz and ExprValue exprConvert_TZ. It implements the functionality for converting between time zones.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added IT PPL and SQL tests for various conditions including time zones that are outside the existing range (consistent with MySQL standard). Added implementation for convert_tz consistent with MySQL implementation.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added DATETIME to OpenSearchPPLParser.g4 and OpenSearchSQLParser.g4.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Rebase merge conflict resolution.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added ppl doctest for convert_tz

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Completed implementation for datetime and convert_Tz.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Removed SQL test from PPL IT test

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Removed redundant convert to string.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Fixed doctests and reverted changes to adddate function in DateTimeFunctionTest.java.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Fixed doctest.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Fixed doctest.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added additional integration tests for PPL and SQL tests for convert_tz.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added null for timezones outside of basic range for DATETIME.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added test cases for null with the datetime function. Made DateTime function call conevrt_tz function.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Seperated out test from DateTimeFunction.java.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Updated tests. Fixed exception to be less general.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Removed changes.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Fixed rel timezone issue.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added support for non valid datetime to return null for convert_tz.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Made exception more verbose.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Removed unneeded format changes in DateTimeFunction.java.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added more doctests.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Removed formatting changes.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Reverting sql/ppl DateTimeFunctionsIT.java.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Reverted changes to DateTimeFunctionTest.java

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added more information about invalid date for convert_tz

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Converted "from Field" and "To Field" to use "Fieldn" where n is the field number.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added date validation. Added test cases in IT to cover cases. Added test in ConvertTZTest.java.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Fixed date validation function. Broke up some unit tests.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Fixed formatting.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added DateTime tests, broke up functions.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added space in DateTimeTest.java.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Tidied up code and tests.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Fixed local date time rst test.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Removed nested try/catch exceptions.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* removed exprConvertTZ function call from within try catch statement.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Reverted change from parse localdate

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Removed extra casting around fromTz variable.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Updated wording for functions.rst

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Updated wording for datetime.rst to describe null for conert_tz

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added more test cases for functions.rst

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Fixed doctests.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Removed extra import.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Renamed isValidTimeZone function to isValidMySqlTimeZoneId.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Has ExprDatetimeValue doing work for exprConvertTZ call.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Deleted fromTZ

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Moved fixed variables to top of class.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added Null to support of exprDateTime.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added missing expr functions for makedate/time

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* cleaning up after rebase merge.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Rebase merge conflict resolution.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Fixed missing DATETIME in SQL Parser.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Fixed IT test.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Addressed PR comments

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Added missing variables after rebase

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Fixed checkstyle errors after rebase.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Moved formatter for date time over.

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Changed function resolved to default

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

* Removed unneeded code

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

Signed-off-by: MitchellGale-BitQuill <mitchellg@bitquilltech.com>

core/src/main/java/org/opensearch/sql/data/model/ExprDatetimeValue.java

@@ -33,7 +33,7 @@ public class ExprDatetimeValue extends AbstractExprValue {
 
   static {
     FORMATTER_VARIABLE_NANOS = new DateTimeFormatterBuilder()
-        .appendPattern("yyyy-MM-dd HH:mm:ss")
+        .appendPattern("uuuu-MM-dd HH:mm:ss[xxx]")
         .appendFraction(
             ChronoField.NANO_OF_SECOND,
             MIN_FRACTION_SECONDS,

core/src/main/java/org/opensearch/sql/expression/DSL.java

@@ -271,10 +271,18 @@ public FunctionExpression adddate(Expression... expressions) {
     return function(BuiltinFunctionName.ADDDATE, expressions);
   }
 
+  public FunctionExpression convert_tz(Expression... expressions) {
+    return function(BuiltinFunctionName.CONVERT_TZ, expressions);
+  }
+
   public FunctionExpression date(Expression... expressions) {
     return function(BuiltinFunctionName.DATE, expressions);
   }
 
+  public FunctionExpression datetime(Expression... expressions) {
+    return function(BuiltinFunctionName.DATETIME, expressions);
+  }
+
   public FunctionExpression date_add(Expression... expressions) {
     return function(BuiltinFunctionName.DATE_ADD, expressions);
   }

core/src/main/java/org/opensearch/sql/expression/datetime/DateTimeFunction.java

@@ -22,20 +22,24 @@
 import static org.opensearch.sql.utils.DateTimeFormatters.DATE_FORMATTER_SHORT_YEAR;
 import static org.opensearch.sql.utils.DateTimeFormatters.DATE_TIME_FORMATTER_LONG_YEAR;
 import static org.opensearch.sql.utils.DateTimeFormatters.DATE_TIME_FORMATTER_SHORT_YEAR;
+import static org.opensearch.sql.utils.DateTimeFormatters.DATE_TIME_FORMATTER_STRICT_WITH_TZ;
 
 import java.math.BigDecimal;
 import java.math.RoundingMode;
 import java.text.DecimalFormat;
+import java.time.DateTimeException;
 import java.time.Instant;
 import java.time.LocalDate;
 import java.time.LocalDateTime;
 import java.time.LocalTime;
 import java.time.ZoneId;
 import java.time.ZoneOffset;
+import java.time.ZonedDateTime;
 import java.time.format.DateTimeFormatter;
 import java.time.format.DateTimeParseException;
 import java.time.format.TextStyle;
 import java.util.Locale;
+import java.util.TimeZone;
 import java.util.concurrent.TimeUnit;
 import javax.annotation.Nullable;
 import lombok.experimental.UtilityClass;
@@ -50,11 +54,13 @@
 import org.opensearch.sql.data.model.ExprTimestampValue;
 import org.opensearch.sql.data.model.ExprValue;
 import org.opensearch.sql.data.type.ExprCoreType;
+import org.opensearch.sql.exception.ExpressionEvaluationException;
 import org.opensearch.sql.expression.function.BuiltinFunctionName;
 import org.opensearch.sql.expression.function.BuiltinFunctionRepository;
 import org.opensearch.sql.expression.function.DefaultFunctionResolver;
 import org.opensearch.sql.expression.function.FunctionName;
 import org.opensearch.sql.expression.function.FunctionResolver;
+import org.opensearch.sql.utils.DateTimeUtils;
 
 /**
  * The definition of date and time functions.
@@ -63,7 +69,6 @@
  */
 @UtilityClass
 public class DateTimeFunction {
-
   // The number of days from year zero to year 1970.
   private static final Long DAYS_0000_TO_1970 = (146097 * 5L) - (30L * 365L + 7L);
 
@@ -78,7 +83,9 @@ public class DateTimeFunction {
    */
   public void register(BuiltinFunctionRepository repository) {
     repository.register(adddate());
+    repository.register(convert_tz());
     repository.register(date());
+    repository.register(datetime());
     repository.register(date_add());
     repository.register(date_sub());
     repository.register(day());
@@ -214,6 +221,21 @@ private DefaultFunctionResolver adddate() {
     return add_date(BuiltinFunctionName.ADDDATE.getName());
   }
 
+  /**
+   * Converts date/time from a specified timezone to another specified timezone.
+   * The supported signatures:
+   * (DATETIME, STRING, STRING) -> DATETIME
+   * (STRING, STRING, STRING) -> DATETIME
+   */
+  private DefaultFunctionResolver convert_tz() {
+    return define(BuiltinFunctionName.CONVERT_TZ.getName(),
+        impl(nullMissingHandling(DateTimeFunction::exprConvertTZ),
+            DATETIME, DATETIME, STRING, STRING),
+        impl(nullMissingHandling(DateTimeFunction::exprConvertTZ),
+            DATETIME, STRING, STRING, STRING)
+    );
+  }
+
   /**
    * Extracts the date part of a date and time value.
    * Also to construct a date type. The supported signatures:
@@ -227,6 +249,21 @@ private DefaultFunctionResolver date() {
         impl(nullMissingHandling(DateTimeFunction::exprDate), DATE, TIMESTAMP));
   }
 
+  /**
+   * Specify a datetime with time zone field and a time zone to convert to.
+   * Returns a local date time.
+   * (STRING, STRING) -> DATETIME
+   * (STRING) -> DATETIME
+   */
+  private FunctionResolver datetime() {
+    return define(BuiltinFunctionName.DATETIME.getName(),
+        impl(nullMissingHandling(DateTimeFunction::exprDateTime),
+            DATETIME, STRING, STRING),
+        impl(nullMissingHandling(DateTimeFunction::exprDateTimeNoTimezone),
+            DATETIME, STRING)
+    );
+  }
+
   private DefaultFunctionResolver date_add() {
     return add_date(BuiltinFunctionName.DATE_ADD.getName());
   }
@@ -570,6 +607,42 @@ private ExprValue exprAddDateDays(ExprValue date, ExprValue days) {
         : exprValue);
   }
 
+  /**
+   * CONVERT_TZ function implementation for ExprValue.
+   * Returns null for time zones outside of +13:00 and -12:00.
+   *
+   * @param startingDateTime ExprValue of DateTime that is being converted from
+   * @param fromTz           ExprValue of time zone, representing the time to convert from.
+   * @param toTz             ExprValue of time zone, representing the time to convert to.
+   * @return DateTime that has been converted to the to_tz timezone.
+   */
+  private ExprValue exprConvertTZ(ExprValue startingDateTime, ExprValue fromTz, ExprValue toTz) {
+    if (startingDateTime.type() == ExprCoreType.STRING) {
+      startingDateTime = exprDateTimeNoTimezone(startingDateTime);
+    }
+    try {
+      ZoneId convertedFromTz = ZoneId.of(fromTz.stringValue());
+      ZoneId convertedToTz = ZoneId.of(toTz.stringValue());
+
+      // isValidMySqlTimeZoneId checks if the timezone is within the range accepted by
+      // MySQL standard.
+      if (!DateTimeUtils.isValidMySqlTimeZoneId(convertedFromTz)
+          || !DateTimeUtils.isValidMySqlTimeZoneId(convertedToTz)) {
+        return ExprNullValue.of();
+      }
+      ZonedDateTime zonedDateTime =
+          startingDateTime.datetimeValue().atZone(convertedFromTz);
+      return new ExprDatetimeValue(
+          zonedDateTime.withZoneSameInstant(convertedToTz).toLocalDateTime());
+
+
+      // Catches exception for invalid timezones.
+      // ex. "+0:00" is an invalid timezone and would result in this exception being thrown.
+    } catch (ExpressionEvaluationException | DateTimeException e) {
+      return ExprNullValue.of();
+    }
+  }
+
   /**
    * Date implementation for ExprValue.
    *
@@ -584,6 +657,62 @@ private ExprValue exprDate(ExprValue exprValue) {
     }
   }
 
+  /**
+   * DateTime implementation for ExprValue.
+   *
+   * @param dateTime ExprValue of String type.
+   * @param timeZone ExprValue of String type (or null).
+   * @return ExprValue of date type.
+   */
+  private ExprValue exprDateTime(ExprValue dateTime, ExprValue timeZone) {
+    String defaultTimeZone = TimeZone.getDefault().getID();
+
+
+    try {
+      LocalDateTime ldtFormatted =
+          LocalDateTime.parse(dateTime.stringValue(), DATE_TIME_FORMATTER_STRICT_WITH_TZ);
+      if (timeZone.isNull()) {
+        return new ExprDatetimeValue(ldtFormatted);
+      }
+
+      // Used if datetime field is invalid format.
+    } catch (DateTimeParseException e) {
+      return ExprNullValue.of();
+    }
+
+    ExprValue convertTZResult;
+    ExprDatetimeValue ldt;
+    String toTz;
+
+    try {
+      ZonedDateTime zdtWithZoneOffset =
+          ZonedDateTime.parse(dateTime.stringValue(), DATE_TIME_FORMATTER_STRICT_WITH_TZ);
+      ZoneId fromTZ = zdtWithZoneOffset.getZone();
+
+      ldt = new ExprDatetimeValue(zdtWithZoneOffset.toLocalDateTime());
+      toTz = String.valueOf(fromTZ);
+    } catch (DateTimeParseException e) {
+      ldt = new ExprDatetimeValue(dateTime.stringValue());
+      toTz = defaultTimeZone;
+    }
+    convertTZResult = exprConvertTZ(
+        ldt,
+        new ExprStringValue(toTz),
+        timeZone);
+
+    return convertTZResult;
+  }
+
+  /**
+   * DateTime implementation for ExprValue without a timezone to convert to.
+   *
+   * @param dateTime ExprValue of String type.
+   * @return ExprValue of date type.
+   */
+  private ExprValue exprDateTimeNoTimezone(ExprValue dateTime) {
+    return exprDateTime(dateTime, ExprNullValue.of());
+  }
+
   /**
    * Name of the Weekday implementation for ExprValue.
    *

core/src/main/java/org/opensearch/sql/expression/function/BuiltinFunctionName.java

@@ -58,7 +58,9 @@ public enum BuiltinFunctionName {
    * Date and Time Functions.
    */
   ADDDATE(FunctionName.of("adddate")),
+  CONVERT_TZ(FunctionName.of("convert_tz")),
   DATE(FunctionName.of("date")),
+  DATETIME(FunctionName.of("datetime")),
   DATE_ADD(FunctionName.of("date_add")),
   DATE_SUB(FunctionName.of("date_sub")),
   DAY(FunctionName.of("day")),

core/src/main/java/org/opensearch/sql/utils/DateTimeFormatters.java

@@ -162,4 +162,10 @@ public class DateTimeFormatters {
           .appendPattern("MMddHHmmss")
           .toFormatter()
           .withResolverStyle(ResolverStyle.STRICT);
+
+  public static final DateTimeFormatter DATE_TIME_FORMATTER_STRICT_WITH_TZ =
+      new DateTimeFormatterBuilder()
+          .appendPattern("uuuu-MM-dd HH:mm:ss[xxx]")
+          .toFormatter()
+          .withResolverStyle(ResolverStyle.STRICT);
 }

core/src/main/java/org/opensearch/sql/utils/DateTimeUtils.java

@@ -6,6 +6,7 @@
 package org.opensearch.sql.utils;
 
 import java.time.Instant;
+import java.time.LocalDateTime;
 import java.time.ZoneId;
 import java.time.ZonedDateTime;
 import lombok.experimental.UtilityClass;
@@ -84,4 +85,34 @@ public static long roundYear(long utcMillis, int interval) {
     return initDateTime.plusYears(yearToAdd).toInstant().toEpochMilli();
   }
 
+
+  /**
+   * isValidMySqlTimeZoneId for timezones which match timezone the range set by MySQL.
+   *
+   * @param zone ZoneId of ZoneId type.
+   * @return Boolean.
+   */
+  public Boolean isValidMySqlTimeZoneId(ZoneId zone) {
+    String timeZoneMax = "+14:00";
+    String timeZoneMin = "-13:59";
+    String timeZoneZero = "+00:00";
+
+    ZoneId maxTz = ZoneId.of(timeZoneMax);
+    ZoneId minTz = ZoneId.of(timeZoneMin);
+    ZoneId defaultTz = ZoneId.of(timeZoneZero);
+
+    ZonedDateTime defaultDateTime = LocalDateTime.of(2000, 1, 2, 12, 0).atZone(defaultTz);
+
+    ZonedDateTime maxTzValidator =
+        defaultDateTime.withZoneSameInstant(maxTz).withZoneSameLocal(defaultTz);
+    ZonedDateTime minTzValidator =
+        defaultDateTime.withZoneSameInstant(minTz).withZoneSameLocal(defaultTz);
+    ZonedDateTime passedTzValidator =
+        defaultDateTime.withZoneSameInstant(zone).withZoneSameLocal(defaultTz);
+
+    return (passedTzValidator.isBefore(maxTzValidator)
+        || passedTzValidator.isEqual(maxTzValidator))
+        && (passedTzValidator.isAfter(minTzValidator)
+        || passedTzValidator.isEqual(minTzValidator));
+  }
 }