Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-23264][SQL] Make INTERVAL keyword optional in INTERVAL clauses when ANSI mode enabled #20433

Closed
wants to merge 3 commits into from
Closed

[SPARK-23264][SQL] Make INTERVAL keyword optional in INTERVAL clauses when ANSI mode enabled #20433

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0743ac3
Fix
maropu Jan 30, 2018
feccbc8
Fix
maropu Mar 13, 2019
8e414ac
Fix
maropu Mar 13, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 18 additions & 6 deletions docs/sql-reserved-and-non-reserved-keywords.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,8 @@ The list of reserved and non-reserved keywords can change according to the confi
<tr><td>DATABASE</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>DATABASES</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>DATE</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>DAY</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>DAY</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>DAYS</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>DBPROPERTIES</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>DEALLOCATE</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>DEC</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
Expand Down Expand Up @@ -230,7 +231,8 @@ The list of reserved and non-reserved keywords can change according to the confi
<tr><td>HANDLER</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>HAVING</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>HOLD</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>HOUR</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>HOUR</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>HOURS</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>IDENTITY</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>IF</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>IGNORE</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
Expand Down Expand Up @@ -313,13 +315,19 @@ The list of reserved and non-reserved keywords can change according to the confi
<tr><td>MEMBER</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>MERGE</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>METHOD</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>MICROSECOND</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>MICROSECONDS</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>MILLISECOND</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>MILLISECONDS</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>MIN</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>MINUS</td><td>reserved</td><td>reserved</td><td>non-reserved</td></tr>
<tr><td>MINUTE</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>MINUTE</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>MINUTES</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>MOD</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>MODIFIES</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>MODULE</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>MONTH</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>MONTH</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>MONTHS</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>MSCK</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>MULTISET</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>NAMES</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
Expand Down Expand Up @@ -448,7 +456,8 @@ The list of reserved and non-reserved keywords can change according to the confi
<tr><td>SCOPE</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>SCROLL</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>SEARCH</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>SECOND</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>SECOND</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>SECONDS</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>SECTION</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>SEEK</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>SELECT</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
Expand Down Expand Up @@ -568,8 +577,11 @@ The list of reserved and non-reserved keywords can change according to the confi
<tr><td>WITH</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>WITHIN</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>WITHOUT</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>WEEK</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>WEEKS</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>WORK</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>WRITE</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>YEAR</td><td>non-reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>YEAR</td><td>reserved</td><td>non-reserved</td><td>reserved</td></tr>
<tr><td>YEARS</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
<tr><td>ZONE</td><td>non-reserved</td><td>non-reserved</td><td>non-reserved</td></tr>
</table>
74 changes: 72 additions & 2 deletions sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -655,18 +655,40 @@ booleanValue
;

interval
: INTERVAL intervalField*
: {ansi}? INTERVAL? intervalField+
| {!ansi}? INTERVAL intervalField*
;

intervalField
: value=intervalValue unit=identifier (TO to=identifier)?
: value=intervalValue unit=intervalUnit (TO to=intervalUnit)?
;

intervalValue
: (PLUS | MINUS)? (INTEGER_VALUE | DECIMAL_VALUE)
| STRING
;

intervalUnit
: DAY
| DAYS
| HOUR
| HOURS
| MICROSECOND
| MICROSECONDS
| MILLISECOND
| MILLISECONDS
| MINUTE
| MINUTES
| MONTH
| MONTHS
| SECOND
| SECONDS
| WEEK
| WEEKS
| YEAR
| YEARS
;

colPosition
: FIRST | AFTER identifier
;
Expand Down Expand Up @@ -795,6 +817,7 @@ ansiNonReserved
| DATA
| DATABASE
| DATABASES
| DAYS
| DBPROPERTIES
| DEFINED
| DELETE
Expand Down Expand Up @@ -825,6 +848,7 @@ ansiNonReserved
| FUNCTIONS
| GLOBAL
| GROUPING
| HOURS
| IF
| IGNORE
| IMPORT
Expand All @@ -851,6 +875,12 @@ ansiNonReserved
| LOGICAL
| MACRO
| MAP
| MICROSECOND
| MICROSECONDS
| MILLISECOND
| MILLISECONDS
| MINUTES
| MONTHS
| MSCK
| NO
| NULLS
Expand Down Expand Up @@ -891,6 +921,7 @@ ansiNonReserved
| ROW
| ROWS
| SCHEMA
| SECONDS
| SEPARATED
| SERDE
| SERDEPROPERTIES
Expand Down Expand Up @@ -924,7 +955,10 @@ ansiNonReserved
| USE
| VALUES
| VIEW
| WEEK
| WEEKS
| WINDOW
| YEARS
;

defaultReserved
Expand Down Expand Up @@ -996,6 +1030,8 @@ nonReserved
| DATA
| DATABASE
| DATABASES
| DAY
| DAYS
| DBPROPERTIES
| DEFINED
| DELETE
Expand Down Expand Up @@ -1037,6 +1073,8 @@ nonReserved
| GROUP
| GROUPING
| HAVING
| HOUR
| HOURS
| IF
| IGNORE
| IMPORT
Expand Down Expand Up @@ -1067,6 +1105,14 @@ nonReserved
| LOGICAL
| MACRO
| MAP
| MICROSECOND
| MICROSECONDS
| MILLISECOND
| MILLISECONDS
| MINUTE
| MINUTES
| MONTH
| MONTHS
| MSCK
| NO
| NOT
Expand Down Expand Up @@ -1115,6 +1161,8 @@ nonReserved
| ROLLUP
| ROW
| ROWS
| SECOND
| SECONDS
| SELECT
| SEPARATED
| SERDE
Expand Down Expand Up @@ -1157,10 +1205,14 @@ nonReserved
| USER
| VALUES
| VIEW
| WEEK
| WEEKS
| WHEN
| WHERE
| WINDOW
| WITH
| YEAR
| YEARS
;

SELECT: 'SELECT';
Expand Down Expand Up @@ -1199,6 +1251,24 @@ ASC: 'ASC';
DESC: 'DESC';
FOR: 'FOR';
INTERVAL: 'INTERVAL';
YEAR: 'YEAR';
YEARS: 'YEARS';
MONTH: 'MONTH';
MONTHS: 'MONTHS';
WEEK: 'WEEK';
WEEKS: 'WEEKS';
DAY: 'DAY';
DAYS: 'DAYS';
HOUR: 'HOUR';
HOURS: 'HOURS';
MINUTE: 'MINUTE';
MINUTES: 'MINUTES';
SECOND: 'SECOND';
SECONDS: 'SECONDS';
MILLISECOND: 'MILLISECOND';
MILLISECONDS: 'MILLISECONDS';
MICROSECOND: 'MICROSECOND';
MICROSECONDS: 'MICROSECONDS';
CASE: 'CASE';
WHEN: 'WHEN';
THEN: 'THEN';
Expand Down
91 changes: 65 additions & 26 deletions sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,6 @@ import org.apache.spark.unsafe.types.CalendarInterval
* CheckAnalysis classes.
*/
class ExpressionParserSuite extends PlanTest {
import CatalystSqlParser._
import org.apache.spark.sql.catalyst.dsl.expressions._
import org.apache.spark.sql.catalyst.dsl.plans._

Expand Down Expand Up @@ -585,49 +584,60 @@ class ExpressionParserSuite extends PlanTest {
}
}

val intervalUnits = Seq(
"year",
"month",
"week",
"day",
"hour",
"minute",
"second",
"millisecond",
"microsecond")

def intervalLiteral(u: String, s: String): Literal = {
Literal(CalendarInterval.fromSingleUnitString(u, s))
}

test("intervals") {
def intervalLiteral(u: String, s: String): Literal = {
Literal(CalendarInterval.fromSingleUnitString(u, s))
def checkIntervals(intervalValue: String, expected: Literal): Unit = {
assertEqual(s"interval $intervalValue", expected)

// SPARK-23264 Support interval values without INTERVAL clauses if ANSI SQL enabled
withSQLConf(SQLConf.ANSI_SQL_PARSER.key -> "true") {
assertEqual(intervalValue, expected)
}
}

// Empty interval statement
intercept("interval", "at least one time unit should be given for interval literal")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we shall still check the empty interval statement, now it shall produce a different error message?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea, antlr just throws an exception when hitting this case;

scala> sql("select cast('2018-01-12' as DATE) + 1 days").show
+---------------------------------------------------------------------------+
|CAST(CAST(CAST(2018-01-12 AS DATE) AS TIMESTAMP) + interval 1 days AS DATE)|
+---------------------------------------------------------------------------+
|                                                                 2018-01-13|
+---------------------------------------------------------------------------+

scala> sql("select cast('2018-01-12' as DATE) + interval").show
org.apache.spark.sql.AnalysisException: cannot resolve '`interval`' given input columns: []; line 1 pos 36;
'Project [unresolvedalias((cast(2018-01-12 as date) + 'interval), None)]
+- OneRowRelation


// Single Intervals.
val units = Seq(
"year",
"month",
"week",
"day",
"hour",
"minute",
"second",
"millisecond",
"microsecond")
val forms = Seq("", "s")
val values = Seq("0", "10", "-7", "21")
units.foreach { unit =>
intervalUnits.foreach { unit =>
forms.foreach { form =>
values.foreach { value =>
val expected = intervalLiteral(unit, value)
assertEqual(s"interval $value $unit$form", expected)
assertEqual(s"interval '$value' $unit$form", expected)
checkIntervals(s"$value $unit$form", expected)
checkIntervals(s"'$value' $unit$form", expected)
}
}
}

// Hive nanosecond notation.
assertEqual("interval 13.123456789 seconds", intervalLiteral("second", "13.123456789"))
assertEqual("interval -13.123456789 second", intervalLiteral("second", "-13.123456789"))
checkIntervals("13.123456789 seconds", intervalLiteral("second", "13.123456789"))
checkIntervals("-13.123456789 second", intervalLiteral("second", "-13.123456789"))

// Non Existing unit
intercept("interval 10 nanoseconds", "No interval can be constructed")
intercept("interval 10 nanoseconds",
"no viable alternative at input 'interval 10 nanoseconds'")

// Year-Month intervals.
val yearMonthValues = Seq("123-10", "496-0", "-2-3", "-123-0")
yearMonthValues.foreach { value =>
val result = Literal(CalendarInterval.fromYearMonthString(value))
assertEqual(s"interval '$value' year to month", result)
checkIntervals(s"'$value' year to month", result)
}

// Day-Time intervals.
Expand All @@ -640,22 +650,51 @@ class ExpressionParserSuite extends PlanTest {
"1 0:0:1")
datTimeValues.foreach { value =>
val result = Literal(CalendarInterval.fromDayTimeString(value))
assertEqual(s"interval '$value' day to second", result)
checkIntervals(s"'$value' day to second", result)
}

// Unknown FROM TO intervals
intercept("interval 10 month to second", "Intervals FROM month TO second are not supported.")
intercept("interval 10 month to second",
"Intervals FROM month TO second are not supported.")

// Composed intervals.
assertEqual(
"interval 3 months 22 seconds 1 millisecond",
checkIntervals(
"3 months 22 seconds 1 millisecond",
Literal(new CalendarInterval(3, 22001000L)))
assertEqual(
"interval 3 years '-1-10' year to month 3 weeks '1 0:0:2' day to second",
checkIntervals(
"3 years '-1-10' year to month 3 weeks '1 0:0:2' day to second",
Literal(new CalendarInterval(14,
22 * CalendarInterval.MICROS_PER_DAY + 2 * CalendarInterval.MICROS_PER_SECOND)))
}

test("SPARK-23264 Interval Compatibility tests") {
def checkIntervals(intervalValue: String, expected: Literal): Unit = {
withSQLConf(SQLConf.ANSI_SQL_PARSER.key -> "true") {
assertEqual(intervalValue, expected)
}

// Compatibility tests: If ANSI SQL disabled, `intervalValue` should be parsed as an alias
withSQLConf(SQLConf.ANSI_SQL_PARSER.key -> "false") {
val aliases = defaultParser.parseExpression(intervalValue).collect {
case a @ Alias(_: Literal, name)
if intervalUnits.exists { unit => name.startsWith(unit) } => a
}
assert(aliases.size === 1)
}
}
val forms = Seq("", "s")
val values = Seq("5", "1", "-11", "8")
intervalUnits.foreach { unit =>
forms.foreach { form =>
values.foreach { value =>
val expected = intervalLiteral(unit, value)
checkIntervals(s"$value $unit$form", expected)
checkIntervals(s"'$value' $unit$form", expected)
}
}
}
}

test("composed expressions") {
assertEqual("1 + r.r As q", (Literal(1) + UnresolvedAttribute("r.r")).as("q"))
assertEqual("1 - f('o', o(bar))", Literal(1) - 'f.function("o", 'o.function('bar)))
Expand Down
Loading