Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-31710][SQL] Fail casting numeric to timestamp by default #28593

Closed
wants to merge 101 commits into from
Closed
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
101 commits
Select commit Hold shift + click to select a range
bb1efa2
Merge pull request #1 from apache/master
GuoPhilipse May 20, 2020
1459d5b
Merge pull request #2 from apache/master
GuoPhilipse May 22, 2020
88c40fe
Merge pull request #3 from apache/master
GuoPhilipse May 22, 2020
8d1deee
fail casting from integral to timestamp
GuoPhilipse May 22, 2020
df22083
Merge pull request #4 from apache/master
GuoPhilipse May 25, 2020
a02c6c7
'fix-testcase-fail'
GuoPhilipse May 25, 2020
e61c484
'add-restore-flag'
GuoPhilipse May 25, 2020
d99cc30
'fix-space'
GuoPhilipse May 25, 2020
0436611
Merge pull request #5 from apache/master
GuoPhilipse May 27, 2020
0a1a6a5
fix code style
GuoPhilipse May 27, 2020
ed80c84
Merge pull request #6 from apache/master
GuoPhilipse May 27, 2020
bae99b1
fix code style
GuoPhilipse May 27, 2020
39da7be
improve code
GuoPhilipse May 27, 2020
39ca87c
Merge pull request #7 from apache/master
GuoPhilipse May 28, 2020
0189b91
fix test cases failure
GuoPhilipse May 28, 2020
2c29c5b
fix test name
GuoPhilipse May 28, 2020
fd3efa2
fix test case error
GuoPhilipse May 28, 2020
69198c6
Merge branch 'master' into 31710-fix-compatibility
GuoPhilipse May 28, 2020
c3b3c89
Merge pull request #8 from apache/master
GuoPhilipse May 28, 2020
38f6e64
fix test case error
GuoPhilipse May 29, 2020
43f9b4e
Merge branch '31710-fix-compatibility' of https://github.com/GuoPhili…
GuoPhilipse May 29, 2020
852d730
fix cmd to bash
GuoPhilipse May 29, 2020
2e55be3
fix scala style
GuoPhilipse May 29, 2020
7637256
fix error test case
GuoPhilipse May 29, 2020
87cada7
fix scala style
GuoPhilipse May 29, 2020
13ae816
fix fail test cases
GuoPhilipse May 29, 2020
4abd8d3
fix scala style
GuoPhilipse May 29, 2020
b13316f
fix error test case
GuoPhilipse May 29, 2020
50cc0cf
fix scala style
GuoPhilipse May 29, 2020
22638e4
fix test case error
GuoPhilipse May 30, 2020
b1dda01
fix test case error
GuoPhilipse May 30, 2020
38f0e76
fix test case error
GuoPhilipse May 30, 2020
e478c90
fix pyspark test error
GuoPhilipse May 31, 2020
7348917
fix python style
GuoPhilipse May 31, 2020
e50e5cb
annotation examples casting from long to timestamp
GuoPhilipse Jun 1, 2020
2344e1f
fix pyspark test error
GuoPhilipse Jun 1, 2020
c3546eb
Merge pull request #9 from apache/master
GuoPhilipse Jun 2, 2020
c7cf8b9
Merge branch 'master' of https://github.com/GuoPhilipse/spark into 31…
GuoPhilipse Jun 2, 2020
5f138af
fix to_date using cast long to timestamp
GuoPhilipse Jun 2, 2020
77a339a
Merge pull request #10 from apache/master
GuoPhilipse Jun 3, 2020
3a3b93e
Merge branch 'master' of https://github.com/GuoPhilipse/spark into 31…
GuoPhilipse Jun 3, 2020
a6b4f74
Merge pull request #11 from apache/master
GuoPhilipse Jun 5, 2020
1b52fd6
submit to local
GuoPhilipse Jun 5, 2020
293fa9f
Merge branch 'master' of https://github.com/GuoPhilipse/spark into 31…
GuoPhilipse Jun 5, 2020
9cf87e8
extend functions and use new functions
GuoPhilipse Jun 5, 2020
cb21d5d
fix code style.etc
GuoPhilipse Jun 5, 2020
726371c
fix code style
GuoPhilipse Jun 5, 2020
8ed4fc4
fix doc format.etc
GuoPhilipse Jun 6, 2020
96c197c
fix testcase error
GuoPhilipse Jun 6, 2020
3d1819d
failed test case
GuoPhilipse Jun 6, 2020
be88f01
fix testcase error
GuoPhilipse Jun 6, 2020
fd7815a
fix find no test result problem
GuoPhilipse Jun 7, 2020
e14c4f9
fix find no test result problem
GuoPhilipse Jun 7, 2020
7cb0a54
fix testcase error
GuoPhilipse Jun 7, 2020
7ad82da
fix found no test result file
GuoPhilipse Jun 7, 2020
00ed960
fix test case error
GuoPhilipse Jun 7, 2020
664277e
Merge pull request #12 from apache/master
GuoPhilipse Jun 7, 2020
6d9eecb
Merge branch 'master' of https://github.com/GuoPhilipse/spark into 31…
GuoPhilipse Jun 7, 2020
b8b2919
fix test case fail
GuoPhilipse Jun 7, 2020
eeb0a61
fix test case fail
GuoPhilipse Jun 7, 2020
409c821
fix check spark streaming ui timeout
GuoPhilipse Jun 8, 2020
ec2cf54
fix additional space
GuoPhilipse Jun 8, 2020
3fd6d02
fix test case fail
GuoPhilipse Jun 8, 2020
fd677c9
Merge pull request #13 from apache/master
GuoPhilipse Jun 8, 2020
108dfea
merge
GuoPhilipse Jun 8, 2020
47f210c
fix test error.etc
GuoPhilipse Jun 8, 2020
860500d
fix test error.etc
GuoPhilipse Jun 8, 2020
4c7ddb4
revert support fraction types.etc
GuoPhilipse Jun 9, 2020
b43a140
remove useless space
GuoPhilipse Jun 9, 2020
1d5c750
fix code style.etc
GuoPhilipse Jun 9, 2020
7b26d0c
remove duplicated test cases
GuoPhilipse Jun 10, 2020
c7f2a9b
fix python style
GuoPhilipse Jun 10, 2020
8a5e9be
fix code style
GuoPhilipse Jun 10, 2020
93b1f63
Merge pull request #14 from apache/master
GuoPhilipse Jun 10, 2020
a6a9bd4
fix failed test case
GuoPhilipse Jun 10, 2020
a5b5474
Merge pull request #15 from apache/master
GuoPhilipse Jun 11, 2020
4fecf9b
Merge branch 'master' of https://github.com/GuoPhilipse/spark into 31…
GuoPhilipse Jun 11, 2020
f995f46
rebase
GuoPhilipse Jun 11, 2020
c8d5aa5
rebase
GuoPhilipse Jun 11, 2020
b4f4d53
delete test file
GuoPhilipse Jun 11, 2020
f4556a4
Merge pull request #16 from apache/master
GuoPhilipse Jun 11, 2020
f4da70b
rebase latest code
GuoPhilipse Jun 11, 2020
7132fca
rebase lastet code
GuoPhilipse Jun 11, 2020
7ffcde2
fix error cases
GuoPhilipse Jun 12, 2020
f84caa1
fix error cases
GuoPhilipse Jun 12, 2020
6d56a83
fix error cases
GuoPhilipse Jun 12, 2020
5716f40
remove cmd test
GuoPhilipse Jun 12, 2020
64acff3
'empty'
GuoPhilipse Jun 12, 2020
d26bd34
fix test case error
GuoPhilipse Jun 12, 2020
9545c9f
delete test file
GuoPhilipse Jun 12, 2020
01f42e6
fix python style
GuoPhilipse Jun 13, 2020
e29409d
fix python style
GuoPhilipse Jun 13, 2020
183d418
trigger rebuild
GuoPhilipse Jun 13, 2020
1764c76
fix python style
GuoPhilipse Jun 13, 2020
74eb6ca
improve python style
GuoPhilipse Jun 13, 2020
5f06714
fix python style
GuoPhilipse Jun 13, 2020
8fe1960
improve test queality
GuoPhilipse Jun 15, 2020
ffa3079
remove unnecessary commit
GuoPhilipse Jun 15, 2020
bc4b62c
remove test file
GuoPhilipse Jun 15, 2020
08aee30
improve test case
GuoPhilipse Jun 15, 2020
12b4239
improve test case
GuoPhilipse Jun 15, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ object Cast {
case (StringType, TimestampType) => true
case (BooleanType, TimestampType) => true
case (DateType, TimestampType) => true
case (_: NumericType, TimestampType) => true
case (_: FractionalType, TimestampType) => true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's forbid fraction as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and provide a legacy config to restore the old behavior.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok , will correct it

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you need forbiding casting timestamp to numeric type at the same time,maybe someone will complain about it later


case (StringType, DateType) => true
case (TimestampType, DateType) => true
Expand Down Expand Up @@ -138,10 +138,8 @@ object Cast {
case (_: CalendarIntervalType, StringType) => true
case (NullType, _) => true

// Spark supports casting between long and timestamp, please see `longToTimestamp` and
// `timestampToLong` for details.
// spark forbid casting from integral to timestamp, more details in [SPARK-31790]
case (TimestampType, LongType) => true
case (LongType, TimestampType) => true

case (ArrayType(fromType, fn), ArrayType(toType, tn)) =>
resolvableNullability(fn, tn) && canUpCast(fromType, toType)
Expand Down Expand Up @@ -266,7 +264,12 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
TypeCheckResult.TypeCheckSuccess
} else {
TypeCheckResult.TypeCheckFailure(
s"cannot cast ${child.dataType.catalogString} to ${dataType.catalogString}")
if (child.dataType.isInstanceOf[IntegralType] && dataType.isInstanceOf[TimestampType]) {
s"cannot cast ${child.dataType.catalogString} to ${dataType.catalogString}," +
s"please use function TIMESTAMP_SECONDS/TIMESTAMP_MILLIS/TIMESTAMP_MICROS instand"
} else {
s"cannot cast ${child.dataType.catalogString} to ${dataType.catalogString}"
})
}
}

Expand Down Expand Up @@ -425,14 +428,6 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
buildCast[UTF8String](_, utfs => DateTimeUtils.stringToTimestamp(utfs, zoneId).orNull)
case BooleanType =>
buildCast[Boolean](_, b => if (b) 1L else 0)
case LongType =>
buildCast[Long](_, l => longToTimestamp(l))
case IntegerType =>
buildCast[Int](_, i => longToTimestamp(i.toLong))
case ShortType =>
buildCast[Short](_, s => longToTimestamp(s.toLong))
case ByteType =>
buildCast[Byte](_, b => longToTimestamp(b.toLong))
case DateType =>
buildCast[Int](_, d => epochDaysToMicros(d, zoneId))
// TimestampWritable.decimalToTimestamp
Expand All @@ -453,8 +448,6 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
if (d.isNaN || d.isInfinite) null else (d * MICROS_PER_SECOND).toLong
}

// converting seconds to us
private[this] def longToTimestamp(t: Long): Long = SECONDS.toMicros(t)
// converting us to seconds
private[this] def timestampToLong(ts: Long): Long = {
Math.floorDiv(ts, MICROS_PER_SECOND)
Expand Down Expand Up @@ -1229,8 +1222,6 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
"""
case BooleanType =>
(c, evPrim, evNull) => code"$evPrim = $c ? 1L : 0L;"
case _: IntegralType =>
(c, evPrim, evNull) => code"$evPrim = ${longToTimeStampCode(c)};"
case DateType =>
val zoneIdClass = classOf[ZoneId]
val zid = JavaCode.global(
Expand Down Expand Up @@ -1277,7 +1268,6 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
val block = inline"new java.math.BigDecimal($MICROS_PER_SECOND)"
code"($d.toBigDecimal().bigDecimal().multiply($block)).longValue()"
}
private[this] def longToTimeStampCode(l: ExprValue): Block = code"$l * (long)$MICROS_PER_SECOND"
private[this] def timestampToLongCode(ts: ExprValue): Block =
code"java.lang.Math.floorDiv($ts, $MICROS_PER_SECOND)"
private[this] def timestampToDoubleCode(ts: ExprValue): Block =
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1311,6 +1311,13 @@ class CastSuite extends CastSuiteBase {
checkEvaluation(cast(negativeTs, LongType), expectedSecs)
}
}

test("SPARK-31710:fail casting from integral to timestamp by default") {
assert(!cast(2.toByte, TimestampType).resolved)
assert(!cast(10.toShort, TimestampType).resolved)
assert(!cast(3, TimestampType).resolved)
assert(!cast(10L, TimestampType).resolved)
}
}

/**
Expand Down