Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] [Spark 4] Exceptions from DateTimeExceptions do not match Spark exceptions with ANSI enabled #11641

Open
rwlee opened this issue Oct 21, 2024 · 0 comments
Labels
Spark 4.0+ Spark 4.0+ issues

Comments

@rwlee
Copy link
Contributor

rwlee commented Oct 21, 2024

Description:
With ANSI enabled, when reading invalid date in EXCEPTION mode, the exception string from Spark does not match the exception from Spark.

Noticed in csv_test.py::test_read_valid_and_invalid_dates

On Spark pre 4.0: DateTimeException
On Spark 4.0+:

'pyspark.errors.exceptions.captured.SparkUpgradeException: [INCONSISTENT_BEHAVIOR_CROSS_VERSION.PARSE_DATETIME_BY_NEW_PARSER] You may get a different result due to the upgrading to Spark >= 3.0:
E       Fail to parse '2020-50-16' in the new parser.
E       You can set "spark.sql.legacy.timeParserPolicy" to "LEGACY" to restore the behavior before Spark 3.0, or set to "CORRECTED" and treat it as an invalid datetime string. SQLSTATE: 42K0B'

On the Spark RAPIDS: DateTimeException: One or more values is not a valid date

Repro:
Workaround for the test failure is in flight, once that fix is in -- run the test_read_valid_and_invalid_dates test on Spark 4.0 with EXCEPTION mode enabled.

Expected behavior
The overflow exception should match what is produced from Spark 4.

Misc:
Similar to #11556 + #11552 + #11550 -- exception names and types not aligning in ansi mode Spark 4.0+

@rwlee rwlee added ? - Needs Triage Need team to review and classify Spark 4.0+ Spark 4.0+ issues labels Oct 21, 2024
@rwlee rwlee changed the title [BUG] [Spark 4] Exceptions from do not match Spark exceptions with ANSI enabled [BUG] [Spark 4] Exceptions from DateTimeExceptions do not match Spark exceptions with ANSI enabled Oct 21, 2024
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Oct 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Spark 4.0+ Spark 4.0+ issues
Projects
None yet
Development

No branches or pull requests

2 participants