[SPARK-31102][SQL][3.0] Spark-sql fails to parse when contains comment #28565
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This PR introduces a change to false for the insideComment flag on a newline. Fixing the issue introduced by SPARK-30049.
Previously on SPARK-30049 a comment containing an unclosed quote produced the following issue:
```
spark-sql> SELECT 1 -- someone's comment here
> ;
Error in query:
extraneous input ';' expecting <EOF>(line 2, pos 0)
== SQL ==
SELECT 1 -- someone's comment here
;
^^^
```
This was caused because there was no flag for comment sections inside the splitSemiColon method to ignore quotes. SPARK-30049 added that flag and fixed the issue, but introduced the follwoing problem:
```
spark-sql> select
> 1,
> -- two
> 2;
Error in query:
mismatched input '<EOF>' expecting {'(', 'ADD', 'AFTER', 'ALL', 'ALTER', ...}(line 3, pos 2)
== SQL ==
select
1,
--^^^
```
This issue is generated by a missing turn-off for the insideComment flag with a newline.
No
- For previous tests using line-continuity(`\`) it was added a line-continuity rule in the SqlBase.g4 file to add the functionality to the SQL context.
- A new test for inline comments was added.
Closes apache#27920 from javierivanov/SPARK-31102.
Authored-by: Javier Fuentes <j.fuentes.m@icloud.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
|
ok to test |
cloud-fan
approved these changes
May 18, 2020
|
Test build #122783 has finished for PR 28565 at commit
|
|
retest this please |
maropu
approved these changes
May 18, 2020
|
Test build #122788 has finished for PR 28565 at commit
|
|
It seems the failure above is not related to this PR. See: #28566 |
|
retest this please |
|
Looks nice! Thanks for re-trigger the tests, @HyukjinKwon . |
|
Test build #122789 has finished for PR 28565 at commit
|
|
retest this please |
|
Test build #122795 has finished for PR 28565 at commit
|
|
The newly added test all fails. Seems there is something different in branch 3.0. |
I added those test by mistake while cherry-picking. Checking. |
|
Test build #122805 has finished for PR 28565 at commit
|
maropu
pushed a commit
that referenced
this pull request
May 18, 2020
This PR introduces a change to false for the insideComment flag on a newline. Fixing the issue introduced by SPARK-30049. Backport to 3.0 from #27920 Previously on SPARK-30049 a comment containing an unclosed quote produced the following issue: ``` spark-sql> SELECT 1 -- someone's comment here > ; Error in query: extraneous input ';' expecting <EOF>(line 2, pos 0) == SQL == SELECT 1 -- someone's comment here ; ^^^ ``` This was caused because there was no flag for comment sections inside the splitSemiColon method to ignore quotes. SPARK-30049 added that flag and fixed the issue, but introduced the follwoing problem: ``` spark-sql> select > 1, > -- two > 2; Error in query: mismatched input '<EOF>' expecting {'(', 'ADD', 'AFTER', 'ALL', 'ALTER', ...}(line 3, pos 2) == SQL == select 1, --^^^ ``` This issue is generated by a missing turn-off for the insideComment flag with a newline. No - For previous tests using line-continuity(`\`) it was added a line-continuity rule in the SqlBase.g4 file to add the functionality to the SQL context. - A new test for inline comments was added. Closes #27920 from javierivanov/SPARK-31102. Authored-by: Javier Fuentes <j.fuentes.micloud.com> Signed-off-by: Wenchen Fan <wenchendatabricks.com> ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? Closes #28565 from javierivanov/SPARK-3.0-31102. Authored-by: Javier Fuentes <j.fuentes.m@icloud.com> Signed-off-by: Takeshi Yamamuro <yamamuro@apache.org>
|
Thanks! Merged to branch-3.0. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces a change to false for the insideComment flag on a newline. Fixing the issue introduced by SPARK-30049.
Backport to 3.0 from #27920
Previously on SPARK-30049 a comment containing an unclosed quote produced the following issue:
This was caused because there was no flag for comment sections inside the splitSemiColon method to ignore quotes. SPARK-30049 added that flag and fixed the issue, but introduced the follwoing problem:
This issue is generated by a missing turn-off for the insideComment flag with a newline.
No
\) it was added a line-continuity rule in the SqlBase.g4 file to add the functionality to the SQL context.Closes #27920 from javierivanov/SPARK-31102.
Authored-by: Javier Fuentes j.fuentes.m@icloud.com
Signed-off-by: Wenchen Fan wenchen@databricks.com
What changes were proposed in this pull request?
Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?