[SPARK-23316][SQL] AnalysisException after max iteration reached for IN query #20548
Conversation
|
Test build #87218 has finished for PR 20548 at commit
|
|
This sounds another regression in Spark 2.3. cc @cloud-fan @dilipbiswal |
|
@gatorsmile Yeah. Its due to the changes made for SPARK-21759. The fix looks okay to me. One other aspect would be to stop adding the additional projects we keep adding during in IN Coercion. |
|
The fix LGTM. cc @sameeragarwal |
|
retest this please |
| @@ -298,22 +298,24 @@ object DataType { | |||
| * Returns true if the two data types share the same "shape", i.e. the types (including | |||
| * nullability) are the same, but the field names don't need to be the same. | |||
| */ | |||
| def equalsStructurally(from: DataType, to: DataType): Boolean = { | |||
| def equalsStructurally(from: DataType, to: DataType, | |||
| ignoreNullability: Boolean = false): Boolean = { | |||
| @@ -298,22 +298,24 @@ object DataType { | |||
| * Returns true if the two data types share the same "shape", i.e. the types (including | |||
| * nullability) are the same, but the field names don't need to be the same. | |||
There was a problem hiding this comment.
This comments need an update too.
|
Test build #87364 has finished for PR 20548 at commit
|
|
retest this please |
|
Anyone knows which commit introduced this bug? |
|
@cloud-fan Its SPARK-21759. Apart from fixing the bug, this PR also refactored the code in checkInputTypes and that seemed to have caused it. |
|
Test build #87381 has finished for PR 20548 at commit
|
…IN query ## What changes were proposed in this pull request? Added flag ignoreNullability to DataType.equalsStructurally. The previous semantic is for ignoreNullability=false. When ignoreNullability=true equalsStructurally ignores nullability of contained types (map key types, value types, array element types, structure field types). In.checkInputTypes calls equalsStructurally to check if the children types match. They should match regardless of nullability (which is just a hint), so it is now called with ignoreNullability=true. ## How was this patch tested? New test in SubquerySuite Author: Bogdan Raducanu <bogdan@databricks.com> Closes #20548 from bogdanrdc/SPARK-23316. (cherry picked from commit 05d0512) Signed-off-by: gatorsmile <gatorsmile@gmail.com>
|
This is a regression introduced by #18968. We have to merge to 2.3. I resolved my comments when I merge the codes. Thanks! Merged to master/2.3 |
What changes were proposed in this pull request?
Added flag ignoreNullability to DataType.equalsStructurally.
The previous semantic is for ignoreNullability=false.
When ignoreNullability=true equalsStructurally ignores nullability of contained types (map key types, value types, array element types, structure field types).
In.checkInputTypes calls equalsStructurally to check if the children types match. They should match regardless of nullability (which is just a hint), so it is now called with ignoreNullability=true.
How was this patch tested?
New test in SubquerySuite