[SPARK-20831] [SQL] Fix INSERT OVERWRITE data source tables with IF NOT EXISTS #18050
Conversation
|
Test build #77153 has finished for PR 18050 at commit
|
|
|
||
| // If the partition already exists, the insert will overwrite the data |
There was a problem hiding this comment.
Seems IF NOT EXISTS works previously with INSERT OVERWRITE? It doesn't overwrite the existing data.
There was a problem hiding this comment.
So seems we should also add a test for IF NOT EXISTS with INSERT OVERWRITE for datasource table?
There was a problem hiding this comment.
That is why I used testPartitionedTable to do the test. It tests both data source and hive serde tables.
|
cc @cloud-fan |
|
LGTM |
| @@ -410,17 +410,20 @@ case class Hint(name: String, parameters: Seq[String], child: LogicalPlan) exten | |||
| * would have Map('a' -> Some('1'), 'b' -> None). | |||
| * @param query the logical plan representing data to write to. | |||
| * @param overwrite overwrite existing table or partitions. | |||
| * @param ifNotExists If true, only write if the table or partition does not exist. | |||
| * @param ifStaticPartitionNotExists If true, only write if the partition does not exist. | |||
There was a problem hiding this comment.
this name is a little verbose, how about ifPartitionNotExists?
| checkAnswer( | ||
| sql(selQuery), | ||
| Row("blarr2", "a", "b")) | ||
| testPartitionedTable("INSERT OVERWRITE - partition IF NOT EXISTS") { tableName => |
There was a problem hiding this comment.
it's a little weird to test data source table insertion in InsertIntoHiveTableSuite...
There was a problem hiding this comment.
anyway, we can fix it later
There was a problem hiding this comment.
Create and move all these tests to a new suite InsertIntoTableSuite?
There was a problem hiding this comment.
If it is not too verbose, I'd like to have them separated, instead of mixing together in one test suite under hive.
There was a problem hiding this comment.
We can move the test cases to a new one in /core and let InsertIntoHiveTableSuite extends that one.
There was a problem hiding this comment.
Yeah, we can do it as a test-only PR.
|
Test build #77173 has finished for PR 18050 at commit
|
…T EXISTS
### What changes were proposed in this pull request?
Currently, we have a bug when we specify `IF NOT EXISTS` in `INSERT OVERWRITE` data source tables. For example, given a query:
```SQL
INSERT OVERWRITE TABLE $tableName partition (b=2, c=3) IF NOT EXISTS SELECT 9, 10
```
we will get the following error:
```
unresolved operator 'InsertIntoTable Relation[a#425,d#426,b#427,c#428] parquet, Map(b -> Some(2), c -> Some(3)), true, true;;
'InsertIntoTable Relation[a#425,d#426,b#427,c#428] parquet, Map(b -> Some(2), c -> Some(3)), true, true
+- Project [cast(9#423 as int) AS a#429, cast(10#424 as int) AS d#430]
+- Project [9 AS 9#423, 10 AS 10#424]
+- OneRowRelation$
```
This PR is to fix the issue to follow the behavior of Hive serde tables
> INSERT OVERWRITE will overwrite any existing data in the table or partition unless IF NOT EXISTS is provided for a partition
### How was this patch tested?
Modified an existing test case
Author: gatorsmile <gatorsmile@gmail.com>
Closes #18050 from gatorsmile/insertPartitionIfNotExists.
(cherry picked from commit f3ed62a)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
|
thanks, merging to master/2.2! |
…T EXISTS
### What changes were proposed in this pull request?
Currently, we have a bug when we specify `IF NOT EXISTS` in `INSERT OVERWRITE` data source tables. For example, given a query:
```SQL
INSERT OVERWRITE TABLE $tableName partition (b=2, c=3) IF NOT EXISTS SELECT 9, 10
```
we will get the following error:
```
unresolved operator 'InsertIntoTable Relation[a#425,d#426,b#427,c#428] parquet, Map(b -> Some(2), c -> Some(3)), true, true;;
'InsertIntoTable Relation[a#425,d#426,b#427,c#428] parquet, Map(b -> Some(2), c -> Some(3)), true, true
+- Project [cast(9#423 as int) AS a#429, cast(10#424 as int) AS d#430]
+- Project [9 AS 9#423, 10 AS 10#424]
+- OneRowRelation$
```
This PR is to fix the issue to follow the behavior of Hive serde tables
> INSERT OVERWRITE will overwrite any existing data in the table or partition unless IF NOT EXISTS is provided for a partition
### How was this patch tested?
Modified an existing test case
Author: gatorsmile <gatorsmile@gmail.com>
Closes apache#18050 from gatorsmile/insertPartitionIfNotExists.
What changes were proposed in this pull request?
Currently, we have a bug when we specify
IF NOT EXISTSinINSERT OVERWRITEdata source tables. For example, given a query:we will get the following error:
This PR is to fix the issue to follow the behavior of Hive serde tables
How was this patch tested?
Modified an existing test case