[SPARK-23348][SQL] append data using saveAsTable should adjust the data types #20527
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #87151 has finished for PR 20527 at commit
|
|
|
||
| Seq("c" -> 3).toDF("i", "j").write.mode("append").saveAsTable("t") | ||
| checkAnswer(spark.table("t"), Row(1, "a") :: Row(2, "b") :: Row(3, "c") | ||
| :: Row(null, "3") :: Nil) |
There was a problem hiding this comment.
Thank you for pining me, @cloud-fan . +1 for the patch, LGTM.
dongjoon-hyun
approved these changes
Feb 7, 2018
jiangxb1987
reviewed
Feb 8, 2018
| @@ -346,37 +349,11 @@ case class PreprocessTableInsertion(conf: SQLConf) extends Rule[LogicalPlan] wit | |||
| """.stripMargin) | |||
| } | |||
|
|
|||
| castAndRenameChildOutput(insert.copy(partition = normalizedPartSpec), expectedColumns) | |||
| insert.copy(query = newQuery, partition = normalizedPartSpec) | |||
There was a problem hiding this comment.
nit: don't need to copy the newQuery if it is the same as query.
There was a problem hiding this comment.
it's also ok to always copy it and the code is neater.
jiangxb1987
reviewed
Feb 8, 2018
| @@ -132,6 +134,32 @@ class InMemoryCatalogedDDLSuite extends DDLSuite with SharedSQLContext with Befo | |||
| checkAnswer(spark.table("t"), Row(Row("a", 1)) :: Nil) | |||
| } | |||
| } | |||
|
|
|||
| // TODO: This test is copied from HiveDDLSuite, unify it later. | |||
| test("SPARK-23348: append data to data source table with saveAsTable") { | |||
There was a problem hiding this comment.
Do we also want to cover the following case:
2) Target tables have column metadata
?
There was a problem hiding this comment.
maybe we can add this when unifying the test cases?
|
LGTM only some nits. |
|
LGTM Thanks! Merged to master/2.3 |
asfgit
pushed a commit
that referenced
this pull request
Feb 8, 2018
…ta types ## What changes were proposed in this pull request? For inserting/appending data to an existing table, Spark should adjust the data types of the input query according to the table schema, or fail fast if it's uncastable. There are several ways to insert/append data: SQL API, `DataFrameWriter.insertInto`, `DataFrameWriter.saveAsTable`. The first 2 ways create `InsertIntoTable` plan, and the last way creates `CreateTable` plan. However, we only adjust input query data types for `InsertIntoTable`, and users may hit weird errors when appending data using `saveAsTable`. See the JIRA for the error case. This PR fixes this bug by adjusting data types for `CreateTable` too. ## How was this patch tested? new test. Author: Wenchen Fan <wenchen@databricks.com> Closes #20527 from cloud-fan/saveAsTable. (cherry picked from commit 7f5f5fb) Signed-off-by: gatorsmile <gatorsmile@gmail.com>
robert3005
pushed a commit
to palantir/spark
that referenced
this pull request
Feb 12, 2018
…ta types ## What changes were proposed in this pull request? For inserting/appending data to an existing table, Spark should adjust the data types of the input query according to the table schema, or fail fast if it's uncastable. There are several ways to insert/append data: SQL API, `DataFrameWriter.insertInto`, `DataFrameWriter.saveAsTable`. The first 2 ways create `InsertIntoTable` plan, and the last way creates `CreateTable` plan. However, we only adjust input query data types for `InsertIntoTable`, and users may hit weird errors when appending data using `saveAsTable`. See the JIRA for the error case. This PR fixes this bug by adjusting data types for `CreateTable` too. ## How was this patch tested? new test. Author: Wenchen Fan <wenchen@databricks.com> Closes apache#20527 from cloud-fan/saveAsTable.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
For inserting/appending data to an existing table, Spark should adjust the data types of the input query according to the table schema, or fail fast if it's uncastable.
There are several ways to insert/append data: SQL API,
DataFrameWriter.insertInto,DataFrameWriter.saveAsTable. The first 2 ways createInsertIntoTableplan, and the last way createsCreateTableplan. However, we only adjust input query data types forInsertIntoTable, and users may hit weird errors when appending data usingsaveAsTable. See the JIRA for the error case.This PR fixes this bug by adjusting data types for
CreateTabletoo.How was this patch tested?
new test.