[SPARK-21769] [SQL] Add a table-specific option for always respecting schemas inferred/controlled by Spark SQL

[gatorsmile]

What changes were proposed in this pull request?

For Hive-serde tables, we always respect the schema stored in Hive metastore, because the schema could be altered by the other engines that share the same metastore. Thus, we always trust the metastore-controlled schema for Hive-serde tables when the schemas are different (without considering the nullability and cases). However, in some scenarios, Hive metastore also could INCORRECTLY overwrite the schemas when the serde and Hive metastore built-in serde are different.

The proposed solution is to introduce a table-specific option for such scenarios. For a specific table, users can make Spark always respect Spark-inferred/controlled schema instead of trusting metastore-controlled schema. By default, we trust Hive metastore-controlled schema.

How was this patch tested?

Added a cross-version test case

[SparkQA]

Test build #80882 has finished for PR 19003 at commit 4c7349f.

• This patch fails SparkR unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class SourceOptions(

[SparkQA]

Test build #80899 has finished for PR 19003 at commit 4c7349f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class SourceOptions(

[gatorsmile]

For such an example that requires users setting TBLPROPERTIES, it sounds like we are unable to use the CREATE TABLE USING command. cc @cloud-fan

[cloud-fan]

There was an argument about whether we should add TBLPROPERTIES, and we decided to not add it. I'm totally fine to add it if it's necessary.

[gatorsmile]

cc @sameeragarwal @cloud-fan

[cloud-fan]

LGTM

[gatorsmile]

retest this please

[sameeragarwal]

nit: update docs

[sameeragarwal]

LGTM, thanks! Are these table properties documented somewhere?

[gatorsmile]

We might need a dedicated section for documenting all the table-specific conf options.

[SparkQA]

Test build #80999 has finished for PR 19003 at commit 4c7349f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class SourceOptions(

[sameeragarwal]

Merging to master, thanks!

[SparkQA]

Test build #81001 has finished for PR 19003 at commit 36339c8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.