[SPARK-3816][SQL] Add table properties from storage handler to output jobConf

[alexliu68]

...ob conf in SparkHadoopWriter class

[AmplabJenkins]

Can one of the admins verify this patch?

[AmplabJenkins]

Can one of the admins verify this patch?

[marmbrus]

Can you explain more about what this is doing and why we need it? Perhaps in comments to the code.

Also, please fix the title so that it doesn't truncate.

[marmbrus]

ok to test

[SparkQA]

QA tests have started for PR 2677 at commit 46edcbc.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2677 at commit 46edcbc.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21508/Test PASSed.

[alexliu68]

I add comment to code

[SparkQA]

QA tests have started for PR 2677 at commit 003d397.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2677 at commit 003d397.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21538/Test FAILed.

[SparkQA]

QA tests have started for PR 2677 at commit 003d397.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2677 at commit 003d397.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Can one of the admins verify this patch?

[chenghao-intel]

Can you please add a unit test which will failure without this PR? I think this is really helpful as we have some serde relevant features / bugs.
e.g. #2570 #2674 etc.

[alexliu68]

Sure, I am pretty busy on my daily work right now. I will find time to add unit test. I fix this issue when I integrate CassandraStorageHandler.

[liancheng]

test this please

[SparkQA]

QA tests have started for PR 2677 at commit 003d397.

• This patch merges cleanly.

[SparkQA]

Tests timed out for PR 2677 at commit 003d397 after a configured wait of 120m.

[marmbrus]

Can you please rebase this?

[alexliu68]

It's rebased.

[SparkQA]

Test build #472 has started for PR 2677 at commit 79c269b.

• This patch merges cleanly.

[SparkQA]

Test build #472 has finished for PR 2677 at commit 79c269b.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[marmbrus]

Hmm, this failure looks related to @liancheng patch on statistics that I
just merged.
On Oct 26, 2014 5:18 PM, "Apache Spark QA" notifications@github.com wrote:

Test build #472 has finished
https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/472/consoleFull
for PR 2677 at commit 79c269b
79c269b
.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

—
Reply to this email directly or view it on GitHub
#2677 (comment).

[liancheng]

@marmbrus Yes, it's related. The testData table was cached by some other test suite executed before JoinSuite, and one of the expected ShuffledHashJoin was optimized to BroadcastedHashJoin because the in-memory table statistics suggest so.

This failure is sensitive to test suite execution order, but can be easily reproduced by caching testData first in JoinSuite.

[liancheng]

#2960 is filed to fix this issue.

[liancheng]

retest this please

This failure is sensitive to test suite execution order, so here I'd like to have another try.

[SparkQA]

Test build #474 has started for PR 2677 at commit 79c269b.

• This patch merges cleanly.

[SparkQA]

Test build #474 has finished for PR 2677 at commit 79c269b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[marmbrus]

Thanks! Merged to master.