[SPARK-11647] Attempt to reduce time/flakiness of Thriftserver CLI and SparkSubmit tests #9623
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #45615 has finished for PR 9623 at commit
|
|
With this patch, CliSuite takes ~2 minutes vs. 6-7 minutes normally. Similarly, HiveSparkSubmitSuite took ~2 minutes vs. 5+ normally. |
| override def beforeEach(): Unit = { | ||
| super.beforeEach() | ||
| System.setProperty("spark.testing", "true") |
There was a problem hiding this comment.
what's the difference between having it in beforeEach and beforeAll for this specific config?
There was a problem hiding this comment.
I think it's marginally easier to reason about the interaction between BeforeAndAfterEach and ResetSystemProperties.
|
LGTM |
|
Thanks - merging this. |
asfgit
pushed a commit
that referenced
this pull request
Nov 11, 2015
…d SparkSubmit tests This patch aims to reduce the test time and flakiness of HiveSparkSubmitSuite, SparkSubmitSuite, and CliSuite. Key changes: - Disable IO synchronization calls for Derby writes, since durability doesn't matter for tests. This was done for HiveCompatibilitySuite in #6651 and resulted in huge test speedups. - Add a few missing `--conf`s to disable various Spark UIs. The CliSuite, in particular, never disabled these UIs, leaving it prone to port-contention-related flakiness. - Fix two instances where tests defined `beforeAll()` methods which were never called because the appropriate traits were not mixed in. I updated these tests suites to extend `BeforeAndAfterEach` so that they play nicely with our `ResetSystemProperties` trait. Author: Josh Rosen <joshrosen@databricks.com> Closes #9623 from JoshRosen/SPARK-11647. (cherry picked from commit 2d76e44) Signed-off-by: Reynold Xin <rxin@databricks.com>
dskrvk
pushed a commit
to dskrvk/spark
that referenced
this pull request
Nov 13, 2015
…d SparkSubmit tests This patch aims to reduce the test time and flakiness of HiveSparkSubmitSuite, SparkSubmitSuite, and CliSuite. Key changes: - Disable IO synchronization calls for Derby writes, since durability doesn't matter for tests. This was done for HiveCompatibilitySuite in apache#6651 and resulted in huge test speedups. - Add a few missing `--conf`s to disable various Spark UIs. The CliSuite, in particular, never disabled these UIs, leaving it prone to port-contention-related flakiness. - Fix two instances where tests defined `beforeAll()` methods which were never called because the appropriate traits were not mixed in. I updated these tests suites to extend `BeforeAndAfterEach` so that they play nicely with our `ResetSystemProperties` trait. Author: Josh Rosen <joshrosen@databricks.com> Closes apache#9623 from JoshRosen/SPARK-11647.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch aims to reduce the test time and flakiness of HiveSparkSubmitSuite, SparkSubmitSuite, and CliSuite.
Key changes:
--confs to disable various Spark UIs. The CliSuite, in particular, never disabled these UIs, leaving it prone to port-contention-related flakiness.beforeAll()methods which were never called because the appropriate traits were not mixed in. I updated these tests suites to extendBeforeAndAfterEachso that they play nicely with ourResetSystemPropertiestrait.