This repository has been archived by the owner on Jan 9, 2020. It is now read-only.
Fix k8s integration tests #44
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lins05
commented
Jan 25, 2017
| log4j.logger.org.apache.hadoop=WARN | ||
| log4j.logger.org.eclipse.jetty=WARN | ||
| log4j.logger.org.mortbay=WARN | ||
| log4j.logger.org.spark_project.jetty=WARN |
There was a problem hiding this comment.
This file is needed for each sub module. Otherwise it would throw errors like the one mentioned in https://issues.apache.org/jira/browse/SPARK-17543.
I guess it's because scalatest-maven-plugin sets the log4j.configuration property to file:src/test/resources/log4j.properties. See https://github.com/apache/spark/blob/v2.1.0/pom.xml#L2019
mccheah
reviewed
Jan 25, 2017
| @@ -152,6 +158,8 @@ private[spark] class KubernetesSuite extends SparkFunSuite with BeforeAndAfter { | |||
| .set("spark.executor.cores", "1") | |||
| .set("spark.executors.instances", "1") | |||
| .set("spark.app.id", "spark-pi") | |||
| .set("spark.ui.enabled", "true") | |||
There was a problem hiding this comment.
I think other tests need this too - wherever we invoke expectationsForStaticAllocation(). Although, part of improving the integration tests would be to find something other than the UI to rely on for checking the correctness.
- Enable spark ui explicitly for in-process submit - Fixed some broken assertions in integration tests - Fixed a scalastyle error in SparkDockerImageBuilder.scala - Log into target/integration-tests.log like other modules
f46c19d to
221050c
Compare
|
ping @mccheah |
422dceb
into
apache-spark-on-k8s:k8s-support-alternate-incremental
ash211
pushed a commit
that referenced
this pull request
Feb 8, 2017
* Fixed k8s integration test - Enable spark ui explicitly for in-process submit - Fixed some broken assertions in integration tests - Fixed a scalastyle error in SparkDockerImageBuilder.scala - Log into target/integration-tests.log like other modules * Fixed line length. * CR
ash211
pushed a commit
that referenced
this pull request
Mar 8, 2017
* Fixed k8s integration test - Enable spark ui explicitly for in-process submit - Fixed some broken assertions in integration tests - Fixed a scalastyle error in SparkDockerImageBuilder.scala - Log into target/integration-tests.log like other modules * Fixed line length. * CR
foxish
pushed a commit
that referenced
this pull request
Jul 24, 2017
* Fixed k8s integration test - Enable spark ui explicitly for in-process submit - Fixed some broken assertions in integration tests - Fixed a scalastyle error in SparkDockerImageBuilder.scala - Log into target/integration-tests.log like other modules * Fixed line length. * CR
ifilonenko
pushed a commit
to ifilonenko/spark
that referenced
this pull request
Feb 25, 2019
…tVersion Bump parquet version
ifilonenko
pushed a commit
to ifilonenko/spark
that referenced
this pull request
Feb 25, 2019
* Fixed k8s integration test - Enable spark ui explicitly for in-process submit - Fixed some broken assertions in integration tests - Fixed a scalastyle error in SparkDockerImageBuilder.scala - Log into target/integration-tests.log like other modules * Fixed line length. * CR
puneetloya
pushed a commit
to puneetloya/spark
that referenced
this pull request
Mar 11, 2019
* Fixed k8s integration test - Enable spark ui explicitly for in-process submit - Fixed some broken assertions in integration tests - Fixed a scalastyle error in SparkDockerImageBuilder.scala - Log into target/integration-tests.log like other modules * Fixed line length. * CR
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
How was this patch tested?
Run the k8s integration tests manually. Before the patch the integration tests would fail even before the first case is executed. Now it finishes successfully with: