[SPARK-3797] Run external shuffle service in Yarn NM #3082

andrewor14 · 2014-11-04T00:39:43Z

This creates a new module network/yarn that depends on network/shuffle recently created in #3001. This PR introduces a custom Yarn auxiliary service that runs the external shuffle service. As of the changes here this shuffle service is required for using dynamic allocation with Spark.

I have tested this on a stable Yarn cluster.

This includes the necessary changes in making the Yarn shuffle service a module of Spark that core doesn't depend on. This is included in the network-yarn module, which depends on the network-shuffle module.

SparkQA · 2014-11-04T00:42:30Z

Test build #22835 has started for PR 3082 at commit 804e7ff.

This patch merges cleanly.

SparkQA · 2014-11-04T00:42:35Z

Test build #22835 has finished for PR 3082 at commit 804e7ff.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class ExecutorLostFailure(execId: String) extends TaskFailedReason
- public class YarnShuffleService extends AuxiliaryService
- class DecimalType(DataType):
- // in some cases, such as when a class is enclosed in an object (in which case
- case class UnscaledValue(child: Expression) extends UnaryExpression
- case class MakeDecimal(child: Expression, precision: Int, scale: Int) extends UnaryExpression
- case class MutableLiteral(var value: Any, dataType: DataType, nullable: Boolean = true)
- abstract class GenericStrategy[PhysicalPlan <: TreeNode[PhysicalPlan]] extends Logging
- case class PrecisionInfo(precision: Int, scale: Int)
- case class DecimalType(precisionInfo: Option[PrecisionInfo]) extends FractionalType
- abstract class UserDefinedType[UserType] extends DataType with Serializable
- final class Decimal extends Ordered[Decimal] with Serializable
- trait DecimalIsConflicted extends Numeric[Decimal]
- public abstract class UserDefinedType<UserType> extends DataType implements Serializable
- trait RunnableCommand extends logical.Command
- case class ExecutedCommand(cmd: RunnableCommand) extends SparkPlan
- protected case class Keyword(str: String)
- sys.error(s"Failed to load class for data source: $provider")
- case class EqualTo(attribute: String, value: Any) extends Filter
- case class GreaterThan(attribute: String, value: Any) extends Filter
- case class GreaterThanOrEqual(attribute: String, value: Any) extends Filter
- case class LessThan(attribute: String, value: Any) extends Filter
- case class LessThanOrEqual(attribute: String, value: Any) extends Filter
- trait RelationProvider
- abstract class BaseRelation
- abstract class TableScan extends BaseRelation
- abstract class PrunedScan extends BaseRelation
- abstract class PrunedFilteredScan extends BaseRelation

AmplabJenkins · 2014-11-04T00:42:36Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22835/
Test FAILed.

SparkQA · 2014-11-04T00:47:38Z

Test build #22837 has started for PR 3082 at commit 5b419b8.

This patch merges cleanly.

vanzin · 2014-11-04T00:54:13Z

core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala

+    if (executorIdleTimeout <= 0) {
+      throw new SparkException(s"spark.dynamicAllocation.executorIdleTimeout must be > 0!")
+    }
+    // Verify that external shuffle service is enabled


nit: comment is redundant given the code.

vanzin · 2014-11-04T01:09:13Z

LGTM aside some nits; I still have to catch up with the rest of the network code changes to make more informed comments, though.

vanzin · 2014-11-04T01:10:40Z

pom.xml

@@ -93,6 +93,7 @@
    <module>tools</module>
    <module>network/common</module>
    <module>network/shuffle</module>
+    <module>network/yarn</module>


BTW, this should be probably gated on some profile (e.g. "-Pyarn"). I don't think this would compile with Hadoop 1.0.4, for example...

Yeah... good point

SparkQA · 2014-11-04T01:47:29Z

Test build #22843 has started for PR 3082 at commit 5bf9b7e.

This patch merges cleanly.

SparkQA · 2014-11-04T01:50:24Z

Test build #22837 has finished for PR 3082 at commit 5b419b8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class YarnShuffleService extends AuxiliaryService

AmplabJenkins · 2014-11-04T01:50:27Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22837/
Test FAILed.

…rvice

SparkQA · 2014-11-04T02:04:55Z

Test build #22848 has started for PR 3082 at commit 761f58a.

This patch merges cleanly.

Currently the user will have to add all three network jars to the NM class path. A future PR will use a maven plugin to merge these into one.

SparkQA · 2014-11-04T02:45:12Z

Test build #22854 has started for PR 3082 at commit f39daa6.

This patch merges cleanly.

SparkQA · 2014-11-04T02:57:42Z

Test build #22843 has finished for PR 3082 at commit 5bf9b7e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

andrewor14 · 2014-11-05T07:56:11Z

Here's a quick status update on this PR. As of the latest commit, the Yarn shuffle service now handles authentication. However, it is not working yet because of two known issues (see commit message andrewor14@d1124e4 for more detail). These are being actively fixed by @aarondav at the moment and this PR will be blocked until that one is merged (#3108). I have tested the tentative changes there and verified that the service functions as expected with authentication with those changes.

SparkQA · 2014-11-05T09:21:35Z

Test build #22922 has finished for PR 3082 at commit d1124e4.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class ShuffleSecretManager implements SecretKeyHolder
- public class YarnShuffleService extends AuxiliaryService
- public class HadoopConfigProvider extends ConfigProvider

AmplabJenkins · 2014-11-05T09:21:39Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22922/
Test PASSed.

On service init, we want to fail fast if the server fails to start for any reason (e.g. port bind exception). However, once the NM has been running for a while, it should be super robust so we need to add try catches around the application initialization / stopping code.

pwendell · 2014-11-05T21:52:02Z

yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala

@@ -89,6 +90,20 @@ class ExecutorRunnable(

    ctx.setApplicationACLs(YarnSparkHadoopUtil.getApplicationAclsForYarn(securityMgr))

+    // If external shuffle service is enabled, register with the


It might be good to say "register and transfer keys" rather than just register

…rvice

SparkQA · 2014-11-05T21:59:58Z

Test build #22950 has started for PR 3082 at commit 0eb6233.

This patch merges cleanly.

pwendell · 2014-11-05T22:06:24Z

network/shuffle/src/main/java/org/apache/spark/network/sasl/ShuffleSecretManager.java

+  private static final Charset UTF8_CHARSET = Charset.forName("UTF-8");
+
+  // Spark user used for authenticating SASL connections
+  // Note that this should match the value in org.apache.spark.SecurityManager


"this must match"

pwendell · 2014-11-05T22:19:56Z

This seems quite straightforward. LGTM.

…rvice

andrewor14 · 2014-11-05T22:57:10Z

Alright, as of the latest commit the two blocking fixes previously mentioned are merged. I think I have addressed all the comments. This is ready to go on my side.

SparkQA · 2014-11-05T22:59:53Z

Test build #22954 has started for PR 3082 at commit ef3ddae.

This patch merges cleanly.

SparkQA · 2014-11-05T23:30:20Z

Test build #22950 has finished for PR 3082 at commit 0eb6233.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public final class JavaGradientBoostedTrees
- case class Params(
- public class ShuffleSecretManager implements SecretKeyHolder
- public class YarnShuffleService extends AuxiliaryService
- public class HadoopConfigProvider extends ConfigProvider

AmplabJenkins · 2014-11-05T23:30:23Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22950/
Test PASSed.

andrewor14 · 2014-11-05T23:41:40Z

Ok I'm merging this into master and 1.2. Thanks everyone.

This creates a new module `network/yarn` that depends on `network/shuffle` recently created in #3001. This PR introduces a custom Yarn auxiliary service that runs the external shuffle service. As of the changes here this shuffle service is required for using dynamic allocation with Spark. This is still WIP mainly because it doesn't handle security yet. I have tested this on a stable Yarn cluster. Author: Andrew Or <andrew@databricks.com> Closes #3082 from andrewor14/yarn-shuffle-service and squashes the following commits: ef3ddae [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-shuffle-service 0ee67a2 [Andrew Or] Minor wording suggestions 1c66046 [Andrew Or] Remove unused provided dependencies 0eb6233 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-shuffle-service 6489db5 [Andrew Or] Try catch at the right places 7b71d8f [Andrew Or] Add detailed java docs + reword a few comments d1124e4 [Andrew Or] Add security to shuffle service (INCOMPLETE) 5f8a96f [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-shuffle-service 9b6e058 [Andrew Or] Address various feedback f48b20c [Andrew Or] Fix tests again f39daa6 [Andrew Or] Do not make network-yarn an assembly module 761f58a [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-shuffle-service 15a5b37 [Andrew Or] Fix build for Hadoop 1.x baff916 [Andrew Or] Fix tests 5bf9b7e [Andrew Or] Address a few minor comments 5b419b8 [Andrew Or] Add missing license header 804e7ff [Andrew Or] Include the Yarn shuffle service jar in the distribution cd076a4 [Andrew Or] Require external shuffle service for dynamic allocation ea764e0 [Andrew Or] Connect to Yarn shuffle service only if it's enabled 1bf5109 [Andrew Or] Use the shuffle service port specified through hadoop config b4b1f0c [Andrew Or] 4 tabs -> 2 tabs 43dcb96 [Andrew Or] First cut integration of shuffle service with Yarn aux service b54a0c4 [Andrew Or] Initial skeleton for Yarn shuffle service (cherry picked from commit 61a5cce) Signed-off-by: Andrew Or <andrew@databricks.com>

SparkQA · 2014-11-06T00:24:53Z

Test build #22954 has finished for PR 3082 at commit ef3ddae.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public final class JavaGradientBoostedTrees
- case class Params(
- public final class LimitedInputStream extends FilterInputStream
- public class ShuffleSecretManager implements SecretKeyHolder
- public class YarnShuffleService extends AuxiliaryService
- public class HadoopConfigProvider extends ConfigProvider
- class Matrices(object):
- class ChiSqTestResult(JavaModelWrapper):
- case class AllocatedBlocks(streamIdToAllocatedBlocks: Map[Int, Seq[ReceivedBlockInfo]])
- class ReceiverTracker(ssc: StreamingContext, skipReceiverLaunch: Boolean = false) extends Logging

AmplabJenkins · 2014-11-06T00:24:57Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22954/
Test PASSed.

This is another addendum to apache#3082, which added the Yarn shuffle service to run inside the NM. This PR makes the feature much more usable by packaging enough dependencies into the jar to run the service inside an NM. After these changes, the user can run `./make-distribution.sh` and find a `spark-network-yarn*.jar` in their `lib` directory. The equivalent change is done in SBT by making the `network-yarn` module an assembly project. Author: Andrew Or <andrew@databricks.com> Closes apache#3147 from andrewor14/yarn-shuffle-build and squashes the following commits: bda58d0 [Andrew Or] Fix line too long 81e9705 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-shuffle-build fb7f398 [Andrew Or] Rename jar to spark-{VERSION}-yarn-shuffle.jar 65db822 [Andrew Or] Actually mark slf4j as provided abcefd1 [Andrew Or] Do the same for SBT c653028 [Andrew Or] Package network-yarn and its dependencies

This is another addendum to #3082, which added the Yarn shuffle service to run inside the NM. This PR makes the feature much more usable by packaging enough dependencies into the jar to run the service inside an NM. After these changes, the user can run `./make-distribution.sh` and find a `spark-network-yarn*.jar` in their `lib` directory. The equivalent change is done in SBT by making the `network-yarn` module an assembly project. Author: Andrew Or <andrew@databricks.com> Closes #3147 from andrewor14/yarn-shuffle-build and squashes the following commits: bda58d0 [Andrew Or] Fix line too long 81e9705 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-shuffle-build fb7f398 [Andrew Or] Rename jar to spark-{VERSION}-yarn-shuffle.jar 65db822 [Andrew Or] Actually mark slf4j as provided abcefd1 [Andrew Or] Do the same for SBT c653028 [Andrew Or] Package network-yarn and its dependencies (cherry picked from commit aa43a8d) Signed-off-by: Andrew Or <andrew@databricks.com>

Andrew Or added 7 commits November 3, 2014 13:43

Initial skeleton for Yarn shuffle service

b54a0c4

This includes the necessary changes in making the Yarn shuffle service a module of Spark that core doesn't depend on. This is included in the network-yarn module, which depends on the network-shuffle module.

First cut integration of shuffle service with Yarn aux service

43dcb96

4 tabs -> 2 tabs

b4b1f0c

Use the shuffle service port specified through hadoop config

1bf5109

Connect to Yarn shuffle service only if it's enabled

ea764e0

Require external shuffle service for dynamic allocation

cd076a4

Include the Yarn shuffle service jar in the distribution

804e7ff

Add missing license header

5b419b8

vanzin reviewed Nov 4, 2014
View reviewed changes

Address a few minor comments

5bf9b7e

Andrew Or added 3 commits November 3, 2014 17:53

Fix tests

baff916

Fix build for Hadoop 1.x

15a5b37

Merge branch 'master' of github.com:apache/spark into yarn-shuffle-se…

761f58a

…rvice

Do not make network-yarn an assembly module

f39daa6

Currently the user will have to add all three network jars to the NM class path. A future PR will use a maven plugin to merge these into one.

Andrew Or added 2 commits November 5, 2014 13:41

Add detailed java docs + reword a few comments

7b71d8f

pwendell reviewed Nov 5, 2014
View reviewed changes

Merge branch 'master' of github.com:apache/spark into yarn-shuffle-se…

0eb6233

…rvice

pwendell reviewed Nov 5, 2014
View reviewed changes

Andrew Or added 3 commits November 5, 2014 14:22

Remove unused provided dependencies

1c66046

Minor wording suggestions

0ee67a2

Merge branch 'master' of github.com:apache/spark into yarn-shuffle-se…

ef3ddae

…rvice

andrewor14 changed the title ~~[WIP][SPARK-3797] Run external shuffle service in Yarn NM~~ [SPARK-3797] Run external shuffle service in Yarn NM Nov 5, 2014

asfgit closed this in 61a5cce Nov 5, 2014

andrewor14 deleted the yarn-shuffle-service branch November 5, 2014 23:47

andrewor14 mentioned this pull request Nov 7, 2014

[SPARK-4281][Build] Package Yarn shuffle service into its own jar #3147

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-3797] Run external shuffle service in Yarn NM #3082

[SPARK-3797] Run external shuffle service in Yarn NM #3082

andrewor14 commented Nov 4, 2014

SparkQA commented Nov 4, 2014

SparkQA commented Nov 4, 2014

AmplabJenkins commented Nov 4, 2014

SparkQA commented Nov 4, 2014

vanzin Nov 4, 2014

vanzin commented Nov 4, 2014

vanzin Nov 4, 2014

andrewor14 Nov 4, 2014

SparkQA commented Nov 4, 2014

SparkQA commented Nov 4, 2014

AmplabJenkins commented Nov 4, 2014

SparkQA commented Nov 4, 2014

SparkQA commented Nov 4, 2014

SparkQA commented Nov 4, 2014

andrewor14 commented Nov 5, 2014

SparkQA commented Nov 5, 2014

AmplabJenkins commented Nov 5, 2014

pwendell Nov 5, 2014

SparkQA commented Nov 5, 2014

pwendell Nov 5, 2014

pwendell commented Nov 5, 2014

andrewor14 commented Nov 5, 2014

SparkQA commented Nov 5, 2014

SparkQA commented Nov 5, 2014

AmplabJenkins commented Nov 5, 2014

andrewor14 commented Nov 5, 2014

SparkQA commented Nov 6, 2014

AmplabJenkins commented Nov 6, 2014

		@@ -89,6 +90,20 @@ class ExecutorRunnable(

		ctx.setApplicationACLs(YarnSparkHadoopUtil.getApplicationAclsForYarn(securityMgr))

		// If external shuffle service is enabled, register with the

[SPARK-3797] Run external shuffle service in Yarn NM #3082

[SPARK-3797] Run external shuffle service in Yarn NM #3082

Conversation

andrewor14 commented Nov 4, 2014

SparkQA commented Nov 4, 2014

SparkQA commented Nov 4, 2014

AmplabJenkins commented Nov 4, 2014

SparkQA commented Nov 4, 2014

vanzin Nov 4, 2014

Choose a reason for hiding this comment

vanzin commented Nov 4, 2014

vanzin Nov 4, 2014

Choose a reason for hiding this comment

andrewor14 Nov 4, 2014

Choose a reason for hiding this comment

SparkQA commented Nov 4, 2014

SparkQA commented Nov 4, 2014

AmplabJenkins commented Nov 4, 2014

SparkQA commented Nov 4, 2014

SparkQA commented Nov 4, 2014

SparkQA commented Nov 4, 2014

andrewor14 commented Nov 5, 2014

SparkQA commented Nov 5, 2014

AmplabJenkins commented Nov 5, 2014

pwendell Nov 5, 2014

Choose a reason for hiding this comment

SparkQA commented Nov 5, 2014

pwendell Nov 5, 2014

Choose a reason for hiding this comment

pwendell commented Nov 5, 2014

andrewor14 commented Nov 5, 2014

SparkQA commented Nov 5, 2014

SparkQA commented Nov 5, 2014

AmplabJenkins commented Nov 5, 2014

andrewor14 commented Nov 5, 2014

SparkQA commented Nov 6, 2014

AmplabJenkins commented Nov 6, 2014