Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-11667] Update dynamic allocation docs to reflect supported cluster managers #9637

Closed
wants to merge 2 commits into from
Closed

[SPARK-11667] Update dynamic allocation docs to reflect supported cluster managers #9637

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
48d7418
Update dynamic allocation docs
Nov 11, 2015
8200bf5
Mention marathon
Nov 12, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 26 additions & 28 deletions docs/job-scheduling.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,36 +56,31 @@ provide another approach to share RDDs.

## Dynamic Resource Allocation

Spark 1.2 introduces the ability to dynamically scale the set of cluster resources allocated to
your application up and down based on the workload. This means that your application may give
resources back to the cluster if they are no longer used and request them again later when there
is demand. This feature is particularly useful if multiple applications share resources in your
Spark cluster. If a subset of the resources allocated to an application becomes idle, it can be
returned to the cluster's pool of resources and acquired by other applications. In Spark, dynamic
resource allocation is performed on the granularity of the executor and can be enabled through
`spark.dynamicAllocation.enabled`.

This feature is currently disabled by default and available only on [YARN](running-on-yarn.html).
A future release will extend this to [standalone mode](spark-standalone.html) and
[Mesos coarse-grained mode](running-on-mesos.html#mesos-run-modes). Note that although Spark on
Mesos already has a similar notion of dynamic resource sharing in fine-grained mode, enabling
dynamic allocation allows your Mesos application to take advantage of coarse-grained low-latency
scheduling while sharing cluster resources efficiently.
Spark provides a mechanism to dynamically adjust the resources your application occupies based
on the workload. This means that your application may give resources back to the cluster if they
are no longer used and request them again later when there is demand. This feature is particularly
useful if multiple applications share resources in your Spark cluster.

This feature is disabled by default and available on all coarse-grained cluster managers, i.e.
[standalone mode](spark-standalone.html), [YARN mode](running-on-yarn.html), and
[Mesos coarse-grained mode](running-on-mesos.html#mesos-run-modes).

### Configuration and Setup

All configurations used by this feature live under the `spark.dynamicAllocation.*` namespace.
To enable this feature, your application must set `spark.dynamicAllocation.enabled` to `true`.
Other relevant configurations are described on the
[configurations page](configuration.html#dynamic-allocation) and in the subsequent sections in
detail.
There are two requirements for using this feature. First, your application must set
`spark.dynamicAllocation.enabled` to `true`. Second, you must set up an *external shuffle service*
on each worker node in the same cluster and set `spark.shuffle.service.enabled` to true in your
application. The purpose of the external shuffle service is to allow executors to be removed
without deleting shuffle files written by them (more detail described
[below](job-scheduling.html#graceful-decommission-of-executors)). The way to set up this service
varies across cluster managers:

In standalone mode, simply start your workers with `spark.shuffle.service.enabled` set to `true`.

Additionally, your application must use an external shuffle service. The purpose of the service is
to preserve the shuffle files written by executors so the executors can be safely removed (more
detail described [below](job-scheduling.html#graceful-decommission-of-executors)). To enable
this service, set `spark.shuffle.service.enabled` to `true`. In YARN, this external shuffle service
is implemented in `org.apache.spark.yarn.network.YarnShuffleService` that runs in each `NodeManager`
in your cluster. To start this service, follow these steps:
In Mesos coarse-grained mode, run `$SPARK_HOME/sbin/start-mesos-shuffle-service.sh` on all
slave nodes with `spark.shuffle.service.enabled` set to `true`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to add that users can run the mesos-shuffle-service.sh with Marathon, and they should start the service in the foreground running spark-class org.apache.spark.deploy.mesos.MesosExternalShuffleService

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure what you mean about the latter part. If you wish you can submit a patch against the same issue to augment the description here.


In YARN mode, start the shuffle service on each `NodeManager` as follows:

1. Build Spark with the [YARN profile](building-spark.html). Skip this step if you are using a
pre-packaged distribution.
Expand All @@ -95,10 +90,13 @@ pre-packaged distribution.
2. Add this jar to the classpath of all `NodeManager`s in your cluster.
3. In the `yarn-site.xml` on each node, add `spark_shuffle` to `yarn.nodemanager.aux-services`,
then set `yarn.nodemanager.aux-services.spark_shuffle.class` to
`org.apache.spark.network.yarn.YarnShuffleService`. Additionally, set all relevant
`spark.shuffle.service.*` [configurations](configuration.html).
`org.apache.spark.network.yarn.YarnShuffleService` and `spark.shuffle.service.enabled` to true.
4. Restart all `NodeManager`s in your cluster.

All other relevant configurations are optional and under the `spark.dynamicAllocation.*` and
`spark.shuffle.service.*` namespaces. For more detail, see the
[configurations page](configuration.html#dynamic-allocation).

### Resource Allocation Policy

At a high level, Spark should relinquish executors when they are no longer used and acquire
Expand Down