Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-11667] Update dynamic allocation docs to reflect supported cluster managers #9637

Closed
wants to merge 2 commits into from
Closed

[SPARK-11667] Update dynamic allocation docs to reflect supported cluster managers #9637

wants to merge 2 commits into from

Conversation

andrewor14
Copy link
Contributor

No description provided.

@andrewor14
Copy link
Contributor Author

@tnachen please verify whether the mesos part is correct.

@SparkQA
Copy link

SparkQA commented Nov 11, 2015

Test build #45674 has finished for PR 9637 at commit 48d7418.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

tnachen
tnachen reviewed Nov 11, 2015
View reviewed changes
docs/job-scheduling.md
is implemented in `org.apache.spark.yarn.network.YarnShuffleService` that runs in each `NodeManager`
in your cluster. To start this service, follow these steps:
In Mesos coarse-grained mode, run `$SPARK_HOME/sbin/start-mesos-shuffle-service.sh` on all
slave nodes with `spark.shuffle.service.enabled` set to `true`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to add that users can run the mesos-shuffle-service.sh with Marathon, and they should start the service in the foreground running spark-class org.apache.spark.deploy.mesos.MesosExternalShuffleService

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure what you mean about the latter part. If you wish you can submit a patch against the same issue to augment the description here.

@tnachen
Copy link
Contributor

tnachen commented Nov 11, 2015

Only have one comment, otherwise everything else LGTM

@SparkQA
Copy link

SparkQA commented Nov 12, 2015

Test build #45767 has finished for PR 9637 at commit 8200bf5.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@tnachen
Copy link
Contributor

tnachen commented Nov 12, 2015

I think this is fine for now, I was thinking leaving some comments about how it should be launched with Marathon but I think later we can add a example json in the Marathon repo that we can link from here.

@tnachen
Copy link
Contributor

tnachen commented Nov 12, 2015

LGTM

@andrewor14
Copy link
Contributor Author

Thanks Tim, merging into master 1.6.

asfgit pushed a commit that referenced this pull request Nov 12, 2015
[SPARK-11667] Update dynamic allocation docs to reflect supported clu…
7828857 
…ster managers

Author: Andrew Or <andrew@databricks.com>

Closes #9637 from andrewor14/update-da-docs.

(cherry picked from commit 12a0784)
Signed-off-by: Andrew Or <andrew@databricks.com>
@asfgit asfgit closed this in 12a0784 Nov 12, 2015
@andrewor14 andrewor14 deleted the update-da-docs branch November 12, 2015 23:52
dskrvk pushed a commit to dskrvk/spark that referenced this pull request Nov 13, 2015
@dskrvk
[SPARK-11667] Update dynamic allocation docs to reflect supported clu…
62a7a68 
…ster managers

Author: Andrew Or <andrew@databricks.com>

Closes apache#9637 from andrewor14/update-da-docs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@andrewor14 @SparkQA @tnachen