Skip to content
This repository has been archived by the owner on Dec 4, 2024. It is now read-only.

Commit

Permalink
Add limits page for Spark (#286)
Browse files Browse the repository at this point in the history
## Description
Addresses: [SPARK-691](https://jira.mesosphere.com/browse/SPARK-691)
This PR updates DC/OS Spark docs to report how much scale-testing we've done.

DC/OS Spark has been run on a 2.5K core cluster. 
The plan was to test on 10K cores but that was cost prohibitive.

## Urgency
- [x] Medium
  • Loading branch information
samvantran authored and susanxhuynh committed Apr 6, 2018
1 parent b63c777 commit ee608c9
Showing 1 changed file with 34 additions and 0 deletions.
34 changes: 34 additions & 0 deletions docs/limits.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
---
layout: layout.pug
navigationTitle:
excerpt:
title: Limits
menuWeight: 0
featureMaturity:

---

<!-- This source repo for this topic is https://github.com/mesosphere/dcos-commons -->

# DC/OS Spark Limits
Mesosphere has scale-tested Spark on DC/OS by running a CPU-bound Monte Carlo application on the following hardware:

## Cluster characteristics
- 2560 cores total
- 40 m4.16xlarge EC2 instances

### Single executor per node:
- 40 executors
- Each executor: 64 cores, 2GB memory
- CPU utilization was > 90%, with majority of time spent in task computation

### Multiple executors per node:
On a smaller, 1024-core, 16 node (m4.16xlarge) cluster, the following variations were tested:

| Executors | Time to Launch all Executors | Executors per Node |
| --------- | --------------------------- | ----------------- |
| 82 | 7 s. | 16 |
| 400 | 17 s. | 64 |
| 820 | 28 s. | 64 |

In all tests, the application completed successfully.

0 comments on commit ee608c9

Please sign in to comment.