[SPARK-7627][SPARK-7472] DAG visualization: style skipped stages #6171

andrewor14 · 2015-05-15T04:31:35Z

This patch fixes two things:

SPARK-7627. Cached RDDs no longer light up on the job page. This is a simple fix.
SPARK-7472. Display skipped stages differently from normal stages.

The latter is a major UX issue. Because we link the job viz to the stage viz even for skipped stages, the user may inadvertently click into the stage page of a skipped stage, which is empty.

This requires storing more state in the listener and making sure we clean them up later. Tests are coming in the next commit.

Previously there was an opportunity to leak in `completedStageIds`. This commit fixes that and includes tests that ensures that it is fixed.

AmplabJenkins · 2015-05-15T04:32:09Z

Merged build triggered.

andrewor14 · 2015-05-15T04:32:15Z

@zsxwing it would be good if you could verify that the listener clean up logic is correct. I have added tests for that purpose but we should test it out in a real long-running streaming application.

AmplabJenkins · 2015-05-15T04:32:19Z

Merged build started.

andrewor14 · 2015-05-15T04:34:04Z

Also cc @mateiz who suggested this change.

SparkQA · 2015-05-15T04:34:13Z

Test build #32773 has started for PR 6171 at commit b928cd4.

zsxwing · 2015-05-15T06:07:52Z

Testing your PR. Let's see if there is any memory leak tomorrow morning.

SparkQA · 2015-05-15T06:16:54Z

Test build #32773 has finished for PR 6171 at commit b928cd4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-15T06:16:59Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-15T06:16:59Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32773/
Test PASSed.

Conflicts: core/src/main/resources/org/apache/spark/ui/static/spark-dag-viz.js

AmplabJenkins · 2015-05-15T20:27:12Z

Merged build triggered.

AmplabJenkins · 2015-05-15T20:27:21Z

Merged build started.

SparkQA · 2015-05-15T20:29:25Z

Test build #32849 has started for PR 6171 at commit 51c95b9.

AmplabJenkins · 2015-05-15T21:12:11Z

Merged build triggered.

AmplabJenkins · 2015-05-15T21:12:21Z

Merged build started.

SparkQA · 2015-05-15T21:13:12Z

Test build #32851 has started for PR 6171 at commit dc32adb.

Right now stage clusters assume the ID space of integers. This is not valid, however, after we merge with dag-viz-streaming, where the cluster ID may just be the operation ID. The result is that certain stage clusters disappear on the UI. This patch by itself doesn't fix anything noticeable. However, it does guard against the potential of collision in future changes.

AmplabJenkins · 2015-05-15T21:17:11Z

Merged build triggered.

AmplabJenkins · 2015-05-15T21:17:16Z

Merged build started.

SparkQA · 2015-05-15T21:17:37Z

Test build #32852 has started for PR 6171 at commit 762b541.

SparkQA · 2015-05-15T22:17:24Z

Test build #32849 has finished for PR 6171 at commit 51c95b9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-15T22:17:29Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-15T22:17:30Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32849/
Test PASSed.

SparkQA · 2015-05-15T22:31:27Z

Test build #32852 has finished for PR 6171 at commit 762b541.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-05-15T23:02:01Z

Test build #32851 has finished for PR 6171 at commit dc32adb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-15T23:02:07Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-15T23:02:08Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32851/
Test PASSed.

mateiz · 2015-05-16T05:46:45Z

A small note on the colors -- make them a grayscale version of the colors used in the normal stages, because right now they're quite a bit darker. Just take a picture of the normal stages and turn it to grayscale to grab the right colors.

mateiz · 2015-05-16T05:47:50Z

Alternatively you could also make them lighter and use dashed lines (http://www.w3schools.com/svg/svg_stroking.asp), but I'm not sure how good that will look.

andrewor14 · 2015-05-16T23:39:32Z

Ok, I've updated the screenshot. I tried out the dotted lines but I thought the gray looks better.

AmplabJenkins · 2015-05-16T23:42:10Z

Merged build triggered.

AmplabJenkins · 2015-05-16T23:42:17Z

Merged build started.

SparkQA · 2015-05-16T23:44:08Z

Test build #32915 has started for PR 6171 at commit c604150.

SparkQA · 2015-05-17T01:36:02Z

Test build #32915 has finished for PR 6171 at commit c604150.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-17T01:36:07Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-17T01:36:08Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32915/
Test PASSed.

mateiz · 2015-05-18T05:05:49Z

Cool, this looks pretty good. Maybe I'd make the border on skipped stages slightly darker since the red is darker than blue on the other ones, but overall it's fine.

AmplabJenkins · 2015-05-18T06:17:12Z

Merged build triggered.

AmplabJenkins · 2015-05-18T06:17:20Z

Merged build started.

SparkQA · 2015-05-18T06:17:43Z

Test build #32973 has started for PR 6171 at commit f261797.

SparkQA · 2015-05-18T08:08:01Z

Test build #32973 has finished for PR 6171 at commit f261797.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-18T08:08:06Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-18T08:08:06Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32973/
Test PASSed.

andrewor14 · 2015-05-18T17:58:37Z

I discussed with @zsxwing offline and he found that the memory usage is stable for long-running jobs. I'm merging this into master and 1.4. Thanks for all the feedback.

This patch fixes two things: **SPARK-7627.** Cached RDDs no longer light up on the job page. This is a simple fix. **SPARK-7472.** Display skipped stages differently from normal stages. The latter is a major UX issue. Because we link the job viz to the stage viz even for skipped stages, the user may inadvertently click into the stage page of a skipped stage, which is empty. ------------------- <img src="https://cloud.githubusercontent.com/assets/2133137/7675241/de1a3da6-fcea-11e4-8101-88055cef78c5.png" width="300px" /> Author: Andrew Or <andrew@databricks.com> Closes #6171 from andrewor14/dag-viz-skipped and squashes the following commits: f261797 [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-skipped 0eda358 [Andrew Or] Tweak skipped stage border color c604150 [Andrew Or] Tweak grayscale colors 7010676 [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-skipped 762b541 [Andrew Or] Use special prefix for stage clusters to avoid collisions 51c95b9 [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-skipped b928cd4 [Andrew Or] Fix potential leak + write tests for it 7c4c364 [Andrew Or] Show skipped stages differently 7cc34ce [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-skipped c121fa2 [Andrew Or] Fix cache color (cherry picked from commit 563bfcc) Signed-off-by: Andrew Or <andrew@databricks.com>

This patch fixes two things: **SPARK-7627.** Cached RDDs no longer light up on the job page. This is a simple fix. **SPARK-7472.** Display skipped stages differently from normal stages. The latter is a major UX issue. Because we link the job viz to the stage viz even for skipped stages, the user may inadvertently click into the stage page of a skipped stage, which is empty. ------------------- <img src="https://cloud.githubusercontent.com/assets/2133137/7675241/de1a3da6-fcea-11e4-8101-88055cef78c5.png" width="300px" /> Author: Andrew Or <andrew@databricks.com> Closes apache#6171 from andrewor14/dag-viz-skipped and squashes the following commits: f261797 [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-skipped 0eda358 [Andrew Or] Tweak skipped stage border color c604150 [Andrew Or] Tweak grayscale colors 7010676 [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-skipped 762b541 [Andrew Or] Use special prefix for stage clusters to avoid collisions 51c95b9 [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-skipped b928cd4 [Andrew Or] Fix potential leak + write tests for it 7c4c364 [Andrew Or] Show skipped stages differently 7cc34ce [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-skipped c121fa2 [Andrew Or] Fix cache color

Andrew Or added 4 commits May 14, 2015 01:06

Fix cache color

c121fa2

Merge branch 'master' of github.com:apache/spark into dag-viz-skipped

7cc34ce

Show skipped stages differently

7c4c364

This requires storing more state in the listener and making sure we clean them up later. Tests are coming in the next commit.

Fix potential leak + write tests for it

b928cd4

Previously there was an opportunity to leak in `completedStageIds`. This commit fixes that and includes tests that ensures that it is fixed.

andrewor14 mentioned this pull request May 15, 2015

[SPARK-7664][WebUI] DAG visualization: Fix incorrect link paths of DAG. #6184

Closed

Merge branch 'master' of github.com:apache/spark into dag-viz-skipped

51c95b9

Conflicts: core/src/main/resources/org/apache/spark/ui/static/spark-dag-viz.js

andrewor14 force-pushed the dag-viz-skipped branch from dc32adb to 762b541 Compare May 15, 2015 21:15

Andrew Or added 2 commits May 16, 2015 16:29

Merge branch 'master' of github.com:apache/spark into dag-viz-skipped

7010676

Tweak grayscale colors

c604150

Andrew Or added 2 commits May 17, 2015 23:16

Tweak skipped stage border color

0eda358

Merge branch 'master' of github.com:apache/spark into dag-viz-skipped

f261797

asfgit closed this in 563bfcc May 18, 2015

andrewor14 deleted the dag-viz-skipped branch May 18, 2015 18:35

[SPARK-7627][SPARK-7472] DAG visualization: style skipped stages #6171

[SPARK-7627][SPARK-7472] DAG visualization: style skipped stages #6171

Conversation

andrewor14 commented May 15, 2015

AmplabJenkins commented May 15, 2015

andrewor14 commented May 15, 2015

AmplabJenkins commented May 15, 2015

andrewor14 commented May 15, 2015

SparkQA commented May 15, 2015

zsxwing commented May 15, 2015

SparkQA commented May 15, 2015

AmplabJenkins commented May 15, 2015

AmplabJenkins commented May 15, 2015

AmplabJenkins commented May 15, 2015

AmplabJenkins commented May 15, 2015

SparkQA commented May 15, 2015

AmplabJenkins commented May 15, 2015

AmplabJenkins commented May 15, 2015

SparkQA commented May 15, 2015

AmplabJenkins commented May 15, 2015

AmplabJenkins commented May 15, 2015

SparkQA commented May 15, 2015

SparkQA commented May 15, 2015

AmplabJenkins commented May 15, 2015

AmplabJenkins commented May 15, 2015

SparkQA commented May 15, 2015

SparkQA commented May 15, 2015

AmplabJenkins commented May 15, 2015

AmplabJenkins commented May 15, 2015

mateiz commented May 16, 2015

mateiz commented May 16, 2015

andrewor14 commented May 16, 2015

AmplabJenkins commented May 16, 2015

AmplabJenkins commented May 16, 2015

SparkQA commented May 16, 2015

SparkQA commented May 17, 2015

AmplabJenkins commented May 17, 2015

AmplabJenkins commented May 17, 2015

mateiz commented May 18, 2015

AmplabJenkins commented May 18, 2015

AmplabJenkins commented May 18, 2015

SparkQA commented May 18, 2015

SparkQA commented May 18, 2015

AmplabJenkins commented May 18, 2015

AmplabJenkins commented May 18, 2015

andrewor14 commented May 18, 2015