SPARK-2250: show stage RDDs in UI #1188

nevillelyh · 2014-06-24T02:47:01Z

No description provided.

AmplabJenkins · 2014-06-24T02:50:13Z

Can one of the admins verify this patch?

andrewor14 · 2014-06-25T21:02:39Z

Hey @nevillelyh, can you attach a screenshot or two?

nevillelyh · 2014-06-26T14:10:49Z

Screenshot for the proposed change:

nevillelyh · 2014-07-01T02:58:27Z

@andrewor14 I resolved merge conflicts and posted screenshot above.

nevillelyh · 2014-07-16T01:50:21Z

@andrewor14 ping
rebased again.

andrewor14 · 2014-07-16T02:39:59Z

Jenkins, test this please

SparkQA · 2014-07-16T02:42:53Z

QA tests have started for PR 1188. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16700/consoleFull

SparkQA · 2014-07-16T02:43:34Z

QA results for PR 1188:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16700/consoleFull

nevillelyh · 2014-07-16T02:53:47Z

Fixed scalastyle warning. I thought I was whitelisted in AmplabJenkins...

andrewor14 · 2014-07-16T20:07:27Z

Jenkins, test this please

SparkQA · 2014-07-16T20:13:00Z

QA tests have started for PR 1188. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16744/consoleFull

SparkQA · 2014-07-16T20:13:38Z

QA results for PR 1188:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16744/consoleFull

nevillelyh · 2014-07-16T20:18:14Z

Sorry missed one. Fixed the style warning.

nevillelyh · 2014-07-22T05:06:39Z

@andrewor14 ping again?
Rebased again and scalastyle/tests passed locally.

andrewor14 · 2014-07-22T19:12:12Z

Hi @nevillelyh, this seems a little too much information, and the use cases aren't immediately clear to me. If we show only persisted RDDs, however, that might be helpful to some people, but even then we still need to run the idea by other people to see if we want this feature.

andrewor14 · 2014-07-22T19:12:17Z

add to whitelist

nevillelyh · 2014-07-22T19:14:43Z

@andrewor14 it's useful in apps with many iterations, i.e. ALS. I can filter out RDDs with (default) numeric names or 0 cached partitions if that make more sense?

SparkQA · 2014-07-22T19:18:27Z

QA tests have started for PR 1188. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16982/consoleFull

SparkQA · 2014-07-22T20:59:11Z

QA results for PR 1188:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16982/consoleFull

pwendell · 2014-07-22T22:17:12Z

@nevillelyh As a first step it would be interesting to see a version that only includes persisted RDD's. In that case the RDD name could link to the relevant page in the storage tab page. I'm not 100% sure on this, but seems like that might be useful.

nevillelyh · 2014-07-23T00:42:51Z

I revised it so that the "+show RDDs" link is shown only when there are cached RDDs (with links). Hope this is more useful.

SparkQA · 2014-07-23T00:43:50Z

QA tests have started for PR 1188. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17000/consoleFull

SparkQA · 2014-07-23T02:22:39Z

QA results for PR 1188:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17000/consoleFull

pwendell · 2014-07-24T04:55:00Z

Sorry I should have said this earlier, but given that there can be only one cached RDD per stage, what about just saying "Persisted RDD: XX"? Having a drop down isn't necessary any more. That will also make this diff a lot smaller I think, which is an added bonus!

nevillelyh · 2014-07-24T05:00:51Z

Sounds good, I'll revert back to "show details" and include RDD in drop down.

pwendell · 2014-07-24T05:05:11Z

@nevillelyh I like "show call stack" actually (it's more accurate). So fine if you keep that. Or maybe it could even just say "call stack". Anyways fine either way.

pwendell · 2014-07-24T05:59:31Z

core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala

    }

    val stageDataOption = listener.stageIdToData.get(s.stageId)
    // Too many nested map/flatMaps with options are just annoying to read. Do this imperatively.
    if (stageDataOption.isDefined && stageDataOption.get.description.isDefined) {
      val desc = stageDataOption.get.description
-      <div><em>{desc}</em></div><div>{nameLink} {killLink}</div>
+      <div><em>{desc}</em></div><div>{killLink} {nameLink} {details}</div>


Is this a separate fix here? (looks like we didn't correctly add details in this case)

Yeah I reused SPARK-2035, the ticket that introduced "details" tab.

ah okay cool

pwendell · 2014-07-24T06:48:04Z

core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala

+        {if (cachedRddInfos.nonEmpty) {
+          <strong>Persisted RDD:</strong> ++
+          cachedRddInfos.map { i =>
+            <a href={"%s/storage/rdd?id=%d".format(UIUtils.prependBaseUri(basePath), i.id)}>


I tested this locally and it would be good to pull this onto one line, because otherwise the hyperlink renders with a space at the beginning. If you need to violate the style guidelines you can disable scalastyle using //scalastyle off and //scalastyle on (see where we do this elsewhere in the UI code).

pwendell · 2014-07-24T06:52:57Z

Built and tested it locally. I was actually proposing pulling the Persisted RDD part outside of the toggle component. I think it's fine either way, but if we decide to put it in the toggle box, I agree the name should be more generically "details". If we decide to pull it outside of the toggle box, then we can leave the box as "stack trace". I slightly prefer leaving it out, but maybe if the name is really long that will be annoying? I'm worried people will never see it if it's hidden in the toggle.

pwendell · 2014-07-24T06:56:34Z

One more thing - after playing in my browser I think just saying "RDD: XX" is sufficient. Not sure Persistent adds much, and it will save space. Still prefer it non-bold.

nevillelyh · 2014-07-24T14:57:53Z

Addressed most of the comments. I kept it inside the box to avoid clutter in case name gets too long. Also renamed link to "+details"

SparkQA · 2014-07-24T14:58:23Z

QA tests have started for PR 1188. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17120/consoleFull

SparkQA · 2014-07-24T15:40:08Z

QA results for PR 1188:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17120/consoleFull

pwendell · 2014-07-24T19:23:00Z

core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala

+      <div class="stage-details collapsed">
+        {if (cachedRddInfos.nonEmpty) {
+          Text("RDD: ") ++
+          // scalastyle:off


Oh fancy! It would also have been okay to just comment out the entire code block :P

pwendell · 2014-07-24T21:12:41Z

Okay I tested locally - this is cool! Merging it.

Author: Neville Li <neville@spotify.com> Closes apache#1188 from nevillelyh/neville/ui and squashes the following commits: d3ac425 [Neville Li] SPARK-2250: show persisted RDD in stage UI f075db9 [Neville Li] SPARK-2035: show call stack even when description is available

nevillelyh closed this Jul 24, 2014

nevillelyh reopened this Jul 24, 2014

SPARK-2035: show call stack even when description is available

f075db9

pwendell reviewed Jul 24, 2014
View reviewed changes

SPARK-2250: show persisted RDD in stage UI

d3ac425

pwendell reviewed Jul 24, 2014
View reviewed changes

asfgit closed this in fec641b Jul 24, 2014

nevillelyh deleted the neville/ui branch July 31, 2014 03:52

wangyum added a commit that referenced this pull request May 26, 2023

CARMEL-6447: Do not push down sum if it is decimal type (#1188)

e03c921

SPARK-2250: show stage RDDs in UI #1188

SPARK-2250: show stage RDDs in UI #1188

Conversation

nevillelyh commented Jun 24, 2014

AmplabJenkins commented Jun 24, 2014

andrewor14 commented Jun 25, 2014

nevillelyh commented Jun 26, 2014

nevillelyh commented Jul 1, 2014

nevillelyh commented Jul 16, 2014

andrewor14 commented Jul 16, 2014

SparkQA commented Jul 16, 2014

SparkQA commented Jul 16, 2014

nevillelyh commented Jul 16, 2014

andrewor14 commented Jul 16, 2014

SparkQA commented Jul 16, 2014

SparkQA commented Jul 16, 2014

nevillelyh commented Jul 16, 2014

nevillelyh commented Jul 22, 2014

andrewor14 commented Jul 22, 2014

andrewor14 commented Jul 22, 2014

nevillelyh commented Jul 22, 2014

SparkQA commented Jul 22, 2014

SparkQA commented Jul 22, 2014

pwendell commented Jul 22, 2014

nevillelyh commented Jul 23, 2014

SparkQA commented Jul 23, 2014

SparkQA commented Jul 23, 2014

pwendell commented Jul 24, 2014

nevillelyh commented Jul 24, 2014

pwendell commented Jul 24, 2014

pwendell Jul 24, 2014

Choose a reason for hiding this comment

nevillelyh Jul 24, 2014

Choose a reason for hiding this comment

pwendell Jul 24, 2014

Choose a reason for hiding this comment

pwendell Jul 24, 2014

Choose a reason for hiding this comment

pwendell commented Jul 24, 2014

pwendell commented Jul 24, 2014

nevillelyh commented Jul 24, 2014

SparkQA commented Jul 24, 2014

SparkQA commented Jul 24, 2014

pwendell Jul 24, 2014

Choose a reason for hiding this comment

pwendell commented Jul 24, 2014