[SPARK-20355] Add per application spark version on the history server headerpage #17658

redsanket · 2017-04-17T14:56:32Z

What changes were proposed in this pull request?

Spark Version for a specific application is not displayed on the history page now. It should be nice to switch the spark version on the UI when we click on the specific application.
Currently there seems to be way as SparkListenerLogStart records the application version. So, it should be trivial to listen to this event and provision this change on the UI.
For Example

{"Event":"SparkListenerLogStart","Spark Version":"2.0.0"}
(Please fill in changes proposed in this fix)
Modified the SparkUI for History server to listen to SparkLogListenerStart event and extract the version and print it.

How was this patch tested?

Manual testing of UI page. Attaching the UI screenshot changes here

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

Please review http://spark.apache.org/contributing.html before opening a pull request.

tgravescs · 2017-04-17T15:33:55Z

Jenkins, test this please

SparkQA · 2017-04-17T15:39:09Z

Test build #75856 has finished for PR 17658 at commit 1f50b27.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

ajbozarth

I think this idea is nice, but it also seems confusing (though leaving it the way it is now is just as confusing). I'm a bit worried about the listener changes as noted in my comments, but you can probably clear my worries with some explanations of your decisions.

ajbozarth · 2017-04-17T21:00:58Z

core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala

+            SparkUI.createHistoryUI(conf, replayBus, appSecManager,
+              appListener.appSparkVersion.getOrElse(""), appInfo.name,
+              HistoryServer.getAttemptURI(appId, attempt.attemptId),
+              attempt.startTime)


I don't think adding this line wrap is necessary

ajbozarth · 2017-04-17T22:29:32Z

core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala

@@ -71,7 +71,6 @@ private[spark] trait SparkListenerBus
        listener.onNodeUnblacklisted(nodeUnblacklisted)
      case blockUpdated: SparkListenerBlockUpdated =>
        listener.onBlockUpdated(blockUpdated)
-      case logStart: SparkListenerLogStart => // ignore event log metadata


I haven't looked at this closely, but will allowing this through cause any issues? Why was it ignored before?

I do not see it getting consumed apart from registering some metadata, so I guess it should be fine as this event already logs the version

Will this change the behavior of SparkListenerBus? Previously we ignored this event, but here with this change we will post this event through listener.onOtherEvent and user will get this event. But from the code this event is not accessible outside of Spark.

private[spark] case class SparkListenerLogStart(sparkVersion: String) extends SparkListenerEvent

I think that if this is gonna be posted on the bus it might be better to make it public and @DeveloperApi like other events.

Adding new events is not an issue, but posting an event that people can't handle is a little odd.

I am not too sure if it will change any behavior and precisely why we post it to other events, in case someone wants to listen to them and utilize the event like in this scenario

ajbozarth · 2017-04-17T22:30:13Z

core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala

@@ -160,7 +160,6 @@ case class SparkListenerApplicationEnd(time: Long) extends SparkListenerEvent

 /**
 * An internal class that describes the metadata of an event log.
- * This event is not meant to be posted to listeners downstream.


Is there a reason this note was here? What was the reasoning behind it?

This was only for metadata info, so when this was written it was just not meant to be consumed but now we can reuse it for this case

tgravescs · 2017-04-18T19:08:50Z

the idea here was mine. I agree that it could be confusing but its also confusing as is and its hard to find the version that was run. I was figuring this way would be consistent with the live UI. If you have other suggestions I am definitely open. I thought about just removing the version from there and putting it elsewhere but then its less consistent.

As far as you questions on the event. I don't really know why it was made private my assumption here is that it was for metadata only and could have potentially changed. However this is the only place we currently store the version and I don't really see any harm in sending this event out but it would be good to check with others.

@andrewor14 @vanzin Do you have any issues with changing to send SparkListenerLogStart event out?

The other option is to add a new event or put the version in with another event but then it won't work with any of the older versions.

jerryshao · 2017-04-19T09:23:20Z

core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala

            // Do not call ui.bind() to avoid creating a new server for each application
          }

-          val fileStatus = fs.getFileStatus(new Path(logDir, attempt.logPath))
-
-          val appListener = replay(fileStatus, isApplicationCompleted(fileStatus), replayBus)


~~Is it correct to remove this two lines? As I know it is necessary because in the previous replay call we only parse SparkListenerApplicationStart and SparkListenerApplicationEnd event.~~

Sorry about it, just saw you moved this two lines above.

vanzin

I think this is fine. But it would be great to expose this in the API too (e.g. in ApplicationInfo).

vanzin · 2017-04-19T17:40:54Z

core/src/main/scala/org/apache/spark/scheduler/ApplicationEventListener.scala

+  override def onOtherEvent(event:SparkListenerEvent): Unit = event match {
+    case SparkListenerLogStart(sparkVersion) =>
+      appSparkVersion = Some(sparkVersion)
+  }


You need a "catch all" here:

scala> class MyListener extends org.apache.spark.scheduler.SparkListener { | override def onOtherEvent(event: org.apache.spark.scheduler.SparkListenerEvent): Unit = event match { | case _: org.apache.spark.scheduler.SparkListenerTaskStart => () | } | } defined class MyListener scala> sc.addSparkListener(new MyListener()) scala> spark.range(1000).write.saveAsTable("test") 17/04/19 10:39:01 ERROR LiveListenerBus: Listener MyListener threw an exception scala.MatchError: SparkListenerSQLExecutionStart(0,saveAsTable at <console>:27,org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:354)

vanzin · 2017-04-19T17:42:04Z

core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala

@@ -71,7 +71,6 @@ private[spark] trait SparkListenerBus
        listener.onNodeUnblacklisted(nodeUnblacklisted)
      case blockUpdated: SparkListenerBlockUpdated =>
        listener.onBlockUpdated(blockUpdated)
-      case logStart: SparkListenerLogStart => // ignore event log metadata


I think that if this is gonna be posted on the bus it might be better to make it public and @DeveloperApi like other events.

Adding new events is not an issue, but posting an event that people can't handle is a little odd.

redsanket · 2017-04-19T17:59:37Z

@vanzin sure will address the concerns thanks for the review

tgravescs · 2017-04-24T13:59:06Z

+1. @vanzin any further comments?

redsanket · 2017-04-24T17:45:02Z

Jenkins, test this please

vanzin

Are you going to add this information to the REST API? I don't think it's there yet, and I'd really like to keep UI and API close when possible. It should be pretty easy here.

My preference would be a new field in RuntimeInfo, or ApplicationInfo if that's too awkward.

vanzin · 2017-04-25T00:46:40Z

core/src/main/scala/org/apache/spark/scheduler/ApplicationEventListener.scala

@@ -57,4 +58,10 @@ private[spark] class ApplicationEventListener extends SparkListener {
      adminAclsGroups = allProperties.get("spark.admin.acls.groups")
    }
  }
+
+  override def onOtherEvent(event:SparkListenerEvent):Unit = event match {


nit: space after :

Scala style check failed Scalastyle checks failed for this in the above run

at following occurrences:
[error] /home/jenkins/workspace/SparkPullRequestBuilder/core/src/main/scala/org/apache/spark/scheduler/ApplicationEventListener.scala:62:33: No space after token :
[error] (core/compile:scalastyle) errors exist
[error] Total time: 11 s, completed Apr 17, 2017 8:39:08 AM
[error] running /home/jenkins/workspace/SparkPullRequestBuilder/dev/lint-scala ; received return code 1
Attempting to post to Github...

oh ok I thought I had space before the style check complained about it but might have interpreted it wrongly

tgravescs · 2017-04-25T13:37:26Z

Jenkins, test this please

tgravescs · 2017-04-25T13:39:14Z

core/src/main/scala/org/apache/spark/ui/SparkUI.scala

@@ -139,6 +140,8 @@ private[spark] abstract class SparkUITab(parent: SparkUI, prefix: String)

  def appName: String = parent.appName

+  def appSparkVersion: String = parent.appSparkVersion
+


nit, remove extra line if you are making changes.

redsanket · 2017-04-25T20:55:11Z

@vanzin Can I add this to SparkConf.scala https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkConf.scala#L58 just to have the application info here it will have the info on the API without modifying the intended interface as I see modifying ApplicationInfo API will have consequences on wherever it is used and might be a bigger change? Also adding it to RuntimeInfo will be awkward like you said, having it is conf would be nice place where it can live without making major changes. Let me know your opinion, I can make changes

vanzin · 2017-04-25T21:02:14Z

Adding fields to API types is ok. It doesn't break the contract.

Adding this in the configuration is weird, because it's not a config.

redsanket · 2017-04-25T21:05:20Z

ok will add it to either RuntimeInfo or ApplicationInfo. I thought it might break the contract underneath but if it doesn't then I should add it here thanks for the input. Will do

redsanket · 2017-04-26T00:43:07Z

Jenkins, test this please

tgravescs · 2017-04-26T13:12:43Z

Jenkins, test this please

tgravescs · 2017-04-26T15:40:41Z

test this please

SparkQA · 2017-04-26T15:57:41Z

Test build #76190 has finished for PR 17658 at commit 5f6e80b.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2017-04-26T16:09:14Z

The mima problem is actually not a problem; the constructor is private so there's no problem in changing it. Just add the exception in MimaExcludes.scala.

vanzin · 2017-04-26T16:23:44Z

core/src/main/scala/org/apache/spark/status/api/v1/ApplicationListResource.scala

@@ -92,7 +92,8 @@ private[spark] object ApplicationsListResource {
          sparkUser = internalAttemptInfo.sparkUser,
          completed = internalAttemptInfo.completed
        )
-      }
+      },
+      appSparkVersion = org.apache.spark.SPARK_VERSION


This is not right. This might be different from the current version, just like the UI case. You have to somehow plumb it from the event log information to here.

…et of application info

tgravescs · 2017-05-02T14:38:25Z

Jenkins, okay to test

tgravescs · 2017-05-02T15:08:17Z

test this please

SparkQA · 2017-05-02T18:47:47Z

Test build #76389 has finished for PR 17658 at commit e7da27d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2017-05-03T20:25:06Z

retest this please

redsanket · 2017-05-03T20:36:37Z

The issue before was that I was replaying the events before adding the listeners in the getSparkUI.
That caused a bunch of tests to fail and other files were to do with the appSparkVersion not being set in the expection_json to ""

tgravescs · 2017-05-03T20:40:51Z

core/src/main/scala/org/apache/spark/ui/SparkUI.scala

@@ -60,6 +60,8 @@ private[spark] class SparkUI private (

  var appId: String = _

+  var appSparkVersion = ""


we are no longer setting this for the liveUI so it either needs to be default here to org.apache.spark.SPARK_VERSION or we need to call the setAppSparkVersion in SparkContext.

Yeah will fix that thanks

tgravescs · 2017-05-03T20:43:23Z

there are also a few formatting things that look like they were just line wraps and extra new lines.

SparkQA · 2017-05-03T22:31:32Z

Test build #76425 has finished for PR 17658 at commit f1fa890.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

redsanket · 2017-05-04T20:00:02Z

SparkContext was not able to read SparkListenerLogStart event as it is not a part of it and the subsequent replay listener suite tries to compare eventLogs and original events emitted via SparkContext

tgravescs · 2017-05-04T20:00:04Z

retest this please

vanzin · 2017-05-04T20:03:51Z

no idea why tests are not triggering automatically for this PR. maybe: ok to test

SparkQA · 2017-05-04T20:09:16Z

Test build #76464 has finished for PR 17658 at commit bafad1a.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

redsanket · 2017-05-04T21:22:23Z

I think I should set up my IDE would be nice to have something like checkstyle.xml instead of configuring these things, sorry abt that

tgravescs · 2017-05-04T21:24:17Z

@srowen @vanzin do either of you know where the jenkins stuff is configured? wondering why this isn't working for me.

vanzin · 2017-05-04T21:26:22Z

I have no idea, sorry.

vanzin · 2017-05-04T23:29:46Z

core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala

@@ -283,10 +283,15 @@ private[spark] object EventLoggingListener extends Logging {
   *
   * @param logStream Raw output stream to the event log file.
   */
-  def initEventLog(logStream: OutputStream): Unit = {
+  def initEventLog(logStream: OutputStream, testing: Boolean,


nit: style

See style for multi-line arguments in ApplicationAttemptInfo, for example. But see below.

vanzin · 2017-05-04T23:31:31Z

core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala

    logStream.write(metadataJson.getBytes(StandardCharsets.UTF_8))
+    if (testing && loggedEvents != null) {
+      loggedEvents += eventJson


Instead of this, how about returning the SparkListenerLogStart event?

I thought the loggedEvents only takes json value. Also the loggedEvents are generated here as a part of spark context and probably through other sources. The ReplayListenerSuite however tests the original events with the replay events (here the replay events are written to the event log but however the loggedEvents will not have the SparkListenerLogStart event, as this is not a part of SparkContext if I understand it correctly).

The downside to returning the event is that this function can't do more in the future. For instance if it wants to add 2 events. I guess its private so it doesn't matter to much and its just for testing stuff.

My main concern is that it makes calling this method from other places awkward. (i.e. the semantics of the new arguments are not clear.)

In any case, there's a single other call to this method, in a test suite, so probably ok.

vanzin · 2017-05-04T23:32:48Z

core/src/main/scala/org/apache/spark/ui/SparkUI.scala

@@ -93,6 +95,10 @@ private[spark] class SparkUI private (
    appId = id
  }

+  def setAppSparkVersion(version: String): Unit = {


appSparkVersion is a var, so this is not technically needed.

vanzin · 2017-05-04T23:37:13Z

core/src/test/resources/HistoryServerExpectations/completed_app_list_json_expectation.json

@@ -22,6 +23,7 @@
    "duration" : 101795,
    "sparkUser" : "jose",
    "completed" : true,
+    "appSparkVersion" : "",


This doesn't look correct. If I look at the log file for this app (core/src/test/resources/spark-events/app-20161115172038-0000), there's a log start event:

{"Event":"SparkListenerLogStart","Spark Version":"2.1.0-SNAPSHOT"}

So this value should be "2.1.0-SNAPSHOT", no?

I am not sure the if the tests hit this code path https://github.com/apache/spark/pull/17658/files#diff-a7befb99e7bd7e3ab5c46c2568aa5b3eR474, so they take the default value

probably I could change the default value, ok will do it

It's not really about the default value; these tests replay the log files, which contain the Spark version, so I would expect the data retrieved through the API to contain the version that was recorded in the event log.

Another way of saying that probably there's a bug somewhere in your code that is preventing the data from the event log from being exposed correctly through the REST API.

I see events being passed to ApplicationEventListener in debug logs but interestingly I doPostEvent seems to be not posting events to the listener to listen to the event which is a bit odd, not sure I need to add a change in contract for the tests to pick this up though

17/05/05 16:39:42.235 qtp1618269752-1127 INFO ReplayListenerBus: listener --- event org.apache.spark.ui.jobs.JobProgressListener@53e58df ---SparkListenerLogStart(2.3.0-SNAPSHOT)
17/05/05 16:39:42.235 qtp1618269752-1127 INFO ReplayListenerBus: listener --- event org.apache.spark.ui.env.EnvironmentListener@53371e87 ---SparkListenerLogStart(2.3.0-SNAPSHOT)
17/05/05 16:39:42.235 qtp1618269752-1127 INFO ReplayListenerBus: listener --- event org.apache.spark.storage.StorageStatusListener@394665e7 ---SparkListenerLogStart(2.3.0-SNAPSHOT)
17/05/05 16:39:42.235 qtp1618269752-1127 INFO ReplayListenerBus: listener --- event org.apache.spark.ui.exec.ExecutorsListener@2d92f38e ---SparkListenerLogStart(2.3.0-SNAPSHOT)
17/05/05 16:39:42.235 qtp1618269752-1127 INFO ReplayListenerBus: listener --- event org.apache.spark.ui.storage.StorageListener@5a9dbe43 ---SparkListenerLogStart(2.3.0-SNAPSHOT)
17/05/05 16:39:42.235 qtp1618269752-1127 INFO ReplayListenerBus: listener --- event org.apache.spark.ui.scope.RDDOperationGraphListener@31b550b1 ---SparkListenerLogStart(2.3.0-SNAPSHOT)
17/05/05 16:39:42.236 qtp1618269752-1127 INFO ReplayListenerBus: listener --- event org.apache.spark.scheduler.ApplicationEventListener@3b56723d ---SparkListenerLogStart(2.3.0-SNAPSHOT)
17/05/05 16:39:43.300 qtp1618269752-1126 INFO ReplayListenerBus: listener --- event org.apache.spark.ui.jobs.JobProgressListener@3647a865 ---SparkListenerLogStart(2.3.0-SNAPSHOT)
17/05/05 16:39:43.300 qtp1618269752-1126 INFO ReplayListenerBus: listener --- event org.apache.spark.ui.env.EnvironmentListener@4357e8cc ---SparkListenerLogStart(2.3.0-SNAPSHOT)
17/05/05 16:39:43.300 qtp1618269752-1126 INFO ReplayListenerBus: listener --- event org.apache.spark.storage.StorageStatusListener@5062f4c0 ---SparkListenerLogStart(2.3.0-SNAPSHOT)
17/05/05 16:39:43.300 qtp1618269752-1126 INFO ReplayListenerBus: listener --- event org.apache.spark.ui.exec.ExecutorsListener@273c097f ---SparkListenerLogStart(2.3.0-SNAPSHOT)
17/05/05 16:39:43.300 qtp1618269752-1126 INFO ReplayListenerBus: listener --- event org.apache.spark.ui.storage.StorageListener@2c33997d ---SparkListenerLogStart(2.3.0-SNAPSHOT)
17/05/05 16:39:43.300 qtp1618269752-1126 INFO ReplayListenerBus: listener --- event org.apache.spark.ui.scope.RDDOperationGraphListener@505e15bf ---SparkListenerLogStart(2.3.0-SNAPSHOT)
17/05/05 16:39:43.300 qtp1618269752-1126 INFO ReplayListenerBus: listener --- event org.apache.spark.scheduler.ApplicationEventListener@175d3f80 ---SparkListenerLogStart(2.3.0-SNAPSHOT)
17/05/05 16:39:44.305 qtp1618269752-1131 INFO ReplayListenerBus: listener --- event org.apache.spark.ui.jobs.JobProgressListener@71dacceb ---SparkListenerLogStart(2.3.0-SNAPSHOT)
17/05/05 16:39:44.305 qtp1618269752-1131 INFO ReplayListenerBus: listener --- event org.apache.spark.ui.env.EnvironmentListener@6f41235c ---SparkListenerLogStart(2.3.0-SNAPSHOT)
17/05/05 16:39:44.305 qtp1618269752-1131 INFO ReplayListenerBus: listener --- event org.apache.spark.storage.StorageStatusListener@3cdcb6d0 ---SparkListenerLogStart(2.3.0-SNAPSHOT)
17/05/05 16:39:44.305 qtp1618269752-1131 INFO ReplayListenerBus: listener --- event org.apache.spark.ui.exec.ExecutorsListener@4abcd5c6 ---SparkListenerLogStart(2.3.0-SNAPSHOT)
17/05/05 16:39:44.305 qtp1618269752-1131 INFO ReplayListenerBus: listener --- event org.apache.spark.ui.storage.StorageListener@6d26f5ee ---SparkListenerLogStart(2.3.0-SNAPSHOT)
17/05/05 16:39:44.305 qtp1618269752-1131 INFO ReplayListenerBus: listener --- event org.apache.spark.ui.scope.RDDOperationGraphListener@b1b60d5 ---SparkListenerLogStart(2.3.0-SNAPSHOT)
17/05/05 16:39:44.305 qtp1618269752-1131 INFO ReplayListenerBus: listener --- event org.apache.spark.scheduler.ApplicationEventListener@52283d50 ---SparkListenerLogStart(2.3.0-SNAPSHOT)
17/05/05 16:39:44.676 qtp1478318899-1195 INFO ReplayListenerBus: listener --- event org.apache.spark.ui.jobs.JobProgressListener@f1563e7 ---SparkListenerLogStart(1.4.0-SNAPSHOT)
17/05/05 16:39:44.676 qtp1478318899-1195 INFO ReplayListenerBus: listener --- event org.apache.spark.ui.env.EnvironmentListener@9122315 ---SparkListenerLogStart(1.4.0-SNAPSHOT)
17/05/05 16:39:44.676 qtp1478318899-1195 INFO ReplayListenerBus: listener --- event org.apache.spark.storage.StorageStatusListener@eb98c21 ---SparkListenerLogStart(1.4.0-SNAPSHOT)
17/05/05 16:39:44.676 qtp1478318899-1195 INFO ReplayListenerBus: listener --- event org.apache.spark.ui.exec.ExecutorsListener@1da79f75 ---SparkListenerLogStart(1.4.0-SNAPSHOT)
17/05/05 16:39:44.676 qtp1478318899-1195 INFO ReplayListenerBus: listener --- event org.apache.spark.ui.storage.StorageListener@3989cd7d ---SparkListenerLogStart(1.4.0-SNAPSHOT)
17/05/05 16:39:44.676 qtp1478318899-1195 INFO ReplayListenerBus: listener --- event org.apache.spark.ui.scope.RDDOperationGraphListener@219b4bd2 ---SparkListenerLogStart(1.4.0-SNAPSHOT)
17/05/05 16:39:44.676 qtp1478318899-1195 INFO ReplayListenerBus: listener --- event org.apache.spark.scheduler.ApplicationEventListener@63811344 ---SparkListenerLogStart(1.4.0-SNAPSHOT)
17/05/05 16:39:44.722 qtp1478318899-1195 INFO ReplayListenerBus: listener --- event org.apache.spark.ui.jobs.JobProgressListener@52177d75 ---SparkListenerLogStart(1.4.0-SNAPSHOT)
17/05/05 16:39:44.722 qtp1478318899-1195 INFO ReplayListenerBus: listener --- event org.apache.spark.ui.env.EnvironmentListener@72409b15 ---SparkListenerLogStart(1.4.0-SNAPSHOT)
17/05/05 16:39:44.722 qtp1478318899-1195 INFO ReplayListenerBus: listener --- event org.apache.spark.storage.StorageStatusListener@5ee11505 ---SparkListenerLogStart(1.4.0-SNAPSHOT)
17/05/05 16:39:44.722 qtp1478318899-1195 INFO ReplayListenerBus: listener --- event org.apache.spark.ui.exec.ExecutorsListener@24bad915 ---SparkListenerLogStart(1.4.0-SNAPSHOT)
17/05/05 16:39:44.722 qtp1478318899-1195 INFO ReplayListenerBus: listener --- event org.apache.spark.ui.storage.StorageListener@4d2fe422 ---SparkListenerLogStart(1.4.0-SNAPSHOT)
17/05/05 16:39:44.723 qtp1478318899-1195 INFO ReplayListenerBus: listener --- event org.apache.spark.ui.scope.RDDOperationGraphListener@3bc156a1 ---SparkListenerLogStart(1.4.0-SNAPSHOT)
17/05/05 16:39:44.723 qtp1478318899-1195 INFO ReplayListenerBus: listener --- event org.apache.spark.scheduler.ApplicationEventListener@6d5e35e1 ---SparkListenerLogStart(1.4.0-SNAPSHOT)

I think I know what this is. FsHistoryProvider has this code to speed up the log parsing for listings:

protected def mergeApplicationListing(fileStatus: FileStatus): Unit = { val newAttempts = try { val eventsFilter: ReplayEventsFilter = { eventString => eventString.startsWith(APPL_START_EVENT_PREFIX) || eventString.startsWith(APPL_END_EVENT_PREFIX) }

It only allows those two events through currently, you probably need to the log start event to that list too.

ok doPost is posting the events
7/05/05 17:17:17.265 qtp927704210-1183 INFO ReplayListenerBus: eventInDoPost SparkListenerLogStart(1.4.0-SNAPSHOT) So it should be good, so looks like ApplicationEventListener is not able to read the events, it used to before something has changed will dig deeper

aah ok the UI will work because it is replaying but the rest end point would break as it is not allowing it to pass through, thanks @vanzin

SparkQA · 2017-05-05T00:10:30Z

Test build #76466 has finished for PR 17658 at commit 52f414c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-05-05T04:55:56Z

Test build #76469 has finished for PR 17658 at commit dad87a6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2017-05-05T23:22:38Z

LGTM pending tests.

SparkQA · 2017-05-06T01:50:17Z

Test build #76505 has finished for PR 17658 at commit 93e7a20.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

redsanket · 2017-05-09T14:02:55Z

@tgravescs @vanzin ready for merge?

tgravescs · 2017-05-09T14:29:08Z

looks good, I'll merge. thanks @redsanket

… headerpage ## What changes were proposed in this pull request? Spark Version for a specific application is not displayed on the history page now. It should be nice to switch the spark version on the UI when we click on the specific application. Currently there seems to be way as SparkListenerLogStart records the application version. So, it should be trivial to listen to this event and provision this change on the UI. For Example <img width="1439" alt="screen shot 2017-04-06 at 3 23 41 pm" src="https://cloud.githubusercontent.com/assets/8295799/25092650/41f3970a-2354-11e7-9b0d-4646d0adeb61.png"> <img width="1399" alt="screen shot 2017-04-17 at 9 59 33 am" src="https://cloud.githubusercontent.com/assets/8295799/25092743/9f9e2f28-2354-11e7-9605-f2f1c63f21fe.png"> {"Event":"SparkListenerLogStart","Spark Version":"2.0.0"} (Please fill in changes proposed in this fix) Modified the SparkUI for History server to listen to SparkLogListenerStart event and extract the version and print it. ## How was this patch tested? Manual testing of UI page. Attaching the UI screenshot changes here (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review http://spark.apache.org/contributing.html before opening a pull request. Author: Sanket <schintap@untilservice-lm> Closes apache#17658 from redsanket/SPARK-20355.

Add per application spark version on the history server headerpage

1f50b27

ajbozarth reviewed Apr 17, 2017

View reviewed changes

jerryshao reviewed Apr 19, 2017

View reviewed changes

vanzin reviewed Apr 19, 2017

View reviewed changes

address style issues and expose SparkListenerLogStart API

21f7903

vanzin reviewed Apr 25, 2017

View reviewed changes

tgravescs reviewed Apr 25, 2017

View reviewed changes

fixing minor space nits

bad5ea2

add appSparkVersion for Rest API to be consistent

5f6e80b

vanzin reviewed Apr 26, 2017

View reviewed changes

Sanket added 3 commits May 1, 2017 15:10

plug in appSparkVersion into applicationattemptInfo which is the subs…

097e7cc

…et of application info

resolve conflicts

5533775

fix prev change to test

7350649

Fix unit tests failures

f1fa890

tgravescs reviewed May 3, 2017

View reviewed changes

fixing indentation and appSparkVersion inital value

298248e

fix ReplayListerSuite Test

bafad1a

fix scala style tests

52f414c

vanzin reviewed May 4, 2017

View reviewed changes

fix minor nits

dad87a6

fix rest end point filter to allow logstart event

93e7a20

asfgit closed this in 181261a May 9, 2017

		@@ -139,6 +140,8 @@ private[spark] abstract class SparkUITab(parent: SparkUI, prefix: String)

		def appName: String = parent.appName

		def appSparkVersion: String = parent.appSparkVersion

		@@ -60,6 +60,8 @@ private[spark] class SparkUI private (

		var appId: String = _

		var appSparkVersion = ""

[SPARK-20355] Add per application spark version on the history server headerpage #17658

[SPARK-20355] Add per application spark version on the history server headerpage #17658

Conversation

redsanket commented Apr 17, 2017 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

tgravescs commented Apr 17, 2017

SparkQA commented Apr 17, 2017

ajbozarth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tgravescs commented Apr 18, 2017

jerryshao Apr 19, 2017 • edited Loading

Choose a reason for hiding this comment

vanzin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

redsanket commented Apr 19, 2017

tgravescs commented Apr 24, 2017

redsanket commented Apr 24, 2017

vanzin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tgravescs commented Apr 25, 2017

Choose a reason for hiding this comment

redsanket commented Apr 25, 2017

vanzin commented Apr 25, 2017

redsanket commented Apr 25, 2017

redsanket commented Apr 26, 2017

tgravescs commented Apr 26, 2017

tgravescs commented Apr 26, 2017

SparkQA commented Apr 26, 2017

vanzin commented Apr 26, 2017

Choose a reason for hiding this comment

tgravescs commented May 2, 2017

tgravescs commented May 2, 2017

SparkQA commented May 2, 2017

vanzin commented May 3, 2017

redsanket commented May 3, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tgravescs commented May 3, 2017

SparkQA commented May 3, 2017

redsanket commented May 4, 2017

tgravescs commented May 4, 2017

vanzin commented May 4, 2017

SparkQA commented May 4, 2017

redsanket commented May 4, 2017

tgravescs commented May 4, 2017

vanzin commented May 4, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

redsanket May 5, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

redsanket May 5, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented May 5, 2017

SparkQA commented May 5, 2017

vanzin commented May 5, 2017

SparkQA commented May 6, 2017

redsanket commented Apr 17, 2017 •

edited

Loading

jerryshao Apr 19, 2017 •

edited

Loading

redsanket May 5, 2017 •

edited

Loading

redsanket May 5, 2017 •

edited

Loading