[SPARK-22343][core] Add support for publishing Spark metrics into Prometheus #19775

matyix · 2017-11-17T13:51:58Z

What changes were proposed in this pull request?

Originally this PR was submitted to the Spark on K8S fork here but has been advised to resend it upstream by @erikerlandson and @foxish. K8S specific items were removed from the PR and been reworked for the Apache version.

Publishing Spark metrics into Prometheus - as highlighted in the JIRA. Implemented a metrics sink that publishes Spark metrics into Prometheus via Prometheus Pushgateway. Metrics data published by Spark is based on Dropwizard. The format of Spark metrics is not supported natively by Prometheus thus these are converted using DropwizardExports prior pushing metrics to the pushgateway.

Also the default Prometheus pushgateway client API implementation does not support metrics timestamp thus the client API has been ehanced to enrich metrics data with timestamp.

How was this patch tested?

This PR is not affecting the existing code base and not altering the functionality. Nevertheless, I have executed all unit and integration tests. Also this setup has been deployed and been monitored via Prometheus (Prometheus 1.7.1 + Pushgateway 0.3.1).

Manual testing through deploying a Spark cluster, Prometheus server, Pushgateway and ran SparkPi.

erikerlandson · 2017-11-17T18:28:21Z

@matyix thanks for re-submitting!

jerryshao · 2017-11-24T03:59:37Z

Do we have to put this in Spark, is it a necessary part of k8s? I think if we pull in that PR(#11994), then this can be stayed out of Spark as a package. Even without #11994 , I believe users can still add their own Metrics source/sink via exposed SparkEnv/MetricsSystem. My concern is that this unnecessarily increases the code base of spark core.

matyix · 2017-11-24T17:31:15Z

@jerryshao this PR is not Kubernetes specific, it's an extension of the Spark Metrics system which is part of the core already. We could externalize if the PR #11994 above ever gets merged. Re-factoring and externalizing it after (for all the other metrics subsystem is not a big work). Although I submitted this PR first on the K8S fork, actually this feature might be beneficial for all using the (becoming) de-facto monitoring solution, Prometheus.

felixcheung · 2017-11-24T20:27:43Z

I agree this is useful to have, @jerryshao is probably right though, it is likely better to add extensibility into the Metrics system.

@jerryshao I'll review your PR
@matyix could you also review #11994 to see if that suits the need to build everything you have here to connect to Prometheus, but external to Spark and on top of #11994? I think your feedback will be very valuable.

We can then come back to this PR as needed.

matyix · 2017-11-27T14:15:18Z

Hello @felixcheung @jerryshao

The PR #11994 generally looks good for adding extensibility into the Metrics system. This PR (Prometheus) works with the changes proposed in PR #11994 out of the box, so no modification needed on our side.

As Prometheus is (becoming) a widely used monitoring system personally I see a lot of value including this into the Spark code base, same as the current metrics. I see PR #11994 rather a solution for those who build custom specialised metrics sink.

AmplabJenkins · 2018-01-18T17:30:57Z

Can one of the admins verify this patch?

matyix · 2018-02-08T09:29:57Z

Hello @erikerlandson @felixcheung @jerryshao - any feedback on this PR? Shall I close it and not worry about this being merged upstream anymore? We've been using this in production for the last 3 months and it's a bit awkward that our CI/CD system needs to patch the upstream version all the time but we can live with that (since it's automated). Please advise. Happy to help to get it merge or eventually just close it.

smurakozi · 2018-02-08T09:59:10Z

.../main/java/org/apache/spark/metrics/prometheus/client/exporter/PushGatewayWithTimestamp.java

+     * This uses the PUT HTTP method.
+     * @deprecated use {@link #push(CollectorRegistry, String, Map)}
+     */
+    @Deprecated


This is a new class, why should we include these deprecated methods?

Removed deprecated methods.

smurakozi · 2018-02-08T10:04:14Z

...c/main/java/org/apache/spark/metrics/prometheus/client/exporter/TextFormatWithTimestamp.java

+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TextFormatWithTimestamp {


smurakozi · 2018-02-08T10:04:16Z

.../main/java/org/apache/spark/metrics/prometheus/client/exporter/PushGatewayWithTimestamp.java

+        delete(job, Collections.singletonMap("instance", instance));
+    }
+
+    void doRequest(CollectorRegistry registry, String job, Map<String,


It should be private.

Changed it to private.

smurakozi · 2018-02-08T10:05:12Z

...c/main/java/org/apache/spark/metrics/prometheus/client/exporter/TextFormatWithTimestamp.java

+
+    private static StringBuilder jsonMessageLogBuilder = new StringBuilder();
+
+    public static void write004(Writer writer,


smurakozi · 2018-02-08T10:15:08Z

...c/main/java/org/apache/spark/metrics/prometheus/client/exporter/TextFormatWithTimestamp.java

+    /* See http://prometheus.io/docs/instrumenting/exposition_formats/
+     * for the output format specification. */
+    while(mfs.hasMoreElements()) {
+        Collector.MetricFamilySamples metricFamilySamples = mfs.nextElement();


I think for(Collector.MetricFamilySamples s: Collections.list(mfs)) { would be nicer.

Also, method body is not indented well.

This class has been refactored.

smurakozi · 2018-02-08T10:36:33Z

core/src/main/scala/org/apache/spark/metrics/sink/PrometheusSink.scala

+        case _ => Map("role" -> role)
+      }
+
+


Nit: extra line

Empty line removed.

smurakozi · 2018-02-08T10:40:23Z

core/src/main/scala/org/apache/spark/metrics/sink/PrometheusSink.scala

+
+  override def start(): Unit = {
+    sparkMetricExports.register(pushRegistry)
+


Nit: extra line

Empty line removed.

smurakozi · 2018-02-08T10:42:19Z

core/src/main/scala/org/apache/spark/metrics/sink/PrometheusSink.scala

+private[spark] class PrometheusSink(
+                                     val property: Properties,
+                                     val registry: MetricRegistry,
+                                     securityMgr: SecurityManager)


securityMgr is never used

The parameter list for a Sink is imposed byMetricsSystem which instantiates the configured sinks (see https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala#L199). PrometheusSink doesn't need SecurityManager this is why securityMgr is not used (similar to CsvSink, ConsoleSink).

smurakozi · 2018-02-08T10:46:13Z

.../main/java/org/apache/spark/metrics/prometheus/client/exporter/PushGatewayWithTimestamp.java

+        } catch (Exception ex) {
+            logger.error("Sending metrics failed due to: ", ex);
+        }
+


Nit: extra line

Removed extra line.

smurakozi · 2018-02-08T10:46:59Z

.../main/java/org/apache/spark/metrics/prometheus/client/exporter/PushGatewayWithTimestamp.java

+        }
+
+        finally {
+            connection.disconnect();


connection can be null if new URL(url).openConnection() at line 272 threw an exception.

The new URL(url).openConnection() is outside of the try-catch-block this in case it throws an exception it exist doRequest() before reaching finally { connection.disconnect();

jerryshao · 2018-02-09T01:48:36Z

My original intention is to expose MetricsSystem related interface in #11994 , so that users can leverage such interface to build their own metrics sink/source out of Spark. Unfortunately I'm stuck on the #11994 , but still I think it is better to leave this as a package out of Spark, pulling to much dependencies for non-core functionalities seems not so reasonable (just my thoughts).

vanzin · 2018-02-09T18:21:22Z

Regardless of the discussion about whether this should live in Spark, does it need to live in core?

Can it be kept in a separate module like the Ganglia stuff (even though that one is for licensing reasons)?

stoader · 2018-02-12T10:39:15Z

@smurakozi thank you for reviewing. The PR has been updated based on your comments.

erikerlandson · 2018-02-14T14:06:17Z

I agree w/ @jerryshao that adding new deps to core isn't ideal. (Also that having #11994 would be really nice)
New deps on a sub-project seems more palatable, but interested in what other Sparkers think.

erikerlandson · 2018-02-14T14:07:51Z

Although this is not kube-specific, kubernetes deployment is a major prometheus use case. Has it been tested in a kube environment?

…nfigurable as not all version of Pushgateway supports metrics timestamp

stoader · 2018-02-17T17:13:31Z

@erikerlandson we tested this on Kubernetes using https://github.com/prometheus/pushgateway/tree/v0.3.1 and https://github.com/prometheus/pushgateway/tree/v0.4.0

GaalDornick · 2018-03-08T14:10:06Z

So, where did we land on the discussion for this PR. Is this change getting in or is it out?

matyix · 2018-03-08T14:35:05Z

@GaalDornick @erikerlandson @jerryshao @felixcheung et all

We gave up this - we have made the requested changes several times and I am not willing to put more time on this and get in the middle of a debate which is not my concern. Currently the Spark monitoring architecture it is how it is - and we have made the PR to align with the current architecture of the existing sinks and metrics subsystem. What did happen is that now the debate is not about that this is good, needed or not but whether it should be part of Spark core, be pluggable, we should refactor the whole metrics subsystem, etc. Most likely this will still be the case later and once these will be changed, nailed down or agreed by all parties I can rework and resend the PR...

Anyways, we (and our customers) are using this in production for months - we have externalized this into a separate jar which we put it on the CP and does not need to be part of Spark (though it should I believe, as Prometheus is one of the best open source monitoring framework).

Should anybody need help to use this sink drop me a mail at janos@banzaicloud.com happy to help if interested in using Prometheus with Spark. We do some advanced scenarios with this sink and the code is all open sourced - you can read more about Monitoring Spark with Prometheus and Federated monitoring of multiple Spark clusters.

Thanks for all the support.
Janos

matyix · 2018-03-10T18:48:13Z

For those who are still interested using Prometheus you can get the standalone package and source code from here: https://github.com/banzaicloud/spark-metrics . Happy monitoring, try to catch the issues and avoid those PagerDuty notifications beforehand :).

andrusha · 2018-04-24T10:11:50Z

You can also try using https://github.com/andrusha/dropwizard-prometheus, which implements pull instead of push.

lony · 2018-05-16T11:09:18Z

@andrusha Do you have a tutorial how to set this up. From my understanding if I have multiple executers pulling it is harder, as prometheus has to have all the hostnames. Am I right or wrong?

andrusha · 2018-05-24T20:49:05Z

You can use Kubernetes service discovery with conjunction with Prometheus to find all the available nodes for scraping. Then you just add in your annotation the host and port. https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Ckubernetes_sd_config%3E <https://prometheus.io/docs/prometheus/latest/configuration/configuration/#<kubernetes_sd_config>> is a good place to start. I think it requires a blog post, but not sure if will ever do it.

…

On 16 May 2018, at 13:10, Goetz Epperlein ***@***.***> wrote: @andrusha <https://github.com/andrusha> Do you have a tutorial how to set this up. From my understanding if I have multiple executers pulling it is harder, as prometheus has to have all the hostnames. Am I right or wrong? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#19775 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAXqz9edXdn5rSat4-m9464k3NM-c2NJks5tzAkSgaJpZM4QiEPC>.

mageru · 2018-08-03T14:31:08Z

Did we really miss out on Prometheus metric functionality because people couldn't just be cool about it?

Add support for prometheus

579cca9

matyix changed the title ~~Add support for publishing Spark metrics into Prometheus~~ [SPARK-22343][core] Add support for publishing Spark metrics into Prometheus Nov 17, 2017

matyix mentioned this pull request Nov 17, 2017

Add support for publishing Spark metrics into Prometheus apache-spark-on-k8s/spark#531

Closed

jerryshao mentioned this pull request Nov 28, 2017

[SPARK-14151][CORE] Expose metrics Source and Sink interface #11994

Closed

matyix mentioned this pull request Jan 3, 2018

Support MetricNameSpace in executors and include shuffle to shuffleservice metrics apache-spark-on-k8s/spark#532

Closed

smurakozi suggested changes Feb 8, 2018

View reviewed changes

stoader added 2 commits February 9, 2018 23:15

Removed deprecated methods from PushGatewayWithTimestamp

c13a053

Refactored TextFormatWithTimestamp

ea772b9

Fix checkstyle errors

c11b363

stoader added 2 commits February 14, 2018 17:41

Fix: PrometheusSink exits if connection to Pushgateway fails

118f718

Metrics timestamp to be passed to Prometheus Pushgateway should be co…

344c464

…nfigurable as not all version of Pushgateway supports metrics timestamp

matyix closed this Mar 8, 2018

itsvikramagr mentioned this pull request Sep 12, 2019

[SPARK-29032][CORE] Add PrometheusServlet to monitor Master/Worker/Driver #25769

Closed

dongjoon-hyun mentioned this pull request Dec 11, 2019

[SPARK-30221] Enhanced implementation of PrometheusPushGateWaySink #26852

Closed


		private static StringBuilder jsonMessageLogBuilder = new StringBuilder();

		public static void write004(Writer writer,


		override def start(): Unit = {
		sparkMetricExports.register(pushRegistry)

[SPARK-22343][core] Add support for publishing Spark metrics into Prometheus #19775

[SPARK-22343][core] Add support for publishing Spark metrics into Prometheus #19775

Conversation

matyix commented Nov 17, 2017

What changes were proposed in this pull request?

How was this patch tested?

erikerlandson commented Nov 17, 2017

jerryshao commented Nov 24, 2017

matyix commented Nov 24, 2017 • edited Loading

felixcheung commented Nov 24, 2017

matyix commented Nov 27, 2017

AmplabJenkins commented Jan 18, 2018

matyix commented Feb 8, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stoader Feb 9, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jerryshao commented Feb 9, 2018

vanzin commented Feb 9, 2018

stoader commented Feb 12, 2018

erikerlandson commented Feb 14, 2018

erikerlandson commented Feb 14, 2018

stoader commented Feb 17, 2018

GaalDornick commented Mar 8, 2018

matyix commented Mar 8, 2018 • edited Loading

matyix commented Mar 10, 2018

andrusha commented Apr 24, 2018

lony commented May 16, 2018

andrusha commented May 24, 2018 via email

mageru commented Aug 3, 2018

matyix commented Nov 24, 2017 •

edited

Loading

stoader Feb 9, 2018 •

edited

Loading

matyix commented Mar 8, 2018 •

edited

Loading