The PrometheusServlet adds a servlet within the existing Spark UI to serve metrics data in Prometheus format. Similarly to what happens with the MetricsServlet, the Prometheus servlet does not work on executors, as executors do not have a Spark UI end point to which the servlet could attach to. Currently Apache Spark does not have an official Prometheus sink for the metrics system. Spark has a Graphite sink that can be used with InfluxDB 1.x, which could be useful in this context. I also see that a Prometheus sink is available as an external libraries from an open source project.
Best, Luca -----Original Message----- From: paulp <[email protected]> Sent: Monday, May 24, 2021 17:09 To: [email protected] Subject: Spark Prometheus Metrics for Executors Not Working Hi, recently our team has evaluated the prometheusServlet configuration in order to have Spark master, worker, driver and executor native Prometheus metrics. We are running a Spark streaming standalone cluster version 3.0.1 on physical servers. metrics.properties *.sink.prometheusServlet.class=org.apache.spark.metrics.sink.PrometheusServlet *.sink.prometheusServlet.path=/metrics/prometheus master.sink.prometheusServlet.path=/metrics/master/prometheus applications.sink.prometheusServlet.path=/metrics/applications/prometheus I'm able to see Spark master, worker and driver metrics but cannot see any spark executor metrics (curling the relevant Prometheus endpoints just results in empty result). For the purposes of executor metrics verification, I have taken one of our jobs and added the following parameters to the job configuration. These are the parameters I pass alongside our spark-submit command: --conf spark.ui.prometheus.enabled=true \ --conf spark.executor.processTreeMetrics.enabled=true Spark official documentation mentions that Prometheus executor endpoint is still experimental which may explain this inconsistent behaviour. Any help will be highly appreciated! -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: [email protected] --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
