Hi, recently our team has evaluated the prometheusServlet configuration in order to have Spark master, worker, driver and executor native Prometheus metrics. We are running a Spark streaming standalone cluster version 3.0.1 on physical servers.
metrics.properties *.sink.prometheusServlet.class=org.apache.spark.metrics.sink.PrometheusServlet *.sink.prometheusServlet.path=/metrics/prometheus master.sink.prometheusServlet.path=/metrics/master/prometheus applications.sink.prometheusServlet.path=/metrics/applications/prometheus I'm able to see Spark master, worker and driver metrics but cannot see any spark executor metrics (curling the relevant Prometheus endpoints just results in empty result). For the purposes of executor metrics verification, I have taken one of our jobs and added the following parameters to the job configuration. These are the parameters I pass alongside our spark-submit command: --conf spark.ui.prometheus.enabled=true \ --conf spark.executor.processTreeMetrics.enabled=true Spark official documentation mentions that Prometheus executor endpoint is still experimental which may explain this inconsistent behaviour. Any help will be highly appreciated! -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
