Setting up prometheus sink in this way:
-c spark.ui.prometheus.enabled=true
-c spark.executor.processTreeMetrics.enabled=true
-c spark.metrics.conf=/spark/conf/metric.properties
*metric.properties:*{}
*.sink.prometheusServlet.class=org.apache.spark.metrics.sink.PrometheusServlet
*.sink.prometheusServlet.path=/metrics/prometheus
Result:
Both of these endpoints have some metrics
<driver-ip>:4040/metrics/prometheus
<driver-ip>:4040/metrics/executors/prometheus
But the executor one misses metrics under the executor namespace described
here:
https://spark.apache.org/docs/latest/monitoring.html#component-instance--executor
*How to expose executor metrics on spark exeuctors pod?*
*Any help will be appreciated.*
--
Regards,
Qian Sun
