Thanks for the reply, I wish it did. We have an internal metrics system where we need to submit to. I am sure that the ways I've tried work with yarn deployment, but not with standalone.
Thanks, KP On Tue, Sep 6, 2016 at 11:36 PM, Benjamin Kim <bbuil...@gmail.com> wrote: > We use Graphite/Grafana for custom metrics. We found Spark’s metrics not > to be customizable. So, we write directly using Graphite’s API, which was > very easy to do using Java’s socket library in Scala. It works great for > us, and we are going one step further using Sensu to alert us if there is > an anomaly in the metrics beyond the norm. > > Hope this helps. > > Cheers, > Ben > > > On Sep 6, 2016, at 9:52 PM, map reduced <k3t.gi...@gmail.com> wrote: > > Hi, anyone has any ideas please? > > On Mon, Sep 5, 2016 at 8:30 PM, map reduced <k3t.gi...@gmail.com> wrote: > >> Hi, >> >> I've written my custom metrics source/sink for my Spark streaming app and >> I am trying to initialize it from metrics.properties - but that doesn't >> work from executors. I don't have control on the machines in Spark cluster, >> so I can't copy properties file in $SPARK_HOME/conf/ in the cluster. I have >> it in the fat jar where my app lives, but by the time my fat jar is >> downloaded on worker nodes in cluster, executors are already started and >> their Metrics system is already initialized - thus not picking my file with >> custom source configuration in it. >> >> Following this post >> <https://stackoverflow.com/questions/38924581/spark-metrics-how-to-access-executor-and-worker-data>, >> I've specified 'spark.files >> <https://spark.apache.org/docs/latest/configuration.html> = >> metrics.properties' and 'spark.metrics.conf=metrics.properties' but by >> the time 'metrics.properties' is shipped to executors, their metric system >> is already initialized. >> >> If I initialize my own metrics system, it's picking up my file but then >> I'm missing master/executor level metrics/properties (eg. >> executor.sink.mySink.propName=myProp - can't read 'propName' from >> 'mySink') since they are initialized >> <https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/master/Master.scala#L84> >> by >> Spark's metric system. >> >> Is there a (programmatic) way to have 'metrics.properties' shipped before >> executors initialize >> <https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkEnv.scala#L335> >> ? >> >> Here's my SO question >> <https://stackoverflow.com/questions/39340080/spark-metrics-custom-source-sink-configurations-not-getting-recognized> >> . >> >> Thanks, >> >> KP >> > > >
