Re: Monitoring the User Metrics for a long running Spark Job

Thanks a lot Sonal.. I will give it a try. Regards Sumit Chawla On Wed, Dec 7, 2016 at 10:45 PM, Sonal Goyal wrote: > You can try updating metrics.properties for the sink of your choice. In > our case, we add the following for getting application metrics in JSON > format using http > > *.sink.

Re: Monitoring the User Metrics for a long running Spark Job

You can try updating metrics.properties for the sink of your choice. In our case, we add the following for getting application metrics in JSON format using http *.sink.reifier.class= org.apache.spark.metrics.sink.MetricsServlet Here, we have defined the sink with name reifier and its class is the

Re: Monitoring the User Metrics for a long running Spark Job

Any pointers on this? Regards Sumit Chawla On Mon, Dec 5, 2016 at 8:30 PM, Chawla,Sumit wrote: > An example implementation i found is : https://github.com/groupon/ > spark-metrics > > Anyone has any experience using this? I am more interested in something > for Pyspark specifically. > > The a

Re: Monitoring the User Metrics for a long running Spark Job

An example implementation i found is : https://github.com/groupon/spark-metrics Anyone has any experience using this? I am more interested in something for Pyspark specifically. The above link pointed to - https://github.com/apache/spark/blob/master/conf/metrics.properties.template. I need to sp

Re: Monitoring the User Metrics for a long running Spark Job

One thing I've done before is to install datadogs statsd agent on the nodes. Then you can emit metrics and stats to it and build dashboards on datadog. Sent from my iPhone > On Dec 5, 2016, at 8:17 PM, Chawla,Sumit wrote: > > Hi Manish > > I am specifically looking for something similar to f

Re: Monitoring the User Metrics for a long running Spark Job

Hi Manish I am specifically looking for something similar to following: https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/common/index.html#accumulators--counters. Flink has this concept of Accumulators, where user can keep its custom counters etc. While the application is exec

Re: Monitoring the User Metrics for a long running Spark Job

http://spark.apache.org/docs/latest/monitoring.html You can even install tools like dstat , iostat , and iotop , *collectd* can provide fine-grained profiling on individual nodes. If you