Hi, My impression is to use StreamingListener to track metrics and react appropriately.
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.streaming.scheduler.StreamingListener Pozdrawiam, Jacek Laskowski ---- https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Mon, Jan 23, 2017 at 7:45 PM, Lan Jiang <ljia...@gmail.com> wrote: > Hi, there > > From the Spark UI, we can monitor the following two metrics: > > • Processing Time - The time to process each batch of data. > • Scheduling Delay - the time a batch waits in a queue for the > processing of previous batches to finish. > > However, what is the best way to monitor them proactively? For example, if > processing time/scheduling delay exceed certain threshold, send alert to the > admin/developer? > > Lan > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
