[Spark Streaming] Are SparkListener/StreamingListener callbacks called concurrently?

2018-06-20 Thread Majid Azimi
Hi, What is the concurrency model behind SparkListener or StreamingListener callbacks? 1. Multiple threads might access callbacks simultaneously. 2. Callbacks are guaranteed to be executed by a single thread.(Thread ids might change on consecutive calls, though) I asked the same question on

Re: StreamingListener, anyone?

2015-06-04 Thread Dmitry Goldenberg
rs, - Dmitry On Thu, Jun 4, 2015 at 3:55 AM, Shixiong Zhu wrote: > You should not call `jssc.stop(true);` in a StreamingListener. It will > cause a dead-lock: `jssc.stop` won't return until `listenerBus` exits. But > since `jssc.stop` blocks `StreamingListener`, `listenerBus` cannot

Re: StreamingListener, anyone?

2015-06-04 Thread Shixiong Zhu
You should not call `jssc.stop(true);` in a StreamingListener. It will cause a dead-lock: `jssc.stop` won't return until `listenerBus` exits. But since `jssc.stop` blocks `StreamingListener`, `listenerBus` cannot exit. Best Regards, Shixiong Zhu 2015-06-04 0:39 GMT+08:00 dgoldenberg :

Re: StreamingListener, anyone?

2015-06-04 Thread Akhil Das
ming listener, like so: > > JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, >Durations.milliseconds(params.getBatchDurationMillis())); > jssc.addStreamingListener(new JobListener(jssc)); > > where JobListener is defined like so > private static cla

StreamingListener, anyone?

2015-06-03 Thread dgoldenberg
where JobListener is defined like so private static class JobListener implements StreamingListener { private JavaStreamingContext jssc; JobListener(JavaStreamingContext jssc) { this.jssc =

Re: StreamingListener

2015-03-11 Thread Akhil Das
At the end of foreachrdd i believe. Thanks Best Regards On Thu, Mar 12, 2015 at 6:48 AM, Corey Nolet wrote: > Given the following scenario: > > dstream.map(...).filter(...).window(...).foreachrdd() > > When would the onBatchCompleted fire? > >

StreamingListener

2015-03-11 Thread Corey Nolet
Given the following scenario: dstream.map(...).filter(...).window(...).foreachrdd() When would the onBatchCompleted fire?

Re: Example usage of StreamingListener

2014-12-04 Thread Hafiz Mujadid
Thanks Akhil You are so helping Dear. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Example-usage-of-StreamingListener-tp20357p20362.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Re: Example usage of StreamingListener

2014-12-04 Thread Akhil Das
Here you go http://stackoverflow.com/questions/20950268/how-to-stop-spark-streaming-context-when-the-network-connection-tcp-ip-is-clos Thanks Best Regards On Thu, Dec 4, 2014 at 4:30 PM, Hafiz Mujadid wrote: > Hi! > > does anybody has some useful example of StreamingListener interf

Example usage of StreamingListener

2014-12-04 Thread Hafiz Mujadid
Hi! does anybody has some useful example of StreamingListener interface. When and how can we use this interface to stop streaming when one batch of data is processed? Thanks alot -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Example-usage-of