That sounds great. Thanks. Can I assume that source for a stream in spark can only be some external source like kafka etc.? Source cannot be some rdd in spark or some external file ?
Thanks, Udbhav From: ayan guha [mailto:guha.a...@gmail.com] Sent: Friday, September 16, 2016 3:01 AM To: Udbhav Agarwal <udbhav.agar...@syncoms.com> Cc: user <user@spark.apache.org> Subject: RE: Spark processing Multiple Streams from a single stream You may consider writing back to Kafka from main stream and then have downstream consumers. This will keep things modular and independent. On 15 Sep 2016 23:29, "Udbhav Agarwal" <udbhav.agar...@syncoms.com<mailto:udbhav.agar...@syncoms.com>> wrote: Thank you Ayan for a reply. Source is kafka but I am reading from this source in my main stream. I will perform some operations here. Then I want to send the output of these operation to 4 parallel tasks. For these 4 parallel tasks I want 4 new streams. Is such an implementation possible here ? Thanks, Udbhav From: ayan guha [mailto:guha.a...@gmail.com<mailto:guha.a...@gmail.com>] Sent: Thursday, September 15, 2016 6:43 PM To: Udbhav Agarwal <udbhav.agar...@syncoms.com<mailto:udbhav.agar...@syncoms.com>> Cc: user <user@spark.apache.org<mailto:user@spark.apache.org>> Subject: Re: Spark processing Multiple Streams from a single stream Depending on source. For example, if source is Kafka then you can write 4 streaming consumers. On 15 Sep 2016 20:11, "Udbhav Agarwal" <udbhav.agar...@syncoms.com<mailto:udbhav.agar...@syncoms.com>> wrote: Hi All, I have a scenario where I want to process a message in various ways in parallel. For instance a message is coming inside spark stream(DStream) and I want to send this message to 4 different tasks in parallel. I want these 4 different tasks to be separate streams in the original spark stream and are always active and waiting for input. Can I implement such a process with spark streaming ? How ? Thanks in advance. Thanks, Udbhav Agarwal
