Re: Dynamic DAG use-case for spark streaming.

2015-09-29 Thread Tathagata Das
A very basic support that is there in DStream is DStream.transform() which take arbitrary RDD => RDD function. This function can actually choose to do different computation with time. That may be of help to you. On Tue, Sep 29, 2015 at 12:06 PM, Archit Thakur wrote: > Hi, > > We are using spark

Dynamic DAG use-case for spark streaming.

2015-09-29 Thread Archit Thakur
Hi, We are using spark streaming as our processing engine, and as part of output we want to push the data to UI. Now there would be multiple users accessing the system with there different filters on. Based on the filters and other inputs we want to either run a SQL Query on DStream or do a custo