Question regarding state cleaning using timer

2018-09-14 Thread bhaskar . ebay77
Hi In the following example given in flink: object ExampleCountWindowAverage extends App { val env = StreamExecutionEnvironment.getExecutionEnvironment env.fromCollection(List( (1L, 3L), (1L, 5L), (1L, 7L), (1L, 4L), (1L, 2L) )).keyBy(_._1) .flatMap(new CountWindowAve

Question Regarding Streaming pipeline

2018-09-13 Thread bhaskar . ebay77
Hi In the streaming pipe line of Scala we want to use intermediary Machine Learning module written in python, Is it possible to use it ? Or entire pipe line should be either Python or Scala, we can't mix? Regards Bhaskar

Re: Question regarding Streaming Resources

2018-09-13 Thread bhaskar . ebay77
On 2018/09/13 03:30:28, Ken Krugler wrote: > Hi Bhaskar, > > > On 2018/09/12 20:42:22, Ken Krugler wrote: > >> Hi Bhaskar, > >> > >> I assume you don’t have 1000 streams, but rather one (keyed) stream with > >> 1000 different key values, yes? > >> > >> If so, then this one stream is phys

Re: Question regarding Streaming Resources

2018-09-12 Thread bhaskar . ebay77
On 2018/09/12 20:42:22, Ken Krugler wrote: > Hi Bhaskar, > > I assume you don’t have 1000 streams, but rather one (keyed) stream with 1000 > different key values, yes? > > If so, then this one stream is physically partitioned based on the > parallelism of the operator following the keyBy()

Re: Question regarding Streaming Resources

2018-09-12 Thread bhaskar . ebay77
On 2018/09/12 16:55:09, bhaskar.eba...@gmail.com wrote: > Hi > > I have created a KeyedStream with state as explained below > For example i have created 1000 streams, out of which 50% of streams data is > going to come once in 8 hours. Will the resources of these under utilized > streams

Question regarding Streaming Resources

2018-09-12 Thread bhaskar . ebay77
Hi I have created a KeyedStream with state as explained below For example i have created 1000 streams, out of which 50% of streams data is going to come once in 8 hours. Will the resources of these under utilized streams are idle for that duration? Or Flink internal task manager is having some