Re: Question regarding Streaming Resources

2018-09-13 Thread bhaskar . ebay77
On 2018/09/13 03:30:28, Ken Krugler wrote: > Hi Bhaskar, > > > On 2018/09/12 20:42:22, Ken Krugler wrote: > >> Hi Bhaskar, > >> > >> I assume you don’t have 1000 streams, but rather one (keyed) stream with > >> 1000 different key values, yes? > >> > >> If so, then this one stream is phys

Re: Question regarding Streaming Resources

2018-09-12 Thread Ken Krugler
Hi Bhaskar, > On 2018/09/12 20:42:22, Ken Krugler wrote: >> Hi Bhaskar, >> >> I assume you don’t have 1000 streams, but rather one (keyed) stream with >> 1000 different key values, yes? >> >> If so, then this one stream is physically partitioned based on the >> parallelism of the operator fo

Re: Question regarding Streaming Resources

2018-09-12 Thread bhaskar . ebay77
On 2018/09/12 20:42:22, Ken Krugler wrote: > Hi Bhaskar, > > I assume you don’t have 1000 streams, but rather one (keyed) stream with 1000 > different key values, yes? > > If so, then this one stream is physically partitioned based on the > parallelism of the operator following the keyBy()

Re: Question regarding Streaming Resources

2018-09-12 Thread Ken Krugler
Hi Bhaskar, I assume you don’t have 1000 streams, but rather one (keyed) stream with 1000 different key values, yes? If so, then this one stream is physically partitioned based on the parallelism of the operator following the keyBy(), not per unique key. The most common per-key “resource” is t

Re: Question regarding Streaming Resources

2018-09-12 Thread bhaskar . ebay77
On 2018/09/12 16:55:09, bhaskar.eba...@gmail.com wrote: > Hi > > I have created a KeyedStream with state as explained below > For example i have created 1000 streams, out of which 50% of streams data is > going to come once in 8 hours. Will the resources of these under utilized > streams

Question regarding Streaming Resources

2018-09-12 Thread bhaskar . ebay77
Hi I have created a KeyedStream with state as explained below For example i have created 1000 streams, out of which 50% of streams data is going to come once in 8 hours. Will the resources of these under utilized streams are idle for that duration? Or Flink internal task manager is having some