On 2018/09/12 20:42:22, Ken Krugler <kkrugler_li...@transpac.com> wrote:
> Hi Bhaskar,
>
> I assume you don’t have 1000 streams, but rather one (keyed) stream with 1000
> different key values, yes?
>
> If so, then this one stream is physically partitioned based on the
> parallelism of the operator following the keyBy(), not per unique key.
>
> The most common per-key “resource” is the memory required for each key's
> state, if you’ve got any operations that need to maintain state
> (accumulators, windows, etc).
>
> For 1000 unique keys, this should be negligible.
>
> — Ken
>
>
> > On Sep 12, 2018, at 9:55 AM, bhaskar.eba...@gmail.com
> > <mailto:bhaskar.eba...@gmail.com> wrote:
> >
> > Hi
> >
> > I have created a KeyedStream with state as explained below
> > For example i have created 1000 streams, out of which 50% of streams data
> > is going to come once in 8 hours. Will the resources of these under
> > utilized streams are idle for that duration? Or Flink internal task manager
> > is having some strategy to utilize them for other new streams that are
> > coming?
> > Regards
> > Bhaskar
>
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com <http://www.scaleunlimited.com/>
> Custom big data solutions & training
> Flink, Solr, Hadoop, Cascading & Cassan
>
>
Hi Ken
As per documentation it is showing:
https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/
On DataStream if we apply KeyBy then output is KeyedStream. Once its stream
means it should execute in parallel right? There will be 1000 streams each is
having Keyed State. What you are saying is the main over head here is only
memory. That means Does these 1000 streams are going to be run across 1000 task
slots in parallel? These 1000 task slots is the main memory over head? Even
50% of them idle its not harm?
Please clarify
Regards
Bhaskar
