Hi Bhaskar, > On 2018/09/12 20:42:22, Ken Krugler <kkrugler_li...@transpac.com> wrote: >> Hi Bhaskar, >> >> I assume you don’t have 1000 streams, but rather one (keyed) stream with >> 1000 different key values, yes? >> >> If so, then this one stream is physically partitioned based on the >> parallelism of the operator following the keyBy(), not per unique key. >> >> The most common per-key “resource” is the memory required for each key's >> state, if you’ve got any operations that need to maintain state >> (accumulators, windows, etc). >> >> For 1000 unique keys, this should be negligible. >> >> — Ken >> >> >>> On Sep 12, 2018, at 9:55 AM, bhaskar.eba...@gmail.com >>> <mailto:bhaskar.eba...@gmail.com> wrote: >>> >>> Hi >>> >>> I have created a KeyedStream with state as explained below >>> For example i have created 1000 streams, out of which 50% of streams data >>> is going to come once in 8 hours. Will the resources of these under >>> utilized streams are idle for that duration? Or Flink internal task manager >>> is having some strategy to utilize them for other new streams that are >>> coming? >>> Regards >>> Bhaskar >> > Hi Ken > As per documentation it is showing: > https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/ > On DataStream if we apply KeyBy then output is KeyedStream.
Correct. > Once its stream means it should execute in parallel right? It will operate in parallel based on the parallelism of the downstream operations being applied to the KeyedStream. Number of unique keys has nothing to do with number of parallel (simultaneous) operators being used to process the KeyedStream. > There will be 1000 streams each is having Keyed State. “Stream” has a specific meaning in Flink, which I think you’re not using as intended here. > What you are saying is the main over head here is only memory. That means > Does these 1000 streams are going to be run across 1000 task slots in > parallel? These 1000 task slots is the main memory over head? Even 50% of > them idle its not harm? See above - you don’t have 1000 “task slots” and you don’t have 1000 stream. You have N operators running at the same time, where N is based on the parallelism that you set (either implicitly, or explicitly) for the operator(s) processing the KeyedStream. Note that If you have 1000 unique keys, and you’ve got (for example) a single ValueState per key, then you’d have 1000 states. But if you have say a sliding window, then the number of states per key can grow significantly, since each key can have multiple states (one per each “open window”). But also note that using RocksDB to handle state means that not all state has to be in memory at the same time, so you’ve got more room to scale. Regards, — Ken -------------------------- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com Custom big data solutions & training Flink, Solr, Hadoop, Cascading & Cassandra
