Re: Questions regarding Key Managed state

2020-04-03 Thread Congxian Qiu
Hi Many keys can be in a single state(each state can have multiple key-group, and keys will be assigned to the right key-group) If you write a custom process function that uses a state you created, then there is only one user state in that instance(do not count the underlying state of Flink if th

Re: Questions regarding Key Managed state

2020-04-03 Thread KristoffSC
Thank you for your answers. I have one more question. The Key Managed state for Keyed stream is per key or per operator? For example I have a keyed stream that is processed by MyProcessFunction with parallelism = 3. So I have three instances of MyProcessFuntion. The process function has a KeyMa

Re: Questions regarding Key Managed state

2020-04-02 Thread Timo Walther
Hi Kristoff, case 1: first of all Flink groups keys internally into so-called "key groups" for reducing the management overhead. The maximum parallelism decides about the number of key groups. When performing a rescale, the key groups are basically distributed using some consistent hashing al

Questions regarding Key Managed state

2020-04-02 Thread KristoffSC
Hi I have few question regarding Flink's state. Lets say we have: Case 1. stream.keybBy(...).process(myProcessFunction).parallelism(3). MyProcessFucntion uses a managed state (mapState, ListState etc). I'm using state checkpoints. Flink will redistribute events across 3 instances of myProcessF