Re: Sorting/grouping keys and State management in BATCH mode

2022-01-12 Thread Dawid Wysakowicz
Hey Krzysztof, Re 1. I believe you are asking where the state is kept. It is stored in memory, but bear in mind there is only ever state kept for the current key. Once all records for a key are processed the corresponding state is discarded as it won't be needed anymore. Re 2. The sorting algorit

Re: Sorting/grouping keys and State management in BATCH mode

2022-01-11 Thread Chesnay Schepler
Looping in Dawid who can hopefully answer your questions. On 11/01/2022 13:00, Krzysztof Chmielewski wrote: Hi, Im reading docs and FLIP-140 available for BATCH mode [1][2] where it reads that " In |BATCH| mode, the configured state backend is ignored. Instead, the input of a keyed operation i

Sorting/grouping keys and State management in BATCH mode

2022-01-11 Thread Krzysztof Chmielewski
Hi, Im reading docs and FLIP-140 available for BATCH mode [1][2] where it reads that " In BATCH mode, the configured state backend is ignored. Instead, the input of a keyed operation is grouped by key (using sorting) and then we process all records of a key in turn." [1] I would like to ask: 1. W