Re: Custom Partitioning for Keyed Streams

2018-01-10 Thread Piotr Nowojski
Hi, I don’t think it is possible to enforce scheduling of two keys to different nodes, since all of that is based on hashes. For some cases, doing the pre-aggregation step (initial aggregation done before keyBy, which is followed by final aggregation after the keyBy) can be the solution for ha

Custom Partitioning for Keyed Streams

2018-01-09 Thread Martin, Nick
Have a set of stateful operators that rely on keyed state. There is substantial skew between keys (i.e. there will be 100 messages on keys A and B, and 10 messages each on keys C-J), and key selection assignment is dictated by the needs of my application such that I can't choose keys in a way th