Hello, To improve performance we have " keyed state" in the operator's memory, basically we keep a Map which contains the state per each of the keys. The problem comes when we want to restore the state after a failure or after rescaling the operator. What we are doing is sending the concatenation of all the state to every operator using an union redistribution and then we restore the "in memory state" every time we see a new key. Then, after a while, we just clear the redistributed state. This is somewhat complex and prone to errors so we would like to find an alternative way of doing this.
As far as I know Flink knows which keys belong to each operator (distributing key groups) so I guess it would be possible to somehow calculate the key id from each of the stored keys and restore the in memory state at once if we could access to the key groups mapping. Is that possible? We could patch Flink if necessary to access that information. Thanks, Gerard -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
