Re: Operator variables in memory scoped by key

2017-09-01 Thread gerardg
Thanks Aljoscha, So I can think of three possible solutions: * Use an instance dictionary to store the trie tree scoped by the same key that the KeyedStream. That should work if the lifetime of the operator object is tied to the keys that it processes. * Store the trie tree in a ValueState but so

Re: Operator variables in memory scoped by key

2017-08-31 Thread Aljoscha Krettek
Hi, I'm afraid that's not possible. One operator instance is in fact, as you suspected, responsible for processing elements with many different keys. Only if you use the keyed state abstractions (ValueState, MapState and such) is your state "per-key". Best, Aljoscha > On 31. Aug 2017, at 15:1

Operator variables in memory scoped by key

2017-08-31 Thread gerardg
I'm using a trie tree to match prefixes efficiently in an operator applied to a KeyedStream. It can grow quite large so I'd like to store the contents of the tree in a keyed MapState so I benefit from incremental checkpoints. Then, I'd just need to recreate the tree in memory from the MapState in c