You are right that probably the best solution would be to be able to use
different state backends for different operators, I hope it gets
implemented at some point. Meanwhile I'll take a look at the methods in
org.apache.flink.runtime.state.KeyGroupRangeAssignment, maybe I can find a
workaround goo
Hi,
ok, now I understand your goal a bit better. If would still like to point out
that it may take a bit more than it looks like. Just to name one example, you
probably also want to support asynchronous snapshots which is most likely
difficult when using just a hashmap. I think the proper solut
Hi Stefan, thanks
Yes, we are also using keyed state in other operators the problem is that
serialization is quite expensive and in some of them we would prefer to
avoid it by storing the state in memory (for our use case one specific
operator with in memory state gives at least a 30% throughput i
Hi,
from what I read, I get the impression that you attempt to implement you own
"keyed state" with a hashmap? Why not using the keyed state that is already
provided by Flink and gives you efficient rescaling etc. out of the box? Please
see [1] for the details.
[1]
https://ci.apache.org/proje
Hello,
To improve performance we have " keyed state" in the operator's memory,
basically we keep a Map which contains the state per each of the keys. The
problem comes when we want to restore the state after a failure or after
rescaling the operator. What we are doing is sending the concatenation
