Hello fellow squirrels!
We just made a PR [1] of a prototype targeting flexible state management for streaming tasks with the prospect of further implementing on top different strategies such as lazy state updates, incremental snapshots and state partitioning. You can read more regarding the motivation behind this design in the doc Gyula created [2]. As described in the doc, "managed" operator state and updates are explicitly specified by the OperatorState abstraction. Furthermore, OperatorState can be partitioned by a key and retrieved by the getState method. If the state is partitioned the getState method implicitly returns the state of the respective key that is currently processed by the operator. It will still be possible to add functionality in order to access keys arbitrarily if needed but for now this looks hopefully clear enough. Let's discuss here how we can use/modify this api to fit our general needs and vision for state management. Paris [1] https://github.com/apache/flink/pull/747 [2] https://docs.google.com/document/d/1nTn4Tpafsnt-TCT6L1vlHtGGgRevU90yRsUQEmkRMjk ?
