Hi Mans What's the meaning of 'processor' you defined here? A user defined function?
When talking about share state, I'm afraid it's not so easy to implement in Flink. As no matter keyed state or operator state, they're both instantiated, used and only thread-safe in operator scope. The only way to read read-only state during runtime is via queryable state[1] For the question of keyBy, the message would only sent to one of task in downstream according to the hashcode [2]. [1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html [2] https://github.com/apache/flink/blob/7a6ca9c03f67f488e40a114e94c389a5cfb67836/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java#L58 Best Yun Tang ________________________________ From: M Singh <mans2si...@yahoo.com> Sent: Friday, January 10, 2020 23:29 To: User <user@flink.apache.org> Subject: Apache Flink - Sharing state in processors Hi: I have a few question about how state is shared in processors in Flink. 1. If I have a processor instantiated in the Flink app, and apply use in multiple times in the Flink - (a) if the tasks are in the same slot - do they share the same processor on the taskmanager ? (b) if the tasks are on same node but different slots - do they share the same processor on the taskmanager ? 2. If I instantiate a single processor with local state and use it in multiple times in Flink (a) if the tasks are in the same slot - do they share the same processor and state on the taskmanager ? (b) if the tasks are on same node but different slots - do they share the same processor and state on the taskmanager ? 3. If I instantiate a multiple processors with shared collection and use it in multiple times in Flink (a) if the tasks are in the same slot - do they share the state on the taskmanager ? (b) if the tasks are on same node but different slots - do they share the state on the taskmanager ? 4. How do the above scenarios affect sharing (a) operator state (b) keyed state 5. If I have have a parallelism of > 1, and use keyBy - is each key handled by only one instance of the processor ? I believe so, but wanted to confirm. Thanks Mans
