Hi, I’m not sure what are you trying to achieve. What do you mean by “state of full outer join”? The result of it? Or it’s internal state? Also keep in mind, that internal state of the operators in Flink is already snapshoted/written down to an external storage during checkpointing mechanism.
The result should be simple, just write it to some Sink. For the internal state, it sounds like you are doing something not the way it was intended… having said that, you can try one of the following options: a) Implement your own outer join operator (might not be as easy if you are using Table API/SQL) and just create a side output for the state changes. b) Use state processor API to read the content of a savepoint/checkpoint [1][2] c) Use queryable state [3] (I’m not sure about this, I have never used queryable state) Piotrek [1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html <https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html> [2] https://flink.apache.org/feature/2019/09/13/state-processor-api.html <https://flink.apache.org/feature/2019/09/13/state-processor-api.html> [3] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html <https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html> > On 29 Oct 2019, at 16:42, kant kodali <kanth...@gmail.com> wrote: > > Hi All, > > I want to do a full outer join on two streaming data sources and store the > state of full outer join in some external storage like rocksdb or something > else. And then want to use this intermediate state as a streaming source > again, do some transformation and write it to some external store. is that > possible with Flink 1.9? > > Also what storage systems support push mechanism for the intermediate data? > For example, In the use case above does rocksdb support push/emit events in a > streaming fashion? > > Thanks!
