This sounds like it might be a use case for something like a KeyedCoProcessFunction (or possibly a KeyedBroadcastProcessFunction, depending on the details). These operators can receive inputs from two different sources, and share state between them.
The rides and fares exercise [1] from the flink-training illustrates this pattern, which you can also read about in the tutorial on Connected Streams in the documentation [2]. If you need to broadcast the signal from the scheduler, see [3]. Regards, David [1] https://github.com/apache/flink-training/tree/master/rides-and-fares [2] https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/learn-flink/etl/#connected-streams [3] https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/fault-tolerance/broadcast_state/ On Tue, May 10, 2022 at 12:09 PM Sigalit Eliazov <e.siga...@gmail.com> wrote: > Hi all > i have 2 pipelines: > A. receives information from kafka and "holds" that info > B. a pipeline which is triggered by a scheduler and every x minutes should > send the info i received in pipeline A to another kafka topic > > As i understood i cannot use the flink state for this since these are > different pipelines/operators.. > is there a way to implement such a case in Flink itself without using any > external application like redis or db? > > Thanks > Sigalit > >
