Re: Synchronisation between different sources in flink

2021-10-15 Thread Akshay Agarwal
Hey, The use case is as follows: Kafka is the source for our frontend/impression events like any user activity and what we are doing is enriching that stream some values from the backend which is the source of certain slowly changing properties(that changes/updated over time) values, it gets dumpe

Re: Synchronisation between different sources in flink

2021-10-15 Thread Martijn Visser
Hi, Can you elaborate a bit more on what your use case is and what you're trying to achieve? Best regards, Martijn On Fri, 15 Oct 2021 at 11:25, Akshay Agarwal wrote: > Hey, > > I have few doubts about HybridSource, it says: > >> To arrange multiple sources in a HybridSource, all sources exce

Re: Synchronisation between different sources in flink

2021-10-15 Thread Akshay Agarwal
Hey, I have few doubts about HybridSource, it says: > To arrange multiple sources in a HybridSource, all sources except the > last one need to be bounded. Therefore, the sources typically need to be > assigned a start and end position. Both sources that I use in the job are unbounded, (file sour

Re: Synchronisation between different sources in flink

2021-10-15 Thread Akshay Agarwal
Nope, I haven't gone through that, will take a look, thanks for the prompt reply. *Akshay Agarwal* On Fri, Oct 15, 2021 at 2:00 PM Martijn Visser wrote: > Hi, > > Have you checked out the Hybrid Source? [1] > > Thanks, Martijn > > [1] > https://ci.apache.org/projects/flink/flink-docs-master/d

Re: Synchronisation between different sources in flink

2021-10-15 Thread Martijn Visser
Hi, Have you checked out the Hybrid Source? [1] Thanks, Martijn [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/connectors/datastream/hybridsource/ On Fri, 15 Oct 2021 at 10:22, Akshay Agarwal wrote: > Hi Team, > > I have a streaming job that creates an enriched stream from st

Synchronisation between different sources in flink

2021-10-15 Thread Akshay Agarwal
Hi Team, I have a streaming job that creates an enriched stream from streams of 2 different sources like one from files in object storage and another one Kafka. But since both of these are from different sources, events from them are read at different moments based on their performance. For exampl