Hi, Can you elaborate a bit more on what your use case is and what you're trying to achieve?
Best regards, Martijn On Fri, 15 Oct 2021 at 11:25, Akshay Agarwal <akshay.agar...@grofers.com> wrote: > Hey, > > I have few doubts about HybridSource, it says: > >> To arrange multiple sources in a HybridSource, all sources except the >> last one need to be bounded. Therefore, the sources typically need to be >> assigned a start and end position. > > Both sources that I use in the job are unbounded, (file source and Kafka > source both get continuous updates). Further schema of both sources is > different since they give different data points. So wanted to clarify that > further you are suggesting converting Kafka source to hybrid source(file > source and Kafka source) and update state based on that and keep file > source as it is. > > *Akshay Agarwal* > > > On Fri, Oct 15, 2021 at 2:06 PM Akshay Agarwal <akshay.agar...@grofers.com> > wrote: > >> Nope, I haven't gone through that, will take a look, thanks for the >> prompt reply. >> >> *Akshay Agarwal* >> >> >> >> On Fri, Oct 15, 2021 at 2:00 PM Martijn Visser <mart...@ververica.com> >> wrote: >> >>> Hi, >>> >>> Have you checked out the Hybrid Source? [1] >>> >>> Thanks, Martijn >>> >>> [1] >>> https://ci.apache.org/projects/flink/flink-docs-master/docs/connectors/datastream/hybridsource/ >>> >>> On Fri, 15 Oct 2021 at 10:22, Akshay Agarwal <akshay.agar...@grofers.com> >>> wrote: >>> >>>> Hi Team, >>>> >>>> I have a streaming job that creates an enriched stream from streams of >>>> 2 different sources like one from files in object storage and another one >>>> Kafka. But since both of these are from different sources, events from them >>>> are read at different moments based on their performance. For example >>>> events from Kafka arrives first than from file storage events and even >>>> though I have used event watermarks, thus at the restart of jobs, enriched >>>> events are wrong and it gets corrected after file events start arriving. I >>>> tried to search for setting up synchronization between different sources in >>>> flink but did not found any blog/material. It might be a noob question, but >>>> if you guys have built something around this, could you let me know. >>>> >>>> Regards >>>> *Akshay Agarwal* >>>> >>>> [image: https://grofers.com] <https://grofers.com> >>> >>> > [image: https://grofers.com] <https://grofers.com>
