RE: Re: [DISCUSS] Contribution of Multi Cluster Kafka Source

2022-09-02 Thread Ryan van Huuksloot via user
Hi Mason, First off, thanks for putting this FLIP together! Sorry for the delay. Full disclosure Mason and I chatted a little bit at Flink Forward 2022 but I have tried to capture the questions I had for him then. I'll start the conversation with a few questions: 1. The concept of streamIds is n

Re: [DISCUSS] Contribution of Multi Cluster Kafka Source

2022-07-14 Thread Mason Chen
Hi all, Circling back on this--I have created a first draft document in confluence: https://cwiki.apache.org/confluence/display/FLINK/FLIP-246%3A+Multi+Cluster+Kafka+Source . Looking forward to hear all your feedback in this email thread! Best, Mason On Thu, Jun 30, 2022 at 6:57 AM Thomas Weise

Re: [DISCUSS] Contribution of Multi Cluster Kafka Source

2022-06-30 Thread Thomas Weise
Hi Mason, I added mason6345 to the Flink confluence space, you should be able to add a FLIP now. Looking forward to the contribution! Thomas On Thu, Jun 30, 2022 at 9:25 AM Martijn Visser wrote: > > Hi Mason, > > I'm sure there's a PMC (*hint*) out there who can grant you access to > create a

Re: [DISCUSS] Contribution of Multi Cluster Kafka Source

2022-06-30 Thread Martijn Visser
Hi Mason, I'm sure there's a PMC (*hint*) out there who can grant you access to create a FLIP. Looking forward to it, this sounds like an improvement that users are looking forward to. Best regards, Martijn Op di 28 jun. 2022 om 09:21 schreef Mason Chen : > Hi all, > > Thanks for the feedback!

Re: [DISCUSS] Contribution of Multi Cluster Kafka Source

2022-06-28 Thread Mason Chen
Hi all, Thanks for the feedback! I'm adding the users, who responded in the user mailing list, to this thread. @Qingsheng - Yes, I would prefer to reuse the existing Kafka connector module. It makes a lot of sense since the dependencies are the same and the implementation can also extend and impr

Re: Re: [DISCUSS] Contribution of Multi Cluster Kafka Source

2022-06-27 Thread Andrew Otto
This sounds very useful! Another potential use case: - Consuming from multiple kafka clusters in different datacenters/regions. I'm not sure if we would ultimately want to do this, but having it as an option when you need events from multiple kafka clusters to get the full history of changes (in

RE: Re: [DISCUSS] Contribution of Multi Cluster Kafka Source

2022-06-27 Thread Ryan van Huuksloot
Hi Mason, Thanks for starting this discussion! The proposed Source sounds awesome and we would be interested in taking a look at the source code and evaluating our use cases. We can provide information and review on a potential FLIP based on other use cases. Do you have a fork/branch that you are