RE: Re: [DISCUSS] Contribution of Multi Cluster Kafka Source

2022-09-02 Thread Ryan van Huuksloot via user
Hi Mason, First off, thanks for putting this FLIP together! Sorry for the delay. Full disclosure Mason and I chatted a little bit at Flink Forward 2022 but I have tried to capture the questions I had for him then. I'll start the conversation with a few questions: 1. The concept of streamIds is n

Re: Re: [DISCUSS] Contribution of Multi Cluster Kafka Source

2022-06-27 Thread Andrew Otto
This sounds very useful! Another potential use case: - Consuming from multiple kafka clusters in different datacenters/regions. I'm not sure if we would ultimately want to do this, but having it as an option when you need events from multiple kafka clusters to get the full history of changes (in

RE: Re: [DISCUSS] Contribution of Multi Cluster Kafka Source

2022-06-27 Thread Ryan van Huuksloot
Hi Mason, Thanks for starting this discussion! The proposed Source sounds awesome and we would be interested in taking a look at the source code and evaluating our use cases. We can provide information and review on a potential FLIP based on other use cases. Do you have a fork/branch that you are