Kudos to Nikhil. your explanation adds to my knowledge. 🙏
On Mon, 25 Sep 2023 at 7:16 PM, Nikhil Srivastava < nikhilsrivastava4...@gmail.com> wrote: > Hi Yeikel, > > Sharing my two cents. Would let others chime in to add to this. > > Based on my understanding, if connect workers (which are all part of the > same cluster) can communicate with the kafka brokers (which happens to be > the Group Coordinator and facilitates Connect Leader Election via Group > Membership Protocol), then only 1 connect worker will be elected as leader > amongst all others in the cluster. Outside of that, I believe a bunch of > REST calls to connect workers are forwarded to the connect leader (if the > REST request lands on a connect worker which isn't a leader). In case of a > non-retriable network partition between the non-leader worker and leader > worker, those REST requests will fail. I'm referring to REST requests like > CREATE / UPDATE / DELETE. > > Hope this helps a little. > > Thanks, > -Nikhil > > On Sun, 24 Sept 2023 at 06:36, Yeikel Santana <em...@yeikel.com> wrote: > > > Hello everyone,I'm currently designing a new Kafka Connect cluster, and > > I'm trying to understand how connectivity functions among workers.In my > > setup, I have a single Kafka Connect cluster connected to the same Kafka > > topics and Kafka cluster. However, the workers are deployed in > > geographically separated data centers, each of which is fully isolated at > > the networkI suspect that this setup might not work with Kafka Connect > > because my current understanding is that ALL workers need to communicate > > with the leader for task coordination and heartbeats.In terms of leader > > election, can this result in multiple leaders and other potential > > issues?Any input and suggestions would be appreciated >
