Connect uses Kafka's group coordination mechanism and normal consumers to balance/assign to the various sink tasks the topic partitions for the subscribed topics. A task will continue to receive the records from its assigned topic partitions until a rebalance occurs, which can happen when any connector config is added/changed/removed or when a worker adds/leaves the cluster.
On Sun, Sep 10, 2017 at 10:40 PM, Behrang Saeedzadeh <behran...@gmail.com> wrote: > Hi, > > How does Kafka Connect distribute records between workers for a sink > connector when the connector is only configured to get data from one topic? > > * Does it ensure all records in a given partition are sent to the same > worker instance? > * When a new worker is added to the cluster, what steps are taken by > Connect? In particular, if Connect unassigns some partitions from the given > workers and assigns them to this new worker, would it wait for all > in-flight records belonging to these partitions to be processed by the > existing workers? > > Best regards, > Behrang Saeedzadeh >
