Re: Kafka connect rebalance problem

2017-10-17 Thread Dhawan Gajendran
Thank you for the reply. I have 7 kafka s3 connectors. Each connector has been configured with 4 tasks. However, I am trying to add more tasks for a few s3 connectors. When I try to change the configuration of these connectors using the rest API, I see the worker rebalance error thrown from the R

Re: Kafka connect rebalance problem

2017-10-17 Thread Ted Yu
bq. There are 7 connectors configured with each connector configured with 8 tasks (Averaging about 4 tasks per connector) Pardon. I don't quite understand the above setup. Can you describe in more detail ? Which version of connector are you using ? Cheers On Mon, Oct 16, 2017 at 11:30 PM, Dhawa

Kafka connect rebalance problem

2017-10-16 Thread Dhawan Gajendran
Hi All, We had done a POC with kafka s3 connect confluent platform and have been trying to move the setup to production. In the current setup we have 7 kafka topics and each topic drains the messages to a s3 bucket using s3 confluent connect. My setup: There are 8 workers distributed across 8 ubu