Thank you for the reply. I have 7 kafka s3 connectors. Each connector has been configured with 4 tasks. However, I am trying to add more tasks for a few s3 connectors. When I try to change the configuration of these connectors using the rest API, I see the worker rebalance error thrown from the REST endpoint.
I am using confluent oss 2.11 version: S3 connect version of 3.3.0, kafka v 11.0.1, and jdk 1.8. On Oct 17, 2017 1:37 AM, "Ted Yu" <yuzhih...@gmail.com> wrote: > bq. There are 7 connectors configured with each connector configured with 8 > tasks (Averaging about 4 tasks per connector) > > Pardon. I don't quite understand the above setup. Can you describe in more > detail ? > > Which version of connector are you using ? > > Cheers > > On Mon, Oct 16, 2017 at 11:30 PM, Dhawan Gajendran < > dhawan.gajend...@datavisor.com> wrote: > > > Hi All, > > > > We had done a POC with kafka s3 connect confluent platform and have been > > trying to move the setup to production. In the current setup we have 7 > > kafka topics and each topic drains the messages to a s3 bucket using s3 > > confluent connect. > > > > My setup: > > There are 8 workers distributed across 8 ubuntu JVM instances. There are > 7 > > connectors configured with each connector configured with 8 tasks > > (Averaging about 4 tasks per connector). Each connector consumes messages > > from an unique kafka topic and writes these messages to an s3 bucket > using > > the HourlyParitioner class. > > > > We recently lost two of our workers and were replaced with two new > workers, > > which were assigned the same group.id. > > However after bringing up the new worker, all connectors started > responding > > with a HTTP 409 error code stating that the workers are rebalancing. I am > > unable to update the configurations of my connectors. > > > > The system has been continuously trying to rebalance for over 12 hours to > > no awhile. > > > > My questions: > > 1.) How do I debug my current system state? Can I look at some logs to > see > > what's happening and how can I fix this issue. > > > > I have read through these resources: https://docs. > > confluent.io/current/connect/design.html and https://docs. > > confluent.io/current/connect/concepts.html. However, I still do not > > understand how to debug or understand the core problem in my specific > case. > > I do realize there seems to be a rebalancing problem occuring when I > bring > > up a new worker or add a new task to a connector, but what I do not > > understand is how do I stop this rebalancing or force a rebalance or > reset > > of the system to the new setup. > > > > *Current commands I use for debugging my setup:* > > curl -XGET localhost:8083/connectors/<connector>/config > > curl -XGET localhost:8083/connectors/<connector>/status > > > > > > Appended the following into my connect-log4j.properties file: > > ====== > > log4j.appender.stdout=org.apache.log4j.ConsoleAppender > > log4j.appender.stdout.layout=org.apache.log4j.PatternLayout > > log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c:%L)%n > > ======== > > > > Is there anything else I could do to give me more visibility into my > > system? > > > > Thanks > > >
