bq. There are 7 connectors configured with each connector configured with 8 tasks (Averaging about 4 tasks per connector)
Pardon. I don't quite understand the above setup. Can you describe in more detail ? Which version of connector are you using ? Cheers On Mon, Oct 16, 2017 at 11:30 PM, Dhawan Gajendran < dhawan.gajend...@datavisor.com> wrote: > Hi All, > > We had done a POC with kafka s3 connect confluent platform and have been > trying to move the setup to production. In the current setup we have 7 > kafka topics and each topic drains the messages to a s3 bucket using s3 > confluent connect. > > My setup: > There are 8 workers distributed across 8 ubuntu JVM instances. There are 7 > connectors configured with each connector configured with 8 tasks > (Averaging about 4 tasks per connector). Each connector consumes messages > from an unique kafka topic and writes these messages to an s3 bucket using > the HourlyParitioner class. > > We recently lost two of our workers and were replaced with two new workers, > which were assigned the same group.id. > However after bringing up the new worker, all connectors started responding > with a HTTP 409 error code stating that the workers are rebalancing. I am > unable to update the configurations of my connectors. > > The system has been continuously trying to rebalance for over 12 hours to > no awhile. > > My questions: > 1.) How do I debug my current system state? Can I look at some logs to see > what's happening and how can I fix this issue. > > I have read through these resources: https://docs. > confluent.io/current/connect/design.html and https://docs. > confluent.io/current/connect/concepts.html. However, I still do not > understand how to debug or understand the core problem in my specific case. > I do realize there seems to be a rebalancing problem occuring when I bring > up a new worker or add a new task to a connector, but what I do not > understand is how do I stop this rebalancing or force a rebalance or reset > of the system to the new setup. > > *Current commands I use for debugging my setup:* > curl -XGET localhost:8083/connectors/<connector>/config > curl -XGET localhost:8083/connectors/<connector>/status > > > Appended the following into my connect-log4j.properties file: > ====== > log4j.appender.stdout=org.apache.log4j.ConsoleAppender > log4j.appender.stdout.layout=org.apache.log4j.PatternLayout > log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c:%L)%n > ======== > > Is there anything else I could do to give me more visibility into my > system? > > Thanks >
