Hi All, We had done a POC with kafka s3 connect confluent platform and have been trying to move the setup to production. In the current setup we have 7 kafka topics and each topic drains the messages to a s3 bucket using s3 confluent connect.
My setup: There are 8 workers distributed across 8 ubuntu JVM instances. There are 7 connectors configured with each connector configured with 8 tasks (Averaging about 4 tasks per connector). Each connector consumes messages from an unique kafka topic and writes these messages to an s3 bucket using the HourlyParitioner class. We recently lost two of our workers and were replaced with two new workers, which were assigned the same group.id. However after bringing up the new worker, all connectors started responding with a HTTP 409 error code stating that the workers are rebalancing. I am unable to update the configurations of my connectors. The system has been continuously trying to rebalance for over 12 hours to no awhile. My questions: 1.) How do I debug my current system state? Can I look at some logs to see what's happening and how can I fix this issue. I have read through these resources: https://docs. confluent.io/current/connect/design.html and https://docs. confluent.io/current/connect/concepts.html. However, I still do not understand how to debug or understand the core problem in my specific case. I do realize there seems to be a rebalancing problem occuring when I bring up a new worker or add a new task to a connector, but what I do not understand is how do I stop this rebalancing or force a rebalance or reset of the system to the new setup. *Current commands I use for debugging my setup:* curl -XGET localhost:8083/connectors/<connector>/config curl -XGET localhost:8083/connectors/<connector>/status Appended the following into my connect-log4j.properties file: ====== log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c:%L)%n ======== Is there anything else I could do to give me more visibility into my system? Thanks
