Hi all, I've faced a very strange situation when rebalancing happens. We have a kafka-connect cluster that manages around 20 Aws S3 sink connectors. Each connector has between 3-10 tasks. All these tasks are handled with 6 kube pods and lands data to S3.
A week ago we found that we missed data from the topic with the highest rate. Deep analytics showed that we missed data from whole partitions when the rebalancing round was triggered. I went through the logs and found out that the rebalancing round assigned empty partitions for several tasks instead of assigning the real one. This particular topic has 10 partitions and the situation looks like this: https://drive.google.com/file/d/1XYZJH8vlk-cCsuFXQ890sZj-hDG12SNA/view?usp=sharing The first question is why does kafka-connect work without certain partitions 0,3,5 as you can see on the most left assignment column on the picture? I feel like all this happens due to what I would call a rebalancing storm. Because we usually are in the deployment process we have around 12-15 rounds of rebalancing. Also, we found out that it happens with a topic that has the most input rate. but I don't understand why kafka-connect works very silently in this case. I mean without any exception or something like this. Could someone navigate me? Thank you in advance. Best regards, Bulanova Daria -- <https://www.hellofresh.com/jobs/?utm_medium=email&utm_source=email_signature> HelloFresh SE, Berlin (Sitz der Gesellschaft) | Vorstände: Dominik S. Richter (Vorsitzender), Thomas W. Griesel, Christian Gärtner, Edward Boyes | Vorsitzender des Aufsichtsrats: John H. Rittenhouse | Eingetragen beim Amtsgericht Charlottenburg, HRB 182382 B | USt-Id Nr.: DE 302210417 *CONFIDENTIALITY NOTICE:* This message (including any attachments) is confidential and may be privileged. It may be read, copied and used only by the intended recipient. If you have received it in error please contact the sender (by return e-mail) immediately and delete this message. Any unauthorized use or dissemination of this message in whole or in parts is strictly prohibited.
