Hi all,

I've faced a very strange situation when rebalancing happens. We have a
kafka-connect cluster that manages around 20 Aws S3 sink connectors. Each
connector has between 3-10 tasks. All these tasks are handled with 6 kube
pods and lands data to S3.

A week ago we found that we missed data from the topic with the highest
rate. Deep analytics showed that we missed data from whole partitions when
the rebalancing round was triggered.

I went through the logs and found out that the rebalancing round assigned
empty partitions for several tasks instead of assigning the real one.

This particular topic has 10 partitions and the situation looks like this:
https://drive.google.com/file/d/1XYZJH8vlk-cCsuFXQ890sZj-hDG12SNA/view?usp=sharing

The first question is why does kafka-connect work without certain
partitions 0,3,5 as you can see on the most left assignment column on the
picture?

I feel like all this happens due to what I would call a rebalancing storm.
Because we usually are in the deployment process we have around 12-15
rounds of rebalancing. Also, we found out that it happens with a topic that
has the most input rate. but I don't understand why kafka-connect works
very silently in this case. I mean without any exception or something like
this.

 Could someone navigate me?
Thank you in advance.

Best regards,
Bulanova Daria

-- 




 
<https://www.hellofresh.com/jobs/?utm_medium=email&utm_source=email_signature>

HelloFresh SE, Berlin (Sitz der Gesellschaft) | Vorstände: Dominik S. 
Richter (Vorsitzender), Thomas W. Griesel, Christian Gärtner, Edward Boyes 
| Vorsitzender des Aufsichtsrats: John H. Rittenhouse | Eingetragen beim 
Amtsgericht Charlottenburg, HRB 182382 B | USt-Id Nr.: DE 302210417

*CONFIDENTIALITY NOTICE:* This message (including any attachments) is 
confidential and may be privileged. It may be read, copied and used only by 
the intended recipient. If you have received it in error please contact the 
sender (by return e-mail) immediately and delete this message. Any 
unauthorized use or dissemination of this message in whole or in parts is 
strictly prohibited.


Reply via email to