Hello everybody, Just to let you know that I have added a chapter about having multiple containers (multiple pods for Kubernetes) running the same application : https://cwiki.apache.org/confluence/display/KAFKA/KIP-874%3A+TopicRoundRobinAssignor#KIP874:TopicRoundRobinAssignor-Howdoesitworkifwehavemultiplecontainersrunningthesameapplication ?
Regards, Mathieu Le mar. 11 oct. 2022 à 20:00, Mathieu Amblard <mathieu.ambl...@gmail.com> a écrit : > Hi Hector, > > > > First, thank you for your questions ! > > > > *If the goal is to do the partition assignments at a topic level, wouldn't > having single-partition topics solve this problem?* > > > > We are in a microservices environment; therefore, we can have multiple > containers running the same application. > > Using the TopicRoundRobinAssignor, the partitions are uniformly balanced > to each container, and we can get better performances. > > Let suppose there are 2 instances of the same application A0 and A1, 2 > consumers C0 and C1, and two topics t0 and t1. t0 has 3 partitions and t1 > has two partitions resulting in partitions : t0p0, t0p1, t0p2, t1p0, t1p1. > > If we use the TopicRoundRobinAssignor, the assignment will be : > > A0 : [ C0: [t0p0, t0p2], C1: [t1p0] ] > > A1 : [ C0: [t0p1], C1: [t1p1] ] > > > > *How will the group leader know that T2 should not be re-assigned on the > next rebalance? Can you elaborate a bit more on the mechanisms used to > communicate this state to the other group members?* > > > > Currently, the group leader will not know that T2 should not be > re-assigned on the next balance. > > For this first iteration, we simply keep in memory that T2 has a poison > pill and therefore we ignore all incoming messages from T2. We basically > consume them without acknowledging them. > > As you can imagine, in the case of having multiple instances of the same > application, in case of error, the partition will be rebalanced to another > instance. > > Nevertheless, this is not really a problem (at least for our use cases), > as soon as the poison pill is consumed, the consumer of this other instance > will be stopped, and so on, and so on. It will take a few tens of seconds > before the last consumer of the poison pill will be stopped and so the > consumption of the entire topic. > > For a second iteration, we have planned to find a solution to avoid this > time lapse between the consumer of the first instance being stopped and the > last one. Currently, I do not have a solution, but we are thinking about > different options, avoiding rebalancing partitions containing a poison pill > is one of them. > > > > Cheers, > > Mathieu > > Le ven. 7 oct. 2022 à 16:54, Hector Geraldino (BLOOMBERG/ 919 3RD A) < > hgerald...@bloomberg.net> a écrit : > >> Hi Mathieu. I took a look at your KIP and have a couple questions. >> >> If the goal is to do the partition assignments at a topic level, wouldn't >> having single-partition topics solve this problem? >> >> You also mentioned that your goal is to minimize the potential of a >> poison pill message breaking all members of a group (by keeping track of >> which topics have 'failed'), but it is not clear how this can be achieved >> with this assignor. If we imagine an scenario where: >> >> * A group has 3 members (A, B, C) >> * Members are subscribed to 3 topics (T1, T2, T3) >> * Each member is assigned one topic (A[T1], B[T2], C[T3]) >> * One member fails to consume from a topic/partition (B[T2]), and goes >> into failed state >> >> How will the group leader know that T2 should not be re-assigned on the >> next rebalance? Can you elaborate a bit more on the mechanisms used to >> communicate this state to the other group members? >> >> Thanks >> >> From: dev@kafka.apache.org At: 10/05/22 03:47:33 UTC-4:00To: >> dev@kafka.apache.org >> Subject: [DISCUSS] KIP-874: TopicRoundRobinAssignor >> >> Hi Kafka Developers, >> >> My proposal is to add a new partition assignment strategy at the topic >> level to : >> - have a better data consistency by consumed topic in case of exception >> - have a solution much thread safe for the consumer >> In case there are multiple consumers and multiple topics. >> >> Here is the link to the KIP with all the explanations : >> https://cwiki.apache.org/confluence/x/XozGDQ >> >> Thank you in advance for your feedbacks, >> Mathieu >> >> >>
