Hey Mathieu, Apologies if you already know this, but the partition assignor interface is fully pluggable. That means you can plug in a custom PartitionAssignor implementation, without having to go through a KIP or commit any code to the AK repo.
I suspect some of the current unknowns and solutions will become clear when you start to actually write this assignor and test it out in your environment. Then you can play around with what works and what doesn't work, and come back to the KIP if desired with a stronger argument for why it's needed. Or you can just take your assignor and publish it in a public git repo for anyone who might have a similar use case as you. Just my two cents, I'd recommend in this case you start with the implementation before worrying about donating your assignor back to the main repo with a KIP. IF you do want to, it would then be much easier to convince people when they can see your assignor logic for themselves, and you'll be able to answer any questions. Best, Sophie On Fri, Oct 21, 2022 at 2:21 AM Mathieu Amblard <mathieu.ambl...@gmail.com> wrote: > Hello everybody, > > Just to let you know that I have added a chapter about having multiple > containers (multiple pods for Kubernetes) running the same application : > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-874%3A+TopicRoundRobinAssignor#KIP874:TopicRoundRobinAssignor-Howdoesitworkifwehavemultiplecontainersrunningthesameapplication > ? > > Regards, > Mathieu > > Le mar. 11 oct. 2022 à 20:00, Mathieu Amblard <mathieu.ambl...@gmail.com> > a > écrit : > > > Hi Hector, > > > > > > > > First, thank you for your questions ! > > > > > > > > *If the goal is to do the partition assignments at a topic level, > wouldn't > > having single-partition topics solve this problem?* > > > > > > > > We are in a microservices environment; therefore, we can have multiple > > containers running the same application. > > > > Using the TopicRoundRobinAssignor, the partitions are uniformly balanced > > to each container, and we can get better performances. > > > > Let suppose there are 2 instances of the same application A0 and A1, 2 > > consumers C0 and C1, and two topics t0 and t1. t0 has 3 partitions and t1 > > has two partitions resulting in partitions : t0p0, t0p1, t0p2, t1p0, > t1p1. > > > > If we use the TopicRoundRobinAssignor, the assignment will be : > > > > A0 : [ C0: [t0p0, t0p2], C1: [t1p0] ] > > > > A1 : [ C0: [t0p1], C1: [t1p1] ] > > > > > > > > *How will the group leader know that T2 should not be re-assigned on the > > next rebalance? Can you elaborate a bit more on the mechanisms used to > > communicate this state to the other group members?* > > > > > > > > Currently, the group leader will not know that T2 should not be > > re-assigned on the next balance. > > > > For this first iteration, we simply keep in memory that T2 has a poison > > pill and therefore we ignore all incoming messages from T2. We basically > > consume them without acknowledging them. > > > > As you can imagine, in the case of having multiple instances of the same > > application, in case of error, the partition will be rebalanced to > another > > instance. > > > > Nevertheless, this is not really a problem (at least for our use cases), > > as soon as the poison pill is consumed, the consumer of this other > instance > > will be stopped, and so on, and so on. It will take a few tens of seconds > > before the last consumer of the poison pill will be stopped and so the > > consumption of the entire topic. > > > > For a second iteration, we have planned to find a solution to avoid this > > time lapse between the consumer of the first instance being stopped and > the > > last one. Currently, I do not have a solution, but we are thinking about > > different options, avoiding rebalancing partitions containing a poison > pill > > is one of them. > > > > > > > > Cheers, > > > > Mathieu > > > > Le ven. 7 oct. 2022 à 16:54, Hector Geraldino (BLOOMBERG/ 919 3RD A) < > > hgerald...@bloomberg.net> a écrit : > > > >> Hi Mathieu. I took a look at your KIP and have a couple questions. > >> > >> If the goal is to do the partition assignments at a topic level, > wouldn't > >> having single-partition topics solve this problem? > >> > >> You also mentioned that your goal is to minimize the potential of a > >> poison pill message breaking all members of a group (by keeping track of > >> which topics have 'failed'), but it is not clear how this can be > achieved > >> with this assignor. If we imagine an scenario where: > >> > >> * A group has 3 members (A, B, C) > >> * Members are subscribed to 3 topics (T1, T2, T3) > >> * Each member is assigned one topic (A[T1], B[T2], C[T3]) > >> * One member fails to consume from a topic/partition (B[T2]), and goes > >> into failed state > >> > >> How will the group leader know that T2 should not be re-assigned on the > >> next rebalance? Can you elaborate a bit more on the mechanisms used to > >> communicate this state to the other group members? > >> > >> Thanks > >> > >> From: dev@kafka.apache.org At: 10/05/22 03:47:33 UTC-4:00To: > >> dev@kafka.apache.org > >> Subject: [DISCUSS] KIP-874: TopicRoundRobinAssignor > >> > >> Hi Kafka Developers, > >> > >> My proposal is to add a new partition assignment strategy at the topic > >> level to : > >> - have a better data consistency by consumed topic in case of exception > >> - have a solution much thread safe for the consumer > >> In case there are multiple consumers and multiple topics. > >> > >> Here is the link to the KIP with all the explanations : > >> https://cwiki.apache.org/confluence/x/XozGDQ > >> > >> Thank you in advance for your feedbacks, > >> Mathieu > >> > >> > >> >
