Hi Mathieu, Thanks for the effort that you have put in creating this KIP. I just read it again and I am still confused by the use cases and the motivation. I suppose that this works for your data model but it does not seem to be a general pattern.
Overall, I stick to the comment that I made in the PR. I still feel like that this is something too use case specific to make it into Apache Kafka. Best, David Le mar. 25 oct. 2022 à 07:25, Mathieu Amblard <mathieu.ambl...@gmail.com> a écrit : > Hi Sophie, > > Thank you for your message. > > The TopicRoundRobinAssignor has been already implemented (because the > existing ones do not address our use cases correctly), it implements the > interface you mentionned, and I have made a PR to the Kafka repo where you > can see the source code. It was in a comment of that PR that someone from > the core team proposes to create a KIP. > > I have also fully unit tested this assignor (see the PR). Moreover, around > 20 microservices are using it in production, for 6 months now, in the > company I am working for. So I think it has been proven and that's why I > have made this proposal. I would never have dared to suggest something I > have never tested and if there is no need for it. > > Finally, this assignor suits well our Kafka architecture and use cases > contrary to the existing ones. For reminder, we are using Kafka as a > messaging system (no streaming) and we can have only one type of message > per topic. Maybe it is too specific, maybe not, that's also why I have made > this KIP, to challenge it, to see if someone has the same needs and if this > assignor can help others. If not, that's not a problem, I will simply keep > it in our source code. > > Regards, > Mathieu > > > Le mar. 25 oct. 2022 à 01:51, Sophie Blee-Goldman > <sop...@confluent.io.invalid> a écrit : > > > Hey Mathieu, > > > > Apologies if you already know this, but the partition assignor interface > is > > fully pluggable. > > That means you can plug in a custom PartitionAssignor implementation, > > without > > having to go through a KIP or commit any code to the AK repo. > > > > I suspect some of the current unknowns and solutions will become clear > when > > you start > > to actually write this assignor and test it out in your environment. Then > > you can play around > > with what works and what doesn't work, and come back to the KIP if > desired > > with a stronger > > argument for why it's needed. Or you can just take your assignor and > > publish it in a public > > git repo for anyone who might have a similar use case as you. > > > > Just my two cents, I'd recommend in this case you start with the > > implementation before > > worrying about donating your assignor back to the main repo with a KIP. > IF > > you do want to, > > it would then be much easier to convince people when they can see your > > assignor logic > > for themselves, and you'll be able to answer any questions. > > > > Best, > > Sophie > > > > On Fri, Oct 21, 2022 at 2:21 AM Mathieu Amblard < > mathieu.ambl...@gmail.com > > > > > wrote: > > > > > Hello everybody, > > > > > > Just to let you know that I have added a chapter about having multiple > > > containers (multiple pods for Kubernetes) running the same application > : > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-874%3A+TopicRoundRobinAssignor#KIP874:TopicRoundRobinAssignor-Howdoesitworkifwehavemultiplecontainersrunningthesameapplication > > > ? > > > > > > Regards, > > > Mathieu > > > > > > Le mar. 11 oct. 2022 à 20:00, Mathieu Amblard < > mathieu.ambl...@gmail.com > > > > > > a > > > écrit : > > > > > > > Hi Hector, > > > > > > > > > > > > > > > > First, thank you for your questions ! > > > > > > > > > > > > > > > > *If the goal is to do the partition assignments at a topic level, > > > wouldn't > > > > having single-partition topics solve this problem?* > > > > > > > > > > > > > > > > We are in a microservices environment; therefore, we can have > multiple > > > > containers running the same application. > > > > > > > > Using the TopicRoundRobinAssignor, the partitions are uniformly > > balanced > > > > to each container, and we can get better performances. > > > > > > > > Let suppose there are 2 instances of the same application A0 and A1, > 2 > > > > consumers C0 and C1, and two topics t0 and t1. t0 has 3 partitions > and > > t1 > > > > has two partitions resulting in partitions : t0p0, t0p1, t0p2, t1p0, > > > t1p1. > > > > > > > > If we use the TopicRoundRobinAssignor, the assignment will be : > > > > > > > > A0 : [ C0: [t0p0, t0p2], C1: [t1p0] ] > > > > > > > > A1 : [ C0: [t0p1], C1: [t1p1] ] > > > > > > > > > > > > > > > > *How will the group leader know that T2 should not be re-assigned on > > the > > > > next rebalance? Can you elaborate a bit more on the mechanisms used > to > > > > communicate this state to the other group members?* > > > > > > > > > > > > > > > > Currently, the group leader will not know that T2 should not be > > > > re-assigned on the next balance. > > > > > > > > For this first iteration, we simply keep in memory that T2 has a > poison > > > > pill and therefore we ignore all incoming messages from T2. We > > basically > > > > consume them without acknowledging them. > > > > > > > > As you can imagine, in the case of having multiple instances of the > > same > > > > application, in case of error, the partition will be rebalanced to > > > another > > > > instance. > > > > > > > > Nevertheless, this is not really a problem (at least for our use > > cases), > > > > as soon as the poison pill is consumed, the consumer of this other > > > instance > > > > will be stopped, and so on, and so on. It will take a few tens of > > seconds > > > > before the last consumer of the poison pill will be stopped and so > the > > > > consumption of the entire topic. > > > > > > > > For a second iteration, we have planned to find a solution to avoid > > this > > > > time lapse between the consumer of the first instance being stopped > and > > > the > > > > last one. Currently, I do not have a solution, but we are thinking > > about > > > > different options, avoiding rebalancing partitions containing a > poison > > > pill > > > > is one of them. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Mathieu > > > > > > > > Le ven. 7 oct. 2022 à 16:54, Hector Geraldino (BLOOMBERG/ 919 3RD A) > < > > > > hgerald...@bloomberg.net> a écrit : > > > > > > > >> Hi Mathieu. I took a look at your KIP and have a couple questions. > > > >> > > > >> If the goal is to do the partition assignments at a topic level, > > > wouldn't > > > >> having single-partition topics solve this problem? > > > >> > > > >> You also mentioned that your goal is to minimize the potential of a > > > >> poison pill message breaking all members of a group (by keeping > track > > of > > > >> which topics have 'failed'), but it is not clear how this can be > > > achieved > > > >> with this assignor. If we imagine an scenario where: > > > >> > > > >> * A group has 3 members (A, B, C) > > > >> * Members are subscribed to 3 topics (T1, T2, T3) > > > >> * Each member is assigned one topic (A[T1], B[T2], C[T3]) > > > >> * One member fails to consume from a topic/partition (B[T2]), and > goes > > > >> into failed state > > > >> > > > >> How will the group leader know that T2 should not be re-assigned on > > the > > > >> next rebalance? Can you elaborate a bit more on the mechanisms used > to > > > >> communicate this state to the other group members? > > > >> > > > >> Thanks > > > >> > > > >> From: dev@kafka.apache.org At: 10/05/22 03:47:33 UTC-4:00To: > > > >> dev@kafka.apache.org > > > >> Subject: [DISCUSS] KIP-874: TopicRoundRobinAssignor > > > >> > > > >> Hi Kafka Developers, > > > >> > > > >> My proposal is to add a new partition assignment strategy at the > topic > > > >> level to : > > > >> - have a better data consistency by consumed topic in case of > > exception > > > >> - have a solution much thread safe for the consumer > > > >> In case there are multiple consumers and multiple topics. > > > >> > > > >> Here is the link to the KIP with all the explanations : > > > >> https://cwiki.apache.org/confluence/x/XozGDQ > > > >> > > > >> Thank you in advance for your feedbacks, > > > >> Mathieu > > > >> > > > >> > > > >> > > > > > >
