Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

Hi TengYao, Thanks for the KIP! I have a couple of comments. 1. In the motivation section, I would really start from the fundamental issue which is that the initial heartbeat is not idempotent. Then, we could describe the undesired side effects (e.g. ghost members, cannot leave without receiving

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

> I don't want to be a hater of idempotent HB, but having a "RPC" used to generate UUID is unnecessary to me. I actually agree with you. I am just trying to argue for it. > I'm not sure whether it is worth requiring the UUID format for member id. In the protocol, we declare the field "memberId" a

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

Hi Andrew, Personally, I don't like the lobby approach. It makes things more complicated and it would require changing the records on the server too. This is why I initially suggested the rejected alternative #2 which is pretty close but also not perfect. I'd like to clarify one thing. The Consum

Re: [DISCUSS] GitHub CI

Hi David, Thanks for working on this. Overall, I am supportive. I have two questions/comments. 1. I wonder if we should discuss with the infra team in order to ensure that they have enough capacity for us to use the action runners. Our CI is pretty greedy in general. We could also discuss with th

Re: Kafka trunk test & build stability

The slowness of the CI is definitely causing us a lot of pain. I wonder if we should move to a dedicated CI infrastructure for Kafka. Our integration tests are quite heavy and ASF's CI is not really tuned for them. We could tune it for our needs and this would also allow external companies to spons

Re: [DISCUSS] Road to Kafka 4.0

Hi Divij, > Release 4.0 as an "experimental" release I don't think that this is something that we should do. If we need more time, we should just do a 3.8 release and then release 4.0 when we are ready. An experimental major release will be more confusing than anything else. We should also keep i

Re: [DISCUSS] Road to Kafka 4.0

Thanks, Ismael. The proposal makes sense. +1 David On Thu, Dec 21, 2023 at 5:59 PM Ismael Juma wrote: > Hi all, > > After understanding the use case Josep and Anton described in more detail, > I think it's fair to say that quorum reconfiguration is necessary for > migration of Apache Kafka user

Re: Kafka trunk test & build stability

ava:36) > > > > > at > > > > > > > > > > > > > > > org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:103) > > > > > at > > > > > >

Re: Kafka trunk test & build stability

I just merged both PRs. Cheers, David Le ven. 22 déc. 2023 à 14:38, David Jacot a écrit : > Hey folks, > > I believe that my two PRs will fix most of the issues. I have also tweaked > the configuration of Jenkins to fix the issues relating to cloning the > repo. There may be o

Re: [ANNOUNCE] New Kafka PMC Member: Divij Vaidya

Congrats! Le jeu. 28 déc. 2023 à 05:13, Ismael Juma a écrit : > Congratulations Divij! > > Ismael > > On Wed, Dec 27, 2023 at 3:46 AM Luke Chen wrote: > > > Hi, Everyone, > > > > Divij has been a Kafka committer since June, 2023. He has remained very > > active and instructive in the community

Re: Apache Kafka 3.7.0 Release

Hi all, Are you talking about publishing the artefacts to maven central? Looking at the history [1], it seems that the metadata module has been published since we have it. I also see other internal modules there too. [1] https://central.sonatype.com/artifact/org.apache.kafka/kafka-metadata/versio

Re: [VOTE] KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

Hi Ziming, Thanks for driving this. I wanted to bring KAFKA-10140 to your attention. It looks like the incremental API does not work for configuring plugins. I think that we need to cover this in the KIP. Best, David On Mon, Jan 22, 2024 at 10:

Re: [VOTE] KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

little overkill to support append/subtract > configs or solve KAFKA-10140 here, so let’s leave it right now, I'm happy > to pick it after finishing this KIP. > > --, > Ziming > > > On Jan 22, 2024, at 18:23, David Jacot > wrote: > > > > Hi Ziming, > > &

Re: [VOTE] KIP-951: Leader discovery optimisations for the client

> > mayanks.nar...@gmail.com> wrote: > > > >> Thank you all for your votes, Jun, David, and Jason! > >> > >> On Tue, Oct 3, 2023 at 11:44 PM Jason Gustafson > >> wrote: > >> > >>> +1 Thanks for the KIP > >>&

Improve flaky test reporting (KAFKA-12216)

Hi folks, I have been playing with `reports.junitXml.mergeReruns` setting in gradle [1]. From the gradle doc: > When mergeReruns is enabled, if a test fails but is then retried and succeeds, its failures will be recorded as instead of , within one . This is effectively the reporting produced by

Re: Improve flaky test reporting (KAFKA-12216)

lure element and hence this information will be > completely missing from the Jenkins report. Have we considered including > the flakyFailure information ourselves? I have seen that being done and it > seems strictly better than totally ignoring it. > > Ismael > > On Mon, Feb 12,

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

Hi, Jun, Justine, Regarding `group.coordinator.version`, the idea is to use it to gate records and APIs of the group coordinator. The first use case will be KIP-848. We will use version 2 of the flag to gate all the new records and the new ConsumerGroupHeartbeat/Describe APIs present in AK 3.8. So

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

gt;> that right? It would be useful to document that. > > > >> > > > >> Also, we should align on the version numbering. "kafka-feature > > disable" > > > >> says "Disable one or more feature flags. This is the same as > > downgrading

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

group coordinator as we may use it for queues too. It would also be aligned with `metadata.version`. I apologize if this was already discussed. Best, David On Tue, Apr 2, 2024 at 11:18 AM David Jacot wrote: > Hi Jun, > > > Historically, the format of all records were controlle

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

gt; version 0 records. > >> > > > > > > KIP-915 introduced flexible versions, but it was never put > to > >> > use. > >> > > MV > >> > > > > has > >> > > > > > > never gated these. This KIP will do that. I can i

Re: Gentle bump on KAFKA-16371 (Unstable committed offsets after triggering commits where metadata for some partitions are over the limit)

Thanks, Michal. Let me add it to my review queue. BR, David On Fri, Apr 5, 2024 at 3:29 PM Michał Łowicki wrote: > Hi there! > > Created https://issues.apache.org/jira/browse/KAFKA-16371 few weeks back > but there wasn't any attention. Any chance someone knowing that code could > take a look at

Re: [VOTE] KIP-1022 Formatting and Updating Features

+1 (binding). Thanks for the KIP! On Mon, Apr 8, 2024 at 7:23 PM Andrew Schofield < andrew_schofield_j...@outlook.com> wrote: > Hi Justine, > Thanks for the KIP. > > +1 (non-binding) > > Thanks, > Andrew > > > On 8 Apr 2024, at 18:07, Justine Olshan > wrote: > > > > Hello all, > > I would like t

Re: the migration of command tools

Hey, I think that we discussed this in this KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines. I don't remember all the details though. Best, David On Wed, Apr 10, 2024 at 2:54 PM Chia-Ping Tsai wrote: > Dear Kafka, > > Migrating command tools from co

Re: [DISCUSS] KIP-932: Queues for Kafka

Hi Andrew, Thanks for the KIP. This work is really exciting. I finally had a bit of time to go through the KIP. I need to read it a second time in order to get into the details. I have noted a few points/questions: 001: The dynamic config to force the group type is really weird. As you said, gro

Re: [DISCUSS] KIP-932: Queues for Kafka

I’ve tweaked the description. > >>>> > >>>> Thanks, > >>>> Andrew > >>>> > >>>>> On 22 Apr 2024, at 20:04, Jun Rao wrote: > >>>>> > >>>>> Hi, Andrew, > >>>>> > >

New Group Coordinator (KIP-848)

Hi folks, I wanted to let you know that the new group coordinator that we developed as part of KIP-848 is now the default group coordinator in trunk (kraft only). Hence all the integration tests, all the system tests and all the kraft clusters created based on trunk use it by default unless specif

[ANNOUNCE] New committer: Lianet Magrans

Hi all, The PMC of Apache Kafka is pleased to announce a new Kafka committer, Lianet Magrans. Lianet has been a Kafka contributor since June 2023. In addition to being a regular contributor and reviewer, she has made significant contributions to the next generation of the consumer rebalance proto

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

; > > >> >>> > > > >> >>> This is definitely a good question. the short answer: no > guarantee > > > but > > > >> >>> best > > > >> >>> efforts > > > >>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

ast the lifetime of a client instance (such as the transaction ID). > > > > Thanks, > > Kirk > > > > > On Aug 29, 2024, at 1:28 AM, TengYao Chi wrote: > > > > > > Hi David, > > > > > > Thank you for pointing that out. > > &g

Re: New Group Coordinator (KIP-848)

imeMin=172473120&search.tags=trunk&search.timeZoneId=America%2FNew_York&tests.container=org.apache.kafka.connect.integration.*&tests.sortField=FLAKY > [3] - https://issues.apache.org/jira/browse/KAFKA-17493 > > Cheers, > > Chris > > On Mon, Aug 26, 2024 at 4:21 AM David

Re: [ANNOUNCE] New Kafka PMC Member: Josep Prat

Congrats! Le sam. 7 sept. 2024 à 05:27, Yash Mayya a écrit : > Congratulations Josep! > > On Fri, 6 Sept, 2024, 21:55 Chris Egerton, wrote: > > > Hi all, > > > > Josep has been a Kafka committer since December 2022. He has remained > very > > active and instructive in the community since then,

[ANNOUNCE] New committer: Jeff Kim

Hi all, The PMC of Apache Kafka is pleased to announce a new Kafka committer, Jeff Kim. Jeff has been a Kafka contributor since May 2020. In addition to being a regular contributor and reviewer, he has made significant contributions to the next generation of the consumer rebalance protocol (KIP-8

Re: [VOTE] KIP-580: Exponential Backoff for Kafka Clients

+1 (non-binding) Thanks for the KIP, great improvement! Le mer. 25 mars 2020 à 04:44, Gwen Shapira a écrit : > +1 (binding) - thank you > > On Mon, Mar 23, 2020, 10:50 AM Sanjana Kaundinya > wrote: > > > Hi Everyone, > > > > I’d like to start a vote for KIP-580: Exponential Backoff for Kafka >

Re: [VOTE] KIP-570: Add leader epoch in StopReplicaRequest

ted the schema in the KIP if you want to see it. I will update the KIP itself if you agree with the amendment. What do you think? Does it sound reasonable? Best, David On Fri, Mar 6, 2020 at 3:37 PM David Jacot wrote: > Hi all, > > The vote has passed with +3 binding votes (Jason

Re: [VOTE] KIP-570: Add leader epoch in StopReplicaRequest

> Is it really true that the controller always sends two requests? Aren't the > operations different (stop replica with delete versus stop replica > without)? > > On Wed, Mar 25, 2020, 9:59 AM David Jacot wrote: > > > Hi all, > > > > I'd like to inform

Re: [DISCUSS] KIP-574: CLI Dynamic Configuration with file input

g / > producer.config / > > command-config parameter. > > > > Shouldn't we have to maintain the uniformity across scripts? > > > > On Mon, Mar 16, 2020 at 4:13 PM David Jacot wrote: > > > > > Hi Aneel, > > > > > > Thanks for the upd

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

Hi Boyang, Thanks for the KIP. Overall, it looks good to me. I really like the envelope RPC! One minor comment regarding the `old-client-connections-count` metric. Is it really necessary? The number of connected clients whose version is not known (prior to KIP-511) is already reported but with an

Re: [VOTE] KIP-584: Versioning scheme for features

Great KIP, thanks! +1 (non-binding) On Fri, Apr 17, 2020 at 8:56 PM Guozhang Wang wrote: > Thanks for the great KIP Kowshik, +1 (binding). > > On Fri, Apr 17, 2020 at 11:22 AM Jun Rao wrote: > > > Hi, Kowshik, > > > > Thanks for the KIP. +1 > > > > Jun > > > > On Thu, Apr 16, 2020 at 11:14 AM K

KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

Hi folks, I'd like to start the discussion for KIP-599: https://cwiki.apache.org/confluence/display/KAFKA/KIP-599%3A+Throttle+Create+Topic%2C+Create+Partition+and+Delete+Topic+Operations It proposes to introduce quotas for the create topics, create partitions and delete topics operations. Let me

Re: KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

the new quota fit into the existing hierarchy? > > 3. It seems that you are proposing a new quota mechanism based on Token > Bucket algorithm. Could you describe its tradeoff with the existing quota > mechanism? Ideally, it would be better if we have a single quota mechanism > within

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

Hi Boyang, While re-reading the KIP, I've got few small questions/comments: 1. When auto topic creation is enabled, brokers will send a CreateTopicRequest to the controller instead of writing to ZK directly. It means that creation of these topics are subject to be rejected with an error if a Crea

Re: [DISCUSS] Kafka 3.0

Hi all, First, I agree with what has been discussed. Having 3.x as the bridge releases and entirely removing ZK in 4.0 makes total sense. Second, what would you think about removing the auto topics creation in 3.0? It is not recommended to use it anymore and that could simplify a bit our path tow

Re: KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

d the difference in burst balance. > The > > > current throttling logic works as follows. Each quota is measured over > a > > > number of time windows. Suppose the Quota is to X/sec. If time passes > and > > > the quota is not being used, we are accumulating credit at the rat

Re: KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

ler. Imagine a low quota, that would result in a huge backlog of pending operations. Best, David On Mon, May 11, 2020 at 4:57 PM David Jacot wrote: > Hi Anna and Jun, > > Anna, thanks for your thoughtful feedback. Overall, I agree with what you > said. If I summarize, you said that u

Re: [DISCUSS] KIP-608: Add a new method to AuthorizerServerInfo Interface

Hey, Thanks for the KIP. I think that having the possibilities to expose metrics in plugins such as the authorizer in a nice improvement. I do wonder if we could come up with a more generic way to do this that would apply to all plugins instead of having something specific for the authorized. Fo

Re: [DISCUSS] KIP-612: Ability to Limit Connection Creation Rate on Brokers

Hi Anna, Thanks for the KIP! I have few questions: 1. You mention that some clients may create a new connections for each requests: "Another example is clients that create a new connection for each produce/consume request". I am curious here but do we know any clients behaving like this? 2. I am

Re: KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

Hi Jun, Coming back to your question regarding the differences between the token bucket algorithm and our current quota mechanism. I did some tests and they confirmed my first intuition that our current mechanism does not work well with a bursty workload. Let me try to illustrate the difference wi

Re: KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

e harder to do this in the controller as the controller is completely agnostic from the principals and the client ids. These reasons made me lean towards the current proposal. Does that make sense? Best, David On Wed, May 13, 2020 at 10:05 AM David Jacot wrote: > Hi Jun, > > Com

Re: [DISCUSS] KIP-612: Ability to Limit Connection Creation Rate on Brokers

point 3 - conceptually one difference between capping > >> the queue size or throttling as presented in the KIP would come from > >> the time it takes to accept a connection and how that time evolves > >> with the connection rate. > >> Assuming that that time

Re: [DISCUSS] KIP-608: Add a new method to AuthorizerServerInfo Interface

, > I accepted your suggestion and add Monitorable interface. Thanks for great > suggestion. Please review KIP-608 again and see any suggestion on name of > function or other stuff. > > jeff. > > On 2020/05/11 15:16:30, David Jacot wrote: > > Hey, > > > >

Re: [VOTE] KIP-612: Ability to Limit Connection Creation Rate on Brokers

+1 (non-binding) Thanks for the KIP, Anna! On Tue, May 19, 2020 at 7:12 AM Satish Duggana wrote: > +1 (non-binding) > Thanks Anna for the nice feature to control the connection creation rate > from the clients. > > On Tue, May 19, 2020 at 8:16 AM Gwen Shapira wrote: > > > +1 (binding) > > > >

Re: [VOTE]: KIP-604: Remove ZooKeeper Flags from the Administrative Tools

+1 (non-binding). Thanks for the KIP. On Fri, May 15, 2020 at 12:41 AM Guozhang Wang wrote: > +1. > > Thanks Colin! > > Guozhang > > On Tue, May 12, 2020 at 3:45 PM Colin McCabe wrote: > > > Hi all, > > > > I'd like to start a vote on KIP-604: Remove ZooKeeper Flags from the > > Administrative

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

t; likely going to be deprecated post KIP-500. > 3. Txn Coordinator -> Broker: TxnMarker > > > Guozhang > > On Wed, May 6, 2020 at 8:58 PM Boyang Chen > wrote: > > > Hey David, > > > > thanks for the feedbacks! > > > > On Wed, May 6, 2020 at

Re: KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

Hi folks, I have updated the KIP. As mentioned by Jun, I have made the quota per principal/clientid similarly to the other quotas. I have also explained how this will work in conjunction with the auto topics creation. Please, take a look at it. I plan to call a vote for it in the next few days if

[VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

Hi folks, I'd like to start the vote for KIP-599 which proposes a new quota to throttle create topic, create partition, and delete topics operations to protect the Kafka controller: https://cwiki.apache.org/confluence/display/KAFKA/KIP-599%3A+Throttle+Create+Topic%2C+Create+Partition+and+Delete+To

Re: KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

n bucket? The reason that I keep asking this is that from > an operational perspective, it's simpler if all types of quotas work in the > same way. > > Jun > > On Tue, May 26, 2020 at 9:52 AM David Jacot wrote: > > > Hi folks, > > > > I have updated the

Re: [VOTE] KIP-518: Allow listing consumer groups per state

> > > > > > > > So far we have 1 binding and 5 non binding votes. > > > > > > > > > > On Mon, Mar 2, 2020 at 4:56 PM Gwen Shapira > wrote: > > > > > > > > > > > > +1 (binding) > >

Re: KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

set only has a > cluster wide default for two reasons.* > > Regards, > > Rajini > > > On Tue, Jun 2, 2020 at 2:55 PM David Jacot wrote: > > > Hi Jun, > > > > Thanks for your reply. > > > > 10. I think that both options are likely equivale

Re: KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

lp to alleviate the tokens bucket vs rate discussions. Please, have a look and let me know your thoughts. Bests, David On Wed, Jun 3, 2020 at 10:17 AM David Jacot wrote: > Hi Rajini, > > Thanks for your feedback. Please find my answers below: > > 1) Our main goal is to protect t

Re: KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

the > KIP. > > Regards, > > Rajini > > On Wed, Jun 3, 2020 at 1:02 PM David Jacot wrote: > > > Hi all, > > > > I have updated the KIP based on our recent discussions. I have mainly > > changed the > > following points: > > * I have renamed

Re: KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

vers topic creation/deletion. It would not be ideal > if > > > in > > > > the future, we have to add yet another type of quota for some other > > > > controller related operations. The quota name in the KIP has > partition > > > > mutat

Re: KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

Hi Tom, That's a good question. As the validation does not create any load on the controller, I was planning to do it without checking the quota at all. Does that sound reasonable? Best, David On Thu, Jun 4, 2020 at 4:23 PM David Jacot wrote: > Hi Jun and Anna, > > Thank you

Re: KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

e Javadoc that the quota is not included in the > validation. > > On balance, I agree with what you're proposing, since the extra utility of > including the quota in the validation seems to be not worth the extra > complication for the retry. > > Thanks, > > Tom > > >

Re: KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

s have to come before the balance becomes 0 again. > Intuitively, > > we are accumulating credits in each sample. If a usage comes in, we first > > use all existing credits to offset that. If we can't, the remaining usage > > will be recorded in the last sample, which will be offse

Re: KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

at the topic or partition level? It would be useful to make > it > > clear in the wiki. > > > > Jun > > > > On Fri, Jun 5, 2020 at 7:23 AM David Jacot wrote: > > > > > Hi Anna and Jun, > > > > > > You are right. We should allocate u

Re: KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

> Thanks for the updated KIP. Another minor comment below. > > 40. For the new `QUOTA_VIOLATED` error in the response to > CreateTopics/CreatePartitions/DeleteTopics, could you clarify > whether ThrottleTimeMs is set when the error code is set to QUOTA_VIOLATED? > > Jun > >

Re: [VOTE] KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

t; Basically, the way we handle RPC timeouts in the controller right now > is > > > not very good. As you know, we time out the RPC, so the client gets > > > TimeoutException, but then keep the event on the queue, so that it > > > eventually gets executed! There'

Re: [DISCISS] KIP-860: Add client-provided option to guard against unintentional replication factor change during partition reassignments

week. > > Best, > Stanislav > > On Tue, Aug 9, 2022 at 5:06 AM David Jacot > wrote: > > > Throwing an UnsupportedVersionException with an appropriate message > > seems to be the best option when the new API is not supported and > > AllowReplicationFactorCh

Re: [DISCUSS] KIP-854 Separate configuration for producer ID expiry

Given that we already have `transaction.abort.timed.out.transaction.cleanup.interval.ms` and `transaction.remove.expired.transaction.cleanup.interval.ms`, it seems OK to add another one for our case here. Regarding the name, I would follow the pattern that we use for those two existing configs. We

Re: [DISCUSS] KIP-854 Separate configuration for producer ID expiry

Sounds good. Thanks, Justine. Le ven. 19 août 2022 à 19:38, Justine Olshan a écrit : > Hi all, > > Followed up with David and Ismael offline. > Ismael explained that we probably don't want to increase complexity and > didn't think the value needed to be modified beyond tests. I agree with > this

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

, WDYT? > > > > [1] > > https://cwiki.apache.org/confluence/display/KAFKA/%5BDRAFT%5DIntegrating+Kafka+Connect+With+New+Consumer+Rebalance+Protocol > > > > Thank you. > > Luke > > > > > > > > On Fri, Aug 12, 2022 at 9:31 PM Sagar wrote: > &

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

implementation detail after all so it does not have to be decided at this stage. We will likely start by trying to refactor the current implementation as a first step. Cheers, David On Mon, Aug 29, 2022 at 3:52 PM David Jacot wrote: > > Hi Luke, > > > 1.1. I think the state machine are: &q

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

ems to have > multiple typos. > 38.1 When the group transitions to epoch 2, B immediately gets into > "epoch=1, partitions=[foo-2]", which seems incorrect. > 38.2 When the group transitions to epoch 3, C seems to get into epoch=3, > partitions=[foo-1] too early. > 38.3 After A

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

Hi Jun, I have updated the KIP to include your feedback. I have also tried to clarify the parts which were not cleared. Best, David On Fri, Sep 2, 2022 at 4:18 PM David Jacot wrote: > > Hi Jun, > > Thanks for your feedback. Let me start by answering your questions > inline an

Re: [VOTE] KIP-860: Add client-provided option to guard against unintentional replication factor change during partition reassignments

+1 from me. Thanks, Stan! On Tue, Aug 23, 2022 at 12:10 PM Luke Chen wrote: > > Hi Stanislav, > > Thanks for the KIP. > The solution looks reasonable to me. > +1 from me. > > Thank you. > Luke > > On Tue, Aug 23, 2022 at 6:07 AM Stanislav Kozlovski > wrote: > > > Hello, > > > > I'd like to start

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

the PartitionAssignor to indicate > the next HB telling broker about so. WDYT about adding such an API on the > PartitionAssignor? > > > Guozhang > > > On Tue, Sep 6, 2022 at 6:09 AM David Jacot > wrote: > > > Hi Jun, > > > > I have updated the K

[VOTE] KIP-848: The Next Generation of the Consumer Rebalance Protocol

Hi all, Thank you all for the very positive discussion about KIP-848. It looks like folks are very positive about it overall. I would like to start a vote on KIP-848, which introduces a brand new consumer rebalance protocol. The KIP is here: https://cwiki.apache.org/confluence/x/HhD1D. Best, Da

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

there's agreement on > using separate APIs for Connect. I would revisit the doc and see what > changes are to be made. > > Thanks! > Sagar. > > On Tue, Aug 9, 2022 at 7:11 PM David Jacot > wrote: > > > Hi Sagar, > > > > Thanks for the feedback and

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

artbeat request handling section, stating when coordinator will > trigger rebalance based on the HB's member metadata / reason? > 2) the "Rebalance Triggers" section to include what we described in "Group > Epoch - Trigger a rebalance" section as well? > > > Guozh

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

e resources handled by each type > > (Topic/Partitions in the case of consumer, Connector/Task in the case of > > Connect, Leadership in the case of Schema Registry, and so on). > > > > > > > > > From: dev@kafka.apache.org At: 08/12/22 09:31:36 UTC-4:00To: > > dev

Re: [VOTE] KIP-792: Add "generation" field into consumer protocol

Hi Luke, Are you still interested in implementing this KIP? We need it for KIP-848. If you are not, we could find someone to take it over. Thanks, David On Thu, Mar 3, 2022 at 10:04 AM Luke Chen wrote: > > Thanks, Ziming! > > So, now, this KIP vote passed with 3 binding +1 votes (David, Tom, >

Re: [VOTE] KIP-792: Add "generation" field into consumer protocol

Thanks, Luke. Feel free to ping me for reviews. I am happy to help on this one. Cheers, David On Thu, Sep 22, 2022 at 11:00 AM Luke Chen wrote: > > Hi David, > > Sorry for the delay. > I'll complete it in v3.4.0. > > Thank you. > Luke > > On Thu, Sep 22, 2

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

e much, and the time > `onPartitionsAssigned` indicate the time when the partitions are actually > assigned to it; while for assignors, the `onAssignment` is used to indicate > what decision is made regarding for this member, i.e. when the partitions > are decided to be given to it, bu

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

do you think that we should not provide the partitions but only the metadata? Best, David On Fri, Sep 23, 2022 at 9:40 PM Guozhang Wang wrote: > > Hello David, > > On Fri, Sep 23, 2022 at 2:00 AM David Jacot > wrote: > > > Hey, > > > > > Just to clarify I

Re: [VOTE] 3.3.0 RC2

Thanks for running the release, José and David. I performed the following validations: * Verified all checksums and signatures. * Built from source and ran unit tests. * Ran the first quickstart steps for both ZK and KRaft. * Spotchecked the Javadocs. I have also noticed that the doc is malformed

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

feel that not providing the > partitions but only the metadata for `onAssignment` may be less confusing > and push users to separate the usage of these two more clearly, but since > we already introduced partitions in `onAssignment` for compatibility I'm > less keen on removing them.

Re: [VOTE] KIP-868 Metadata Transactions

+1 (binding). Thanks for the KIP. I really like the approach! Best, David On Wed, Sep 28, 2022 at 4:20 AM Luke Chen wrote: > > Hi David, > > +1 (binding) from me. > > Thank you > Luke > > On Tue, Sep 27, 2022 at 11:02 PM David Arthur > wrote: > > > Hey folks, I'd like to start a vote on KIP-868

Re: [VOTE] 3.3.1 RC0

Hey, I performed the following validations: * Verified all checksums and signatures. * Built from source and ran unit tests. * Ran the first quickstart steps for both ZK and KRaft. * Spotchecked the Javadocs. I am +1 (binding), assuming that the system tests look good. Thanks for running the rel

Re: [DISCUSS] Apache Kafka 3.4.0 release

+1. Thanks, Sophie! Le mer. 5 oct. 2022 à 19:57, Luke Chen a écrit : > Hi Sophie, > > Thanks for volunteering! > > Luke > > On Thu, Oct 6, 2022 at 6:17 AM José Armando García Sancio > wrote: > > > Thanks for volunteering Sophie. > > > > On Wed, Oct 5, 2022 at 3:01 PM Sophie Blee-Goldman > > wr

Re: [ANNOUNCE] New committer: Deng Ziming

Congrats! Well deserved. Best, David Le lun. 10 oct. 2022 à 20:40, Satish Duggana a écrit : > Congratulations Ziming!! > > On Mon, 10 Oct 2022 at 11:12, Chris Egerton > wrote: > > > Congrats, Ziming! > > > > On Mon, Oct 10, 2022, 13:29 Tom Bentley wrote: > > > > > Congratulations! > > > > > >

Re: [DISCUSS] KIP-876: Time based cluster metadata snapshots

Hi José, Thanks for the KIP. That makes total sense. On nit, I would name the new property `metadata.log.snapshot.interval.ms` as `between` is implied by the `interval`. Best, David On Tue, Oct 11, 2022 at 9:16 PM José Armando García Sancio wrote: > > Hey all, > > I am interested in allowing br

Re: [VOTE] KIP-876: Time based cluster metadata snapshots

+1 (binding) Thanks for the KIP! Le ven. 14 oct. 2022 à 05:47, deng ziming a écrit : > Thanks for this KIP, > > +1 for this(binding). > > -- > Best, > Ziming > > > On Oct 14, 2022, at 8:11 AM, José Armando García Sancio > wrote: > > > > Hello all, > > > > I would like to start voting for "KIP-

Re: [DISCUSS] (continued) KIP-848: The Next Generation of the Consumer Rebalance Protocol

Hi Jun, Thanks for your comments. Please find my answers below. 60. Yes, we need both. PartitionAssignor.onAssignment is here to inform the customer assignor about the assignment decision taken with the full set of assigned partitions regardless of whether they are already revoked or not and the

Re: [DISCUSS] (continued) KIP-848: The Next Generation of the Consumer Rebalance Protocol

Hi Luke, Thanks for your questions. > 1. We will store the "targetAssignment" into log now. But as we know, there's max batch size limit (default 1MB), which means, we cannot support 1M partitions in one group (actually, it should be less than 60k partitions since we'll store {topicID+partition i

Re: [DISCUSS] (continued) KIP-848: The Next Generation of the Consumer Rebalance Protocol

e just because a topic is recreated. For the assigned partitions, > perhaps we could include both topicId and name just like FetchOffsetRequest. > > Thanks, > > Jun > > On Fri, Oct 14, 2022 at 2:49 AM Luke Chen wrote: > > > Thanks for the update. > > Yes, I think using simila

Re: [DISCUSS] (continued) KIP-848: The Next Generation of the Consumer Rebalance Protocol

etry the commit once it has received an > updated member epoch? If so, isn't there a case where another member might > be assigned the partition-to-be-committed for some time before the > partition is assigned back to this consumer, which would cause the > old-but-retried (with a newly

Re: [DISCUSS] (continued) KIP-848: The Next Generation of the Consumer Rebalance Protocol

> Thanks, > > Jun > > > On Mon, Oct 17, 2022 at 2:35 AM David Jacot > wrote: > > > Hi Jun, > > > > Thanks for your comments. Please find my answers below. > > > > 60. Sure. Let me use a concrete example to illustrate it. Let's assume >

Re: [DISCUSS] (continued) KIP-848: The Next Generation of the Consumer Rebalance Protocol

ator should be responsible > for mapping the topic names to topic ids. > > 81. group.consumer.assignors: Should we change the default values to > include the full class name? > > Thanks, > > Jun > > On Tue, Oct 18, 2022 at 2:36 AM David Jacot > wrote: > > &g

Re: [DISCUSS] (continued) KIP-848: The Next Generation of the Consumer Rebalance Protocol

Hi Jun, 81. I forgot to say that I put UniformAssignor as the first one in the list. I think that it should be the default one. Best, David On Tue, Oct 18, 2022 at 8:33 PM David Jacot wrote: > > Hi Jun, > > 80. Hmm. It seems prefer

Re: [DISCUSS] (continued) KIP-848: The Next Generation of the Consumer Rebalance Protocol

, using topicId in > ConsumerGroupInstallAssignmentRequest makes sense. > > 81. Sounds good. > > Jun > > On Tue, Oct 18, 2022 at 11:46 AM David Jacot > wrote: > > > Hi Jun, > > > > 81. I forgot to say that I put UniformAssignor as the first one in the &g

<    1   2   3   4   5   6   7   8   9   10   >