Re: [DISCUSS] KIP-1101: Trigger rebalance on rack topology changes

etadata by a group hash, so the > coordinator calculates group > hash where we calculated subscription metadata. The trigger points are > like a group member is updated, > , new group member, new subscribed topic names, new regular expressions, > and new metadata (topic change). >

Re: [DISCUSS] KIP-1101: Trigger rebalance on rack topology changes

Hi PoAn, I was late to the discussion but I had some questions. JK01. I foresee scenarios where the cache leaves stale topic hashes around when a group leaves. Should we maintain a counter for the number of groups subscribed to the topic (hash) to only keep active topics? JK02. Can you help me u

Re: [ANNOUNCE] Apache Kafka 4.0.0

au, GangHuo, > Gantigmaa Selenge, Gaurav Narula, Greg Harris, Guozhang Wang, Hailey > Ni, Hongten, Hyunsang Han, Ian McDonald, Igor Soarez, Ismael Juma, > Ivan Yurchenko, Jakub Scholz, Jason Gustafson, Jason Taylor, Jeff Kim, > Jhen-Yung Hsu, Jim Galasyn, João Pedro Fonseca Dantas, Jo

[jira] [Resolved] (KAFKA-18655) Implement the consumer group size counter with scheduled task

[ https://issues.apache.org/jira/browse/KAFKA-18655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Kim resolved KAFKA-18655. -- Resolution: Fixed > Implement the consumer group size counter with scheduled t

[jira] [Created] (KAFKA-18610) Add flush metrics to runtime

Jeff Kim created KAFKA-18610: Summary: Add flush metrics to runtime Key: KAFKA-18610 URL: https://issues.apache.org/jira/browse/KAFKA-18610 Project: Kafka Issue Type: Sub-task

[jira] [Resolved] (KAFKA-18174) Handle hdr histogram exceptions gracefully

[ https://issues.apache.org/jira/browse/KAFKA-18174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Kim resolved KAFKA-18174. -- Fix Version/s: 4.0.0 Resolution: Fixed > Handle hdr histogram exceptions gracefu

[jira] [Created] (KAFKA-18174) Handle hdr histogram exceptions gracefully

Jeff Kim created KAFKA-18174: Summary: Handle hdr histogram exceptions gracefully Key: KAFKA-18174 URL: https://issues.apache.org/jira/browse/KAFKA-18174 Project: Kafka Issue Type: Sub-task

Re: [VOTE] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group

Thanks! +1 (binding) Jeff On 2024/11/25 15:46:04 "Lianet M." wrote: > Thanks for the KIP and updates PoAn! > > +1 (binding) > > Lianet > > On Mon, Nov 25, 2024, 10:43 a.m. Andrew Schofield < > andrew_schofield_j...@outlook.com> wrote: > > > +1 (non-binding) > > > > Thanks, > > Andrew > > > >

Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group

ribe > > —offsets —verbose`, > > because `—describe` also shows output as same as `—describe —offsets`. > > > > JK2: Yes, I only use consumer group in the sample output, but the classic > > group will use the same format as well. > > > > Thanks, > > PoAn > &g

[jira] [Resolved] (KAFKA-18019) Convert INVALID_PRODUCER_ID_MAPPING from abortable error to fatal error

[ https://issues.apache.org/jira/browse/KAFKA-18019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Kim resolved KAFKA-18019. -- Fix Version/s: 4.0.0 Resolution: Fixed > Convert INVALID_PRODUCER_ID_MAPPING from aborta

Re: [DISCUSS] KIP-1099: Extend kafka-consumer-groups command line tool to support new consumer group

Hi PoAn, Thanks for the KIP. I have some questions: JK1: (Realized this is similar to LM5) On `--describe --verbose` proposed changes - doesn't `--describe $group` default to printing the offsets? Perhaps you're referring to the `--state` argument? Also, would that mean the default `--describe

[jira] [Created] (KAFKA-17578) Remove partitionRacks from TopicMetadata

Jeff Kim created KAFKA-17578: Summary: Remove partitionRacks from TopicMetadata Key: KAFKA-17578 URL: https://issues.apache.org/jira/browse/KAFKA-17578 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-16831) CoordinatorRuntime should initialize MemoryRecordsBuilder with max batch size write limit

Jeff Kim created KAFKA-16831: Summary: CoordinatorRuntime should initialize MemoryRecordsBuilder with max batch size write limit Key: KAFKA-16831 URL: https://issues.apache.org/jira/browse/KAFKA-16831

[jira] [Created] (KAFKA-16664) Re-add EventAccumulator.take(timeout)

Jeff Kim created KAFKA-16664: Summary: Re-add EventAccumulator.take(timeout) Key: KAFKA-16664 URL: https://issues.apache.org/jira/browse/KAFKA-16664 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-16663) CoordinatorRuntime write timer tasks should be cancelled once HWM advances

Jeff Kim created KAFKA-16663: Summary: CoordinatorRuntime write timer tasks should be cancelled once HWM advances Key: KAFKA-16663 URL: https://issues.apache.org/jira/browse/KAFKA-16663 Project: Kafka

[jira] [Created] (KAFKA-16379) add metric to measure time spent in purgatory

Jeff Kim created KAFKA-16379: Summary: add metric to measure time spent in purgatory Key: KAFKA-16379 URL: https://issues.apache.org/jira/browse/KAFKA-16379 Project: Kafka Issue Type: Sub-task

[jira] [Resolved] (KAFKA-16299) Classic group error responses should contain a no generation id and an empty members list

[ https://issues.apache.org/jira/browse/KAFKA-16299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Kim resolved KAFKA-16299. -- Resolution: Invalid Not needed. Refer to https://github.com/apache/kafka/pull/15446

[jira] [Created] (KAFKA-16317) Add event rate in GroupCoordinatorRuntimeMetrics

Jeff Kim created KAFKA-16317: Summary: Add event rate in GroupCoordinatorRuntimeMetrics Key: KAFKA-16317 URL: https://issues.apache.org/jira/browse/KAFKA-16317 Project: Kafka Issue Type: Sub

[jira] [Created] (KAFKA-16307) investigate EventAccumulator thread idle ratio metric

Jeff Kim created KAFKA-16307: Summary: investigate EventAccumulator thread idle ratio metric Key: KAFKA-16307 URL: https://issues.apache.org/jira/browse/KAFKA-16307 Project: Kafka Issue Type

[jira] [Created] (KAFKA-16306) GroupCoordinatorService logger is not configured

Jeff Kim created KAFKA-16306: Summary: GroupCoordinatorService logger is not configured Key: KAFKA-16306 URL: https://issues.apache.org/jira/browse/KAFKA-16306 Project: Kafka Issue Type: Sub

[jira] [Created] (KAFKA-16299) Classic group error responses should contain a no generation id and an empty members list

Jeff Kim created KAFKA-16299: Summary: Classic group error responses should contain a no generation id and an empty members list Key: KAFKA-16299 URL: https://issues.apache.org/jira/browse/KAFKA-16299

[jira] [Created] (KAFKA-16148) Implement GroupMetadataManager#onUnloaded

Jeff Kim created KAFKA-16148: Summary: Implement GroupMetadataManager#onUnloaded Key: KAFKA-16148 URL: https://issues.apache.org/jira/browse/KAFKA-16148 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-16106) group size counters do not reflect the actual sizes when operations fail

Jeff Kim created KAFKA-16106: Summary: group size counters do not reflect the actual sizes when operations fail Key: KAFKA-16106 URL: https://issues.apache.org/jira/browse/KAFKA-16106 Project: Kafka

[jira] [Created] (KAFKA-16030) new group coordinator should check if partition goes offline during load

Jeff Kim created KAFKA-16030: Summary: new group coordinator should check if partition goes offline during load Key: KAFKA-16030 URL: https://issues.apache.org/jira/browse/KAFKA-16030 Project: Kafka

[jira] [Created] (KAFKA-15982) Move GenericGroup state metrics to `GroupCoordinatorMetricsShard`

Jeff Kim created KAFKA-15982: Summary: Move GenericGroup state metrics to `GroupCoordinatorMetricsShard` Key: KAFKA-15982 URL: https://issues.apache.org/jira/browse/KAFKA-15982 Project: Kafka

[jira] [Created] (KAFKA-15981) only record GenericGroup state metrics once per group during load

Jeff Kim created KAFKA-15981: Summary: only record GenericGroup state metrics once per group during load Key: KAFKA-15981 URL: https://issues.apache.org/jira/browse/KAFKA-15981 Project: Kafka

[jira] [Created] (KAFKA-15910) New group coordinator needs to generate snapshots while loading

Jeff Kim created KAFKA-15910: Summary: New group coordinator needs to generate snapshots while loading Key: KAFKA-15910 URL: https://issues.apache.org/jira/browse/KAFKA-15910 Project: Kafka

[jira] [Created] (KAFKA-15870) Move new group coordinator metrics from Yammer to Metrics

Jeff Kim created KAFKA-15870: Summary: Move new group coordinator metrics from Yammer to Metrics Key: KAFKA-15870 URL: https://issues.apache.org/jira/browse/KAFKA-15870 Project: Kafka Issue Type

[jira] [Created] (KAFKA-15587) Limit the number of records per batch in GroupCoordinatorShard#cleanupGroupMetadata

Jeff Kim created KAFKA-15587: Summary: Limit the number of records per batch in GroupCoordinatorShard#cleanupGroupMetadata Key: KAFKA-15587 URL: https://issues.apache.org/jira/browse/KAFKA-15587 Project

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

Hi Calvin, Thanks for the KIP! I'm still digesting it but I have two questions: > In the scenario raised in the motivation section, the server may receive ack=1 messages during T1 and advance High Watermark when the leader is the only one in ISR. To confirm, the current protocol allows advancing

[jira] [Created] (KAFKA-15207) ProducerIdManager#generateProducerId() should return different error code for newer clients

Jeff Kim created KAFKA-15207: Summary: ProducerIdManager#generateProducerId() should return different error code for newer clients Key: KAFKA-15207 URL: https://issues.apache.org/jira/browse/KAFKA-15207

[jira] [Created] (KAFKA-15029) Make ProducerIdBlock size configurable

Jeff Kim created KAFKA-15029: Summary: Make ProducerIdBlock size configurable Key: KAFKA-15029 URL: https://issues.apache.org/jira/browse/KAFKA-15029 Project: Kafka Issue Type: Task

Re: [DISCUSS] Apache Kafka 3.5.0 release

Hi Mickael, Thank you. Best, Jeff On Wed, Apr 5, 2023 at 1:28 PM Mickael Maison wrote: > Hi Jeff, > > Ok, I've added KIP-915 to the release plan. > > Thanks, > Mickael > > On Wed, Apr 5, 2023 at 6:48 PM Jeff Kim > wrote: > > > > Hi Mickael, &

Re: [DISCUSS] Apache Kafka 3.5.0 release

Hi Mickael, I would like to bring up that KIP-915 proposes to patch 3.5 although it missed the KIP freeze date. If the patch is done before the feature freeze date, 4/13, would this be acceptable? If so, should this be added to the 3.5.0 Release Plan wiki? Best, Jeff On Mon, Mar 27, 2023 at 1:02

Re: [VOTE] KIP-915: Txn and Group Coordinator Downgrade Foundation

jeu. 30 mars 2023 à 14:41, David Jacot > > > > a écrit : > > > > > > > > > > Thanks for the KIP, Jeff. > > > > > > > > > > +1 (binding) > > > > > > > > > > On Wed, Mar 29, 2023 at 4:48 PM Jeff Kim &

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

y Jeff. > > 2. Could we document that removing a non-tagged field is not backward > compatible in the KIP? > > Jun > > On Fri, Mar 31, 2023 at 7:17 AM Jeff Kim > wrote: > > > Hi Jun, > > > > Thanks for the response. > > > > 1. Thanks for the c

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

f "// KIP-915 do not bump the version" in green > in the KIP. Are those expected? > > 2. So, does that mean downgrade doesn't support the removal of a non-tagged > field? > > Thanks, > > Jun > > On Thu, Mar 30, 2023 at 2:57 PM Jeff Kim > wrote: &g

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

> commit > > to releasing an update for each of these versions or we reduce the set of > > versions we backport the change to. > > > > Ismael > > > > On Wed, Mar 15, 2023 at 4:57 PM Jeff Kim > > wrote: > > > > > Hi folks, > > > > > &g

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

to read it? > I think this is the sort of thing we already do, but just wanted to > clarify. We don't need to distinguish versions since we just need to > distinguish if the tagged field is there or not. > > Thanks, > Justine > > > On Tue, Mar 28, 2023 at 3:42 PM

[VOTE] KIP-915: Txn and Group Coordinator Downgrade Foundation

Hi all, I would like to start the vote on KIP-915: Txn and Group Coordinator Downgrade Foundation. KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-915%3A+Txn+and+Group+Coordinator+Downgrade+Foundation Discussion thread: https://lists.apache.org/thread/myyl7phrnl6vsnzdskrntgzpvshoj4n3

[jira] [Created] (KAFKA-14869) txn and group coordinator downgrade foundation

Jeff Kim created KAFKA-14869: Summary: txn and group coordinator downgrade foundation Key: KAFKA-14869 URL: https://issues.apache.org/jira/browse/KAFKA-14869 Project: Kafka Issue Type: Task

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

ew field. The new field is > semantically optional (structurally it is always optional since it is > a tagged field) but we want to make the distinction between a record > generated by 3.6 and one generated by 3.7. How do we resolve the > ambiguity? > > Thanks! > Alexandre >

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

this become a problem in > the future if there is ever a need to segregate two distinct > "versions" of the appended record structure held by the tagged fields? > > Thanks, > Alexandre > > Le jeu. 23 mars 2023 à 18:15, Jeff Kim a > écrit : > > > > Hi Y

[jira] [Created] (KAFKA-14856) Next gen group coordinator dynamic configs should be forward compatible

Jeff Kim created KAFKA-14856: Summary: Next gen group coordinator dynamic configs should be forward compatible Key: KAFKA-14856 URL: https://issues.apache.org/jira/browse/KAFKA-14856 Project: Kafka

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

introduced? Will there be any need to bump the version again in > the future? > To enforce the version not bumping, is it possible to have a unit test to > gate? > > > On Wed, Mar 22, 2023 at 3:19 PM Jeff Kim > wrote: > > > Hi all, > > > > After discus

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

/confluence/display/KAFKA/KIP-915%3A+Txn+and+Group+Coordinator+Downgrade+Foundation Thanks, Jeff On Mon, Mar 20, 2023 at 7:37 PM Jeff Kim wrote: > Hi David and Justine, > > Thank you both for the detailed feedback. > > David, > > 1. That makes sense. I revised the "Readin

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

ve. The > > alternative sounds clearly better when you read it so we should really > > explain the reason to reject it. 1) One issue that you mention is that > the > > log must be compacted before downgrading and we don't really control this > > process. 2) Transaction

[DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

Hi folks, I would like to start a discussion thread for KIP-915: Next Gen Group Coordinator Downgrade Path which proposes the downgrade design for the new group coordinator introduced in KIP-848

Request for Permission

Hi, I would like to request permission to contribute to the Apache Kafka wiki. For JIRA ID: jeffkbkim Wiki ID: jeff.kim Thanks, Jeff

[jira] [Created] (KAFKA-14694) RPCProducerIdManager should not wait for a new block

Jeff Kim created KAFKA-14694: Summary: RPCProducerIdManager should not wait for a new block Key: KAFKA-14694 URL: https://issues.apache.org/jira/browse/KAFKA-14694 Project: Kafka Issue Type: Bug

Re: [DISCUSS] KIP-890 Server Side Defense

Hi Justine, Thanks for the KIP! I have two questions: 1) For new clients, we can once again return an error UNKNOWN_PRODUCER_ID for sequences that are non-zero when there is no producer state present on the server. This will indicate we missed the 0 sequence and we don't yet want to write to the

[jira] [Created] (KAFKA-14407) Able to access offset index memory map even after memory map flag is turned off

Jeff Kim created KAFKA-14407: Summary: Able to access offset index memory map even after memory map flag is turned off Key: KAFKA-14407 URL: https://issues.apache.org/jira/browse/KAFKA-14407 Project

[jira] [Created] (KAFKA-14380) consumer should refresh preferred read replica on metadata update

Jeff Kim created KAFKA-14380: Summary: consumer should refresh preferred read replica on metadata update Key: KAFKA-14380 URL: https://issues.apache.org/jira/browse/KAFKA-14380 Project: Kafka

[jira] [Created] (KAFKA-14378) consumer should refresh preferred read replica on update metadata

Jeff Kim created KAFKA-14378: Summary: consumer should refresh preferred read replica on update metadata Key: KAFKA-14378 URL: https://issues.apache.org/jira/browse/KAFKA-14378 Project: Kafka

[jira] [Created] (KAFKA-14379) consumer should refresh preferred read replica on update metadata

Jeff Kim created KAFKA-14379: Summary: consumer should refresh preferred read replica on update metadata Key: KAFKA-14379 URL: https://issues.apache.org/jira/browse/KAFKA-14379 Project: Kafka

[jira] [Created] (KAFKA-14372) RackAwareReplicaSelector should choose a replica from the isr

Jeff Kim created KAFKA-14372: Summary: RackAwareReplicaSelector should choose a replica from the isr Key: KAFKA-14372 URL: https://issues.apache.org/jira/browse/KAFKA-14372 Project: Kafka Issue

[jira] [Created] (KAFKA-14334) DelayedFetch purgatory not completed when appending as follower

Jeff Kim created KAFKA-14334: Summary: DelayedFetch purgatory not completed when appending as follower Key: KAFKA-14334 URL: https://issues.apache.org/jira/browse/KAFKA-14334 Project: Kafka

[jira] [Created] (KAFKA-14075) Consumer Group deletion does not delete pending transactional offset commits

Jeff Kim created KAFKA-14075: Summary: Consumer Group deletion does not delete pending transactional offset commits Key: KAFKA-14075 URL: https://issues.apache.org/jira/browse/KAFKA-14075 Project: Kafka

[jira] [Created] (KAFKA-13007) KafkaAdminClient getListOffsetsCalls builds cluster snapshot for every topic partition

Jeff Kim created KAFKA-13007: Summary: KafkaAdminClient getListOffsetsCalls builds cluster snapshot for every topic partition Key: KAFKA-13007 URL: https://issues.apache.org/jira/browse/KAFKA-13007

[jira] [Created] (KAFKA-12264) JMH BaseRecordBatchBenchmark ZSTD error

Jeff Kim created KAFKA-12264: Summary: JMH BaseRecordBatchBenchmark ZSTD error Key: KAFKA-12264 URL: https://issues.apache.org/jira/browse/KAFKA-12264 Project: Kafka Issue Type: Bug

[jira] [Created] (KAFKA-10452) Only expire preferred read replica if a leader is alive for the topic

Jeff Kim created KAFKA-10452: Summary: Only expire preferred read replica if a leader is alive for the topic Key: KAFKA-10452 URL: https://issues.apache.org/jira/browse/KAFKA-10452 Project: Kafka

[jira] [Created] (KAFKA-10328) MockConsumer behaves differently than KafkaConsumer

Jeff Kim created KAFKA-10328: Summary: MockConsumer behaves differently than KafkaConsumer Key: KAFKA-10328 URL: https://issues.apache.org/jira/browse/KAFKA-10328 Project: Kafka Issue Type: Bug

[jira] [Created] (KAFKA-10193) Add preemption for ControllerEvent that have callbacks

Jeff Kim created KAFKA-10193: Summary: Add preemption for ControllerEvent that have callbacks Key: KAFKA-10193 URL: https://issues.apache.org/jira/browse/KAFKA-10193 Project: Kafka Issue Type

[jira] [Created] (KAFKA-10189) Reset metric EventQueueTimeMs

Jeff Kim created KAFKA-10189: Summary: Reset metric EventQueueTimeMs Key: KAFKA-10189 URL: https://issues.apache.org/jira/browse/KAFKA-10189 Project: Kafka Issue Type: Bug Components