Re: [VOTE] KIP-1118: Add Deadlock Protection on Producer Network Thread

[TengYao Chi]

Hello everyone, Happy New Year 😊

I'm writing to inform you that the vote is now closed.
The KIP-1118 has been accepted with 3 binding +1s from Chia-Ping, Andrew,
and Kamal, as well as 3 non-binding +1s from Kirk, TaiJu, and 郭骏旺.

Thank you all for your participation!

Sincerely,
TengYao

Kamal Chandraprakash  於 2025年1月2日 週四
下午9:16寫道：

> +1 (binding). Thanks for the KIP!
>
> On Mon, Dec 23, 2024, 08:28 TengYao Chi  wrote:
>
> > Hi everyone,
> >
> > As the vote has been pending for a week, I would like to bump it
> manually.
> > Thank you for your attention.
> >
> > Sincerely,
> > TengYao
> >
> > Andrew Schofield  於 2024年12月16日 週一
> > 下午10:25寫道：
> >
> > > +1 (binding)
> > >
> > > 
> > > From: TaiJu Wu 
> > > Sent: 16 December 2024 09:41
> > > To: dev@kafka.apache.org 
> > > Subject: Re: [VOTE] KIP-1118: Add Deadlock Protection on Producer
> Network
> > > Thread
> > >
> > > +1（non-binding)
> > >
> > > On Mon, Dec 16, 2024 at 5:41 PM Chia-Ping Tsai 
> > wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > 郭骏旺  於 2024年12月16日 週一 上午9:16寫道：
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > On Mon, Dec 9, 2024 at 10:33 AM TengYao Chi 
> > > wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > Based on our discussion
> > > > > > <
> > >
> >
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2F8npgn4d7wwlwvdy1h4xpbdlffksstddl&data=05%7C02%7C%7Ccc8b694fdaba493857f208dd1db5f2b9%7C84df9e7fe9f640afb435%7C1%7C0%7C638699389480859615%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=EfyW7tm2vii6%2BTbkr8avwh2vr8UHnK6FwZffCqhcSMM%3D&reserved=0
> > > >
> > > > > > regarding
> > > > > > KIP-1118
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-1118%253A%2BAdd%2BDeadlock%2BProtection%2Bon%2BProducer%2BNetwork%2BThread&data=05%7C02%7C%7Ccc8b694fdaba493857f208dd1db5f2b9%7C84df9e7fe9f640afb435%7C1%7C0%7C638699389480882996%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=v%2FrTVcmFizDqDGiD9HPYBDaIjBTjWgvwqMyPOEUU8ew%3D&reserved=0
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1118%3A+Add+Deadlock+Protection+on+Producer+Network+Thread
> > > >
> > > > > > >,
> > > > > > I believe this KIP is now ready for a vote.
> > > > > >
> > > > > > Sincerely,
> > > > > > TengYao
> > > > > >
> > > > >
> > > >
> > >
> >
>

Jenkins build is still unstable: Kafka » Kafka PowerPC Daily » test-powerpc #167

[Apache Jenkins Server]

See 

[DISCUSS] KIP-1095: Kafka Canary Isolation

[Chen Zhifeng]

Hi Everyone,

Started a thread to discuss KIP-1095: Kafka Canary Isolation (link

)

Canary isolation aims to improve Kafka service quality by reducing blast
radius of bad Kafka deployment to a small portion of traffic.

The key solution including
1. define canary broker (a new broker metadata - pod is introduced)
2. define canary partition - a small portion of partitions placed on canary
brokers
3. producer/consumer use topic metadata to route and isolate canary traffic
in canary

With canary isolation, it's expected to detect deployment caused failure in
canary and rollback before it impact whole production.

Regards,

[jira] [Created] (KAFKA-18395) Initialize KafkaRaftMetrics without QuorumState to prevent circularity

[Kevin Wu (Jira)]

Kevin Wu created KAFKA-18395:


 Summary: Initialize KafkaRaftMetrics without QuorumState to 
prevent circularity
 Key: KAFKA-18395
 URL: https://issues.apache.org/jira/browse/KAFKA-18395
 Project: Kafka
  Issue Type: Improvement
Reporter: Kevin Wu
Assignee: José Armando García Sancio


To implement https://issues.apache.org/jira/browse/KAFKA-16524, `QuorumState` 
needs to be removed from `KafkaRaftMetrics` constructor to avoid a circularity. 
That PR's approach is to move `QuorumState` to a `KafkaRaftMetrics#initialize` 
method for now.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-18397) Fix the scenario where acknowledgement callback is being called on null acknowledgements

[Chirag Wadhwa (Jira)]

Chirag Wadhwa created KAFKA-18397:
-

 Summary: Fix the scenario where acknowledgement callback is being 
called on null acknowledgements
 Key: KAFKA-18397
 URL: https://issues.apache.org/jira/browse/KAFKA-18397
 Project: Kafka
  Issue Type: Sub-task
Reporter: Chirag Wadhwa






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-18273) Implement kafka-share-groups.sh --describe --verbose

[Andrew Schofield (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-18273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Schofield resolved KAFKA-18273.
--
Resolution: Fixed

> Implement kafka-share-groups.sh --describe --verbose
> 
>
> Key: KAFKA-18273
> URL: https://issues.apache.org/jira/browse/KAFKA-18273
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Andrew Schofield
>Assignee: Andrew Schofield
>Priority: Major
> Fix For: 4.1.0
>
>
> This is the share groups part of KIP-1099.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-4273) Streams DSL - Add TTL / retention period support for intermediate topics and state stores

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-4273.

Fix Version/s: 2.1.0
   Resolution: Fixed

> Streams DSL - Add TTL / retention period support for intermediate topics and 
> state stores
> -
>
> Key: KAFKA-4273
> URL: https://issues.apache.org/jira/browse/KAFKA-4273
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Davor Poldrugo
>Priority: Major
> Fix For: 2.1.0
>
>
> Hi!
> I'm using Streams DSL (0.10.0.1), which can only use RocksDB for local state 
> as far as I know - it's not configurable.
> In my use case my data has TTL / retnetion period. It's 48 hours. After that 
> - data can be discarded.
> I join two topics: "messages" and "prices" using windowed inner join.
> The two intermediate Kafka topics for this join are named:
>  * messages-prices-join-this-changelog
>  * messages-prices-join-other-changelog
> Since these topics are created as compacted by Kafka Streams, and I don't 
> wan't to keep data forever, I have altered them to not use compaction. Right 
> now my RocksDB state stores grow indefinitely, and I don't have any options 
> to define TTL, or somehow periodically clean the older data.
> A "hack" that I use to keep my disk usage low - I have schedulled a job to 
> periodically stop Kafka Streams instances - one at the time. This triggers a 
> rebalance, and partitions migrate to other instances. When the instance is 
> started again, there's another rebalance, and sometimes this instance starts 
> processing partitions that wasn't processing before the stop - which leads to 
> deletion of the RocksDB state store for those partitions 
> (state.cleanup.delay.ms). In the next rebalance the local store is recreated 
> with a restore consumer - which reads data from - as previously mentioned - a 
> non compacted topic. And this effectively leads to a "hacked TTL support" in 
> Kafka Streams DSL.
> Questions:
>  * Do you think would be reasonable to add support in the DSL api to define 
> TTL for local store?
>  * Which opens another question - there are use cases which don't need the 
> intermediate topics to be created as "compact". Could also this be added to 
> the DSL api? Maybe only this could be added, and this flag should also be 
> used for the RocksDB TTL. Of course in this case another config would be 
> mandatory - the retention period or TTL for the intermediate topics and the 
> state stores. I saw there is a new cleanup.policy - compact_and_delete - 
> added with KAFKA-4015.
>  * Which also leads to another question, maybe some intermediate topics / 
> state stores need different TTL, so a it's not as simple as that. But after 
> KAFKA-3870, it will be easier.
> RocksDB supports TTL:
>  * 
> https://github.com/apache/kafka/blob/0.10.0.1/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java#L166
>  * https://github.com/facebook/rocksdb/wiki/Time-to-Live
>  * 
> https://github.com/facebook/rocksdb/blob/master/java/src/main/java/org/rocksdb/TtlDB.java
> A somehow similar issue: KAFKA-4212



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-4212) Add a key-value store that is a TTL persistent cache

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-4212.

Fix Version/s: 2.1.0
   Resolution: Fixed

> Add a key-value store that is a TTL persistent cache
> 
>
> Key: KAFKA-4212
> URL: https://issues.apache.org/jira/browse/KAFKA-4212
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Priority: Major
>  Labels: api
> Fix For: 2.1.0
>
>
> Some jobs needs to maintain as state a large set of key-values for some 
> period of time.  I.e. they need to maintain a TTL cache of values potentially 
> larger than memory. 
> Currently Kafka Streams provides non-windowed and windowed key-value stores.  
> Neither is an exact fit to this use case.  
> The {{RocksDBStore}}, a {{KeyValueStore}}, stores one value per key as 
> required, but does not support expiration.  The TTL option of RocksDB is 
> explicitly not used.
> The {{RocksDBWindowsStore}}, a {{WindowsStore}}, can expire items via segment 
> dropping, but it stores multiple items per key, based on their timestamp.  
> But this store can be repurposed as a cache by fetching the items in reverse 
> chronological order and returning the first item found.
> KAFKA-2594 introduced a fixed-capacity in-memory LRU caching store, but here 
> we desire a variable-capacity memory-overflowing TTL caching store.
> Although {{RocksDBWindowsStore}} can be repurposed as a cache, it would be 
> useful to have an official and proper TTL cache API and implementation.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[VOTE] KIP-1107: Adding topic-level acks and compressions for producers

[TaiJu Wu]

Hello everyone,

Happy New Year!
There is no response about KIP-1107 for a long time so I would like to vote.

Discussion thread:
https://lists.apache.org/thread/v1sm6c59j5wppg1w6t17bvkgqlt3orp7

KIP-1107
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1107%3A++Adding+topic-level+acks+and+compressions+for+producers

Best,
TaiJuWu

Re: [VOTE] KIP-1107: Adding topic-level acks and compressions for producers

[Chia-Ping Tsai]

hi TaiJuWu

I propose that we engage in further discussions before initiating the voting 
process. This KIP involves substantial changes to the producer, and it's 
essential to ensure we have thoroughly considered all perspectives.

Best,
Chia-Ping

On 2025/01/03 03:03:03 TaiJu Wu wrote:
> Hello everyone,
> 
> Happy New Year!
> There is no response about KIP-1107 for a long time so I would like to vote.
> 
> Discussion thread:
> https://lists.apache.org/thread/v1sm6c59j5wppg1w6t17bvkgqlt3orp7
> 
> KIP-1107
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1107%3A++Adding+topic-level+acks+and+compressions+for+producers
> 
> Best,
> TaiJuWu
> 

Re: [VOTE] KIP-1107: Adding topic-level acks and compressions for producers

[TaiJu Wu]

Hi Chia-Ping

Thanks for your reply.
I will bump discussion thread and discuss it again.

Best,
TaiJuWu

Chia-Ping Tsai  於 2025年1月3日 週五 上午11:33寫道：

> hi TaiJuWu
>
> I propose that we engage in further discussions before initiating the
> voting process. This KIP involves substantial changes to the producer, and
> it's essential to ensure we have thoroughly considered all perspectives.
>
> Best,
> Chia-Ping
>
> On 2025/01/03 03:03:03 TaiJu Wu wrote:
> > Hello everyone,
> >
> > Happy New Year!
> > There is no response about KIP-1107 for a long time so I would like to
> vote.
> >
> > Discussion thread:
> > https://lists.apache.org/thread/v1sm6c59j5wppg1w6t17bvkgqlt3orp7
> >
> > KIP-1107
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1107%3A++Adding+topic-level+acks+and+compressions+for+producers
> >
> > Best,
> > TaiJuWu
> >
>

Re: [DISCUSS]KIP-1107: Adding record-level acks for producers

[TaiJu Wu]

Hello folk,

This thread is pending for a long time, I want to bump this thread and get
more feedback.
Any questions are welcome.

Best,
TaiJuWu

TaiJu Wu  於 2024年11月23日 週六 下午9:15寫道：

> Hi Chia-Ping,
>
> Sorry for late reply and thanks for your feedback to make this KIP more
> valuable.
> After initial verification, I think this can do without large changes.
>
> I have updated KIP, thanks a lot.
>
> Best,
> TaiJuWu
>
>
> Chia-Ping Tsai  於 2024年11月20日 週三 下午5:06寫道：
>
>> hi TaiJuWu
>>
>> Is there a possibility to extend this KIP to include topic-level
>> compression for the producer? This is another issue that prevents us from
>> sharing producers across different threads, as it's common to use different
>> compression types for different topics (data).
>>
>> Best,
>> Chia-Ping
>>
>> On 2024/11/18 08:36:25 TaiJu Wu wrote:
>> > Hi Chia-Ping,
>> >
>> > Thanks for your suggestions and feedback.
>> >
>> > Q1: I have updated this according your suggestions.
>> > Q2: This is necessary change since there is a assumption about
>> > *RecourdAccumulator
>> > *that all records have same acks(e.g. ProducerConfig.acks) so we need
>> to a
>> > method to distinguish which acks belong to each Batch.
>> >
>> > Best,
>> > TaiJuWu
>> >
>> > Chia-Ping Tsai  於 2024年11月18日 週一 上午2:17寫道：
>> >
>> > > hi TaiJuWu
>> > >
>> > > Q0:
>> > >
>> > > `Format: topic.acks`  the dot is acceptable character in topic
>> naming, so
>> > > maybe we should reverse the format to "acks.${topic}" to get the acks
>> of
>> > > topic easily
>> > >
>> > > Q1: `Return Map> when
>> > > RecordAccumulator#drainBatchesForOneNode is called.`
>> > >
>> > > this is weird to me, as all we need to do is pass `Map
>> to
>> > > `Sender` and make sure `Sender#sendProduceRequest` add correct acks to
>> > > ProduceRequest, right?
>> > >
>> > > Best,
>> > > Chia-Ping
>> > >
>> > >
>> > >
>> > > On 2024/11/15 05:12:33 TaiJu Wu wrote:
>> > > > Hi all,
>> > > >
>> > > > I have updated the contents of this KIP
>> > > > Please take a look and let me know what you think.
>> > > >
>> > > > Thanks,
>> > > > TaiJuWu
>> > > >
>> > > > On Thu, Nov 14, 2024 at 2:21 PM TaiJu Wu 
>> wrote:
>> > > >
>> > > > > Hi all,
>> > > > >
>> > > > > Thanks for your feeback and @Chia-Ping's help.
>> > > > > .
>> > > > > I also agree topic-level acks config is more reasonable and it can
>> > > simply
>> > > > > the story.
>> > > > > When I try implementing record-level acks, I notice I don't have
>> good
>> > > idea
>> > > > > to avoid iterating batches for get partition information (need by
>> > > > > *RecordAccumulator#partitionChanged*).
>> > > > >
>> > > > > Back to the init question how can I handle different acks for
>> batches:
>> > > > > First, we can attach *topic-level acks *to
>> > > *RecordAccumulator#TopicInfo*.
>> > > > > Second,  we can return *Map>* when
>> > > *RecordAccumulator#drainBatchesForOneNode
>> > > > > *is called. In this step, we can propagate acks to *sender*.
>> > > > > Finally, we can get the acks info and group same acks into a
>> > > > > *List>* for a node in *sender#sendProduceRequests*.
>> > > > >
>> > > > > If I missed something or there is any mistake, please let me know.
>> > > > > I will update this KIP later, thank your feedback.
>> > > > >
>> > > > > Best,
>> > > > > TaiJuWu
>> > > > >
>> > > > >
>> > > > > Chia-Ping Tsai  於 2024年11月14日 週四 上午9:46寫道：
>> > > > >
>> > > > >> hi All
>> > > > >>
>> > > > >> This KIP is based on our use case where an edge application with
>> many
>> > > > >> sensors wants to use a single producer to deliver ‘few but
>> varied’
>> > > records
>> > > > >> with different acks settings. The reason for using a single
>> producer
>> > > is to
>> > > > >> minimize resource usage on edge devices with limited hardware
>> > > capabilities.
>> > > > >> Currently, we use a producer pool to handle different acks
>> values,
>> > > which
>> > > > >> requires 3x producer instances. Additionally, this approach
>> creates
>> > > many
>> > > > >> idle producers if a sensor with a specific acks setting has no
>> data
>> > > for a
>> > > > >> while.
>> > > > >>
>> > > > >> I love David’s suggestion since the acks configuration is closely
>> > > related
>> > > > >> to the topic. Maybe we can introduce an optional configuration
>> in the
>> > > > >> producer to define topic-level acks, with the existing acks
>> being the
>> > > > >> default for all topics. This approach is not only simple but also
>> > > easy to
>> > > > >> understand and implement.
>> > > > >>
>> > > > >> Best,
>> > > > >> Chia-Ping
>> > > > >>
>> > > > >> On 2024/11/13 16:04:24 Andrew Schofield wrote:
>> > > > >> > Hi TaiJuWu,
>> > > > >> > I've been thinking for a while about this KIP before jumping
>> into
>> > > the
>> > > > >> discussion.
>> > > > >> >
>> > > > >> > I'm afraid that I don't think the approach in the KIP is the
>> best,
>> > > > >> given the design
>> > > > >> > of the Kafka protocol in this area. Essentially, each Produce
>> > > request
>> > > > >> contains
>> > > 

[jira] [Created] (KAFKA-18396) Migrate log4j1 configuration to log4j2 in KafkaDockerWrapperTest

[TengYao Chi (Jira)]

TengYao Chi created KAFKA-18396:
---

 Summary: Migrate log4j1 configuration to log4j2 in 
KafkaDockerWrapperTest
 Key: KAFKA-18396
 URL: https://issues.apache.org/jira/browse/KAFKA-18396
 Project: Kafka
  Issue Type: Improvement
Reporter: TengYao Chi
Assignee: TengYao Chi
 Fix For: 4.0.0


After log4j migration, we need to update the logging configuration in 
{{KafkaDockerWrapperTest}} from log4j1 to log4j2.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-18390) Use LinkedHashMap instead of Map in creating MetricName and SensorBuilder

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-18390:
--

 Summary: Use LinkedHashMap instead of Map in creating MetricName 
and SensorBuilder
 Key: KAFKA-18390
 URL: https://issues.apache.org/jira/browse/KAFKA-18390
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai
 Fix For: 4.1.0


see https://github.com/apache/kafka/pull/18232#discussion_r1900425014



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-18391) Skip running "Flaky Test Report" job on forks

[Vadym Zhytkevych (Jira)]

Vadym Zhytkevych created KAFKA-18391:


 Summary: Skip running "Flaky Test Report" job on forks
 Key: KAFKA-18391
 URL: https://issues.apache.org/jira/browse/KAFKA-18391
 Project: Kafka
  Issue Type: Improvement
  Components: core
Reporter: Vadym Zhytkevych
Assignee: Vadym Zhytkevych
 Attachments: image-2025-01-02-12-28-26-839.png

Job

 
{code:java}
name: Flaky Test Report
on:
  workflow_dispatch:  # Let us run manually

  schedule:
- cron: '0 6 * * *'   # Run daily at 6am UTC

jobs:
  flaky-test-report:
name: Flaky Test Report {code}
keeps running and failing on forked repositories due to missing secrets

 
{code:java}
DEVELOCITY_ACCESS_TOKEN: ${{ secrets.DV_API_ACCESS }}{code}
Update ci to skip the run on forked repositories to decrease number of 
unnecessary emails and failed builds. 

!image-2025-01-02-12-28-26-839.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1114 Introducing Chunk in Partition

[David Arthur]

Hey De Gao, thanks for the KIP!

As you’re probably aware, a Partition is a logical construct in Kafka. A
broker hosts a partition which is composed of physical log segments. Only
the active segment is being written to and the others are immutable. The
concept of a Chunk sounds quite similar to our log segments.

>From what I can tell reading the KIP, the main difference is that a Chunk
can have its own assignment and therefore be replicated across different
brokers.

> Horizontal scalability: the data was distributed more evenly to brokers
in cluster. Also achieving a more flexible resource allocation.

I think this is only true in cases where we have a small number of
partitions with a large amount of data. I have certainly seen cases where a
small number of partitions can cause trouble with balancing the cluster.

The idea of shuffling around older data in order to spread out the load is
interesting. It does seem like it would increase the complexity of the
client a bit when it comes to consuming the old data. Usually the client
can just read from a single replica from the beginning of the log to the
end. With this proposal, the client would need to hop around between
replicas as it crossed the chunk boundaries.

> Better load balancing: The read of partition data, especially early data
can be distributed to more nodes other than just leader nodes.

As you know, this is already possible with KIP-392. I guess the idea with
the chunks is that clients would be reading older data from less busy
brokers (i.e., brokers which are not the leader, or perhaps not even a
follower of the active chunk). I’m not sure this would always result in
better load balancing. It seems a bit situational.

> Increased fault tolerance: failure of leader node will not impact read
older data.

I don’t think this proposal changes the fault tolerance. A failure of a
leader results in a failover to a follower. If a client is consuming using
KIP-392, a leader failure will not affect the consumption (besides updating
the clients metadata).

--

I guess I'm missing a key point here. What problem is this trying to solve?
Is it a solution for the "single partition" problem? (i.e., a topic with
one partition and a lot of data)

Thanks!
David A

On Tue, Dec 31, 2024 at 3:24 PM De Gao  wrote:

> Thanks for the comments. I have updated the proposal to compare with
> tiered storage and fetch from replica. Please check.
>
> Thanks.
>
> On 11 December 2024 08:51:43 GMT, David Jacot 
> wrote:
> >Hi,
> >
> >Thanks for the KIP. The community is pretty busy with the Apache Kafka 4.0
> >release so I suppose that no one really had the time to engage in
> reviewing
> >the KIP yet. Sorry for this!
> >
> >I just read the motivation section. I think that it is an interesting
> idea.
> >However, I wonder if this is still needed now that we have tier storage in
> >place. One of the big selling points of tier storage was that clusters
> >don't have to replicate tiered data anymore. Could you perhaps extend the
> >motivation of the KIP to include tier storage in the reflexion?
> >
> >Best,
> >David
> >
> >On Tue, Dec 10, 2024 at 10:46 PM De Gao  wrote:
> >
> >> Hi All:
> >>
> >> There were no discussion in the past week. Just want to double check if
> I
> >> missed anything?
> >> What should be the expectations on KIP discussion?
> >>
> >> Thank you!
> >>
> >> De Gao
> >>
> >> On 1 December 2024 19:36:37 GMT, De Gao  wrote:
> >> >Hi All:
> >> >
> >> >I would like to start the discussion of KIP-1114 Introducing Chunk in
> >> Partition.
> >> >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1114%3A+Introducing+Chunk+in+Partition
> >> >This KIP is complicated so I expect discussion will take longer time.
> >> >
> >> >Thank you in advance.
> >> >
> >> >De Gao
> >>
>


-- 
David Arthur

Re: [DISCUSS] KIP-1114 Introducing Chunk in Partition

[Kamal Chandraprakash]

Hi Deo,

Thanks for the KIP!

"However the limit of messages in a single partition replica is very big.
This could lead to very big partitions (~TBs). Moving those partitions are
very time consuming and have a big impact on system performance."

One way to do faster rebalance is to have a latest-offset replica build
strategy when expanding the replicas for a partition
and ensure that the expanded replica does not serve as a leader until the
data in the older nodes expires by retention time/size.
Currently, Kafka supports only the earliest-offset strategy during
reassignment. And, this strategy will only work for topics
with cleanup policy set to "delete".

--
Kamal

On Thu, Jan 2, 2025 at 10:23 PM David Arthur  wrote:

> Hey De Gao, thanks for the KIP!
>
> As you’re probably aware, a Partition is a logical construct in Kafka. A
> broker hosts a partition which is composed of physical log segments. Only
> the active segment is being written to and the others are immutable. The
> concept of a Chunk sounds quite similar to our log segments.
>
> From what I can tell reading the KIP, the main difference is that a Chunk
> can have its own assignment and therefore be replicated across different
> brokers.
>
> > Horizontal scalability: the data was distributed more evenly to brokers
> in cluster. Also achieving a more flexible resource allocation.
>
> I think this is only true in cases where we have a small number of
> partitions with a large amount of data. I have certainly seen cases where a
> small number of partitions can cause trouble with balancing the cluster.
>
> The idea of shuffling around older data in order to spread out the load is
> interesting. It does seem like it would increase the complexity of the
> client a bit when it comes to consuming the old data. Usually the client
> can just read from a single replica from the beginning of the log to the
> end. With this proposal, the client would need to hop around between
> replicas as it crossed the chunk boundaries.
>
> > Better load balancing: The read of partition data, especially early data
> can be distributed to more nodes other than just leader nodes.
>
> As you know, this is already possible with KIP-392. I guess the idea with
> the chunks is that clients would be reading older data from less busy
> brokers (i.e., brokers which are not the leader, or perhaps not even a
> follower of the active chunk). I’m not sure this would always result in
> better load balancing. It seems a bit situational.
>
> > Increased fault tolerance: failure of leader node will not impact read
> older data.
>
> I don’t think this proposal changes the fault tolerance. A failure of a
> leader results in a failover to a follower. If a client is consuming using
> KIP-392, a leader failure will not affect the consumption (besides updating
> the clients metadata).
>
> --
>
> I guess I'm missing a key point here. What problem is this trying to solve?
> Is it a solution for the "single partition" problem? (i.e., a topic with
> one partition and a lot of data)
>
> Thanks!
> David A
>
> On Tue, Dec 31, 2024 at 3:24 PM De Gao  wrote:
>
> > Thanks for the comments. I have updated the proposal to compare with
> > tiered storage and fetch from replica. Please check.
> >
> > Thanks.
> >
> > On 11 December 2024 08:51:43 GMT, David Jacot
> 
> > wrote:
> > >Hi,
> > >
> > >Thanks for the KIP. The community is pretty busy with the Apache Kafka
> 4.0
> > >release so I suppose that no one really had the time to engage in
> > reviewing
> > >the KIP yet. Sorry for this!
> > >
> > >I just read the motivation section. I think that it is an interesting
> > idea.
> > >However, I wonder if this is still needed now that we have tier storage
> in
> > >place. One of the big selling points of tier storage was that clusters
> > >don't have to replicate tiered data anymore. Could you perhaps extend
> the
> > >motivation of the KIP to include tier storage in the reflexion?
> > >
> > >Best,
> > >David
> > >
> > >On Tue, Dec 10, 2024 at 10:46 PM De Gao  wrote:
> > >
> > >> Hi All:
> > >>
> > >> There were no discussion in the past week. Just want to double check
> if
> > I
> > >> missed anything?
> > >> What should be the expectations on KIP discussion?
> > >>
> > >> Thank you!
> > >>
> > >> De Gao
> > >>
> > >> On 1 December 2024 19:36:37 GMT, De Gao  wrote:
> > >> >Hi All:
> > >> >
> > >> >I would like to start the discussion of KIP-1114 Introducing Chunk in
> > >> Partition.
> > >> >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1114%3A+Introducing+Chunk+in+Partition
> > >> >This KIP is complicated so I expect discussion will take longer time.
> > >> >
> > >> >Thank you in advance.
> > >> >
> > >> >De Gao
> > >>
> >
>
>
> --
> David Arthur
>

[jira] [Created] (KAFKA-18392) Require client-generated member IDs for ShareGroupHeartbeat

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-18392:


 Summary: Require client-generated member IDs for 
ShareGroupHeartbeat
 Key: KAFKA-18392
 URL: https://issues.apache.org/jira/browse/KAFKA-18392
 Project: Kafka
  Issue Type: Sub-task
Reporter: Andrew Schofield
Assignee: Andrew Schofield
 Fix For: 4.1.0


This implements KIP-1082 for share groups.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-1118: Add Deadlock Protection on Producer Network Thread

[Kamal Chandraprakash]

+1 (binding). Thanks for the KIP!

On Mon, Dec 23, 2024, 08:28 TengYao Chi  wrote:

> Hi everyone,
>
> As the vote has been pending for a week, I would like to bump it manually.
> Thank you for your attention.
>
> Sincerely,
> TengYao
>
> Andrew Schofield  於 2024年12月16日 週一
> 下午10:25寫道：
>
> > +1 (binding)
> >
> > 
> > From: TaiJu Wu 
> > Sent: 16 December 2024 09:41
> > To: dev@kafka.apache.org 
> > Subject: Re: [VOTE] KIP-1118: Add Deadlock Protection on Producer Network
> > Thread
> >
> > +1（non-binding)
> >
> > On Mon, Dec 16, 2024 at 5:41 PM Chia-Ping Tsai 
> wrote:
> >
> > > +1 (binding)
> > >
> > > 郭骏旺  於 2024年12月16日 週一 上午9:16寫道：
> > >
> > > > +1 (non-binding)
> > > >
> > > > On Mon, Dec 9, 2024 at 10:33 AM TengYao Chi 
> > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > Based on our discussion
> > > > > <
> >
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2F8npgn4d7wwlwvdy1h4xpbdlffksstddl&data=05%7C02%7C%7Ccc8b694fdaba493857f208dd1db5f2b9%7C84df9e7fe9f640afb435%7C1%7C0%7C638699389480859615%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=EfyW7tm2vii6%2BTbkr8avwh2vr8UHnK6FwZffCqhcSMM%3D&reserved=0
> > >
> > > > > regarding
> > > > > KIP-1118
> > > > > <
> > > > >
> > > >
> > >
> >
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-1118%253A%2BAdd%2BDeadlock%2BProtection%2Bon%2BProducer%2BNetwork%2BThread&data=05%7C02%7C%7Ccc8b694fdaba493857f208dd1db5f2b9%7C84df9e7fe9f640afb435%7C1%7C0%7C638699389480882996%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=v%2FrTVcmFizDqDGiD9HPYBDaIjBTjWgvwqMyPOEUU8ew%3D&reserved=0
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1118%3A+Add+Deadlock+Protection+on+Producer+Network+Thread
> > >
> > > > > >,
> > > > > I believe this KIP is now ready for a vote.
> > > > >
> > > > > Sincerely,
> > > > > TengYao
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (KAFKA-18393) Remove code deprecated to implement KIP-1043

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-18393:


 Summary: Remove code deprecated to implement KIP-1043
 Key: KAFKA-18393
 URL: https://issues.apache.org/jira/browse/KAFKA-18393
 Project: Kafka
  Issue Type: Sub-task
  Components: clients
Affects Versions: 5.0.0
Reporter: Andrew Schofield
Assignee: Andrew Schofield
 Fix For: 5.0.0


The following deprecated code is removed in AK 5.0 (all in o.a.k.clients.admin 
package):
* Admin.listConsumerGroups
* ConsumerGroupDescription constructors
* ConsumerGroupDescription.state()
* ListConsumerGroupsOptions
* ListConsumerGroupsResult
* ConsumerGroupState
* ConsumerGroupListing



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15920) Flaky test - PlaintextConsumerTest.testCoordinatorFailover

[David Arthur (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Arthur resolved KAFKA-15920.
--
Resolution: Fixed

> Flaky test - PlaintextConsumerTest.testCoordinatorFailover
> --
>
> Key: KAFKA-15920
> URL: https://issues.apache.org/jira/browse/KAFKA-15920
> Project: Kafka
>  Issue Type: Bug
>Reporter: Haruki Okada
>Priority: Major
>  Labels: flaky-test
> Attachments: stdout.log
>
>
> [https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14242/14/tests/]
> {code:java}
> Error
> org.opentest4j.AssertionFailedError: expected: <0> but was: <1>
> Stacktrace
> org.opentest4j.AssertionFailedError: expected: <0> but was: <1>
> at 
> app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
> at 
> app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
> at 
> app//org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
> at 
> app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)
> at 
> app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:145)
> at app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:527)
> at 
> app//kafka.api.AbstractConsumerTest.ensureNoRebalance(AbstractConsumerTest.scala:326)
> at 
> app//kafka.api.BaseConsumerTest.testCoordinatorFailover(BaseConsumerTest.scala:109)
> at 
> java.base@11.0.16.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>  Method)
> at 
> java.base@11.0.16.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> java.base@11.0.16.1/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base@11.0.16.1/java.lang.reflect.Method.invoke(Method.java:566)
> at 
> app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
> at 
> app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> at 
> app//org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
> at 
> app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
> at 
> app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:94)
> at 
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
> at 
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
> at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> at 
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
> at 
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
> at 
> app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:218)
> at 
> app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at 
> app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:214)
> at 
> app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:139)
> at 
> app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
> at 
> app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
> at 
> app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at 
> app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
> at 
> app//org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
> at 
> app//org

[jira] [Resolved] (KAFKA-18036) TransactionsWithTieredStoreTest testReadCommittedConsumerShouldNotSeeUndecidedData is flaky

[David Arthur (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-18036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Arthur resolved KAFKA-18036.
--
Resolution: Fixed

> TransactionsWithTieredStoreTest 
> testReadCommittedConsumerShouldNotSeeUndecidedData is flaky
> ---
>
> Key: KAFKA-18036
> URL: https://issues.apache.org/jira/browse/KAFKA-18036
> Project: Kafka
>  Issue Type: Test
>Reporter: David Arthur
>Assignee: Chia-Ping Tsai
>Priority: Major
>  Labels: flaky-test
>
> https://ge.apache.org/scans/tests?search.names=CI%20workflow%2CGit%20repository&search.rootProjectNames=kafka&search.tags=github%2Ctrunk&search.tasks=test&search.timeZoneId=America%2FNew_York&search.values=CI%2Chttps:%2F%2Fgithub.com%2Fapache%2Fkafka&tests.container=org.apache.kafka.tiered.storage.integration.TransactionsWithTieredStoreTest&tests.sortField=FLAKY&tests.test=testReadCommittedConsumerShouldNotSeeUndecidedData(String%2C%20String)%5B2%5D



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1071: Streams Rebalance Protocol

[Lucas Brutschy]

Hi PoAn,

thanks for looking into the KIP!

PY1: You are right, zkBroker was added following KIP-848 during
development, and is indeed not required anymore. We don't want to
allow these RPCs on the controller, so we have to remove `zkBroker`,
not replace it with `controller`. I updated the KIP, I see you already
fixed it in trunk.

Cheers,
Lucas

On Thu, Dec 26, 2024 at 11:28 AM PoAn Yang  wrote:
>
> Hi Lucas,
>
> Thanks for the KIP.
>
> PY1: In StreamsGroupHeartbeatRequest, it uses broker and zkBroker in 
> listeners field.
> Is this intended? IIUC, the KIP will be implemented in 4.1. The zookeeper 
> will be
> removed in 4.0. Probably, we should change it as broker and controller.
> We may need a similar change for StreamsGroupDescribeRequest.
>
> Best,
> PoAn
>
> On 2024/09/02 12:23:22 Bruno Cadonna wrote:
> > Hi all,
> >
> > For your info, I updated the StreamsGroupInitialize request with the
> > following changes:
> >
> > 1. I added the topology ID to the request so that the group coordinator
> > knows for which topology it got the initialization.
> >
> > 2. I renamed field "Subtopology" to "SubtopologyId" since the field is
> > the ID of the subtopology but that was not clear from the name.
> >
> > Best,
> > Bruno
> >
> >
> > On 8/28/24 2:06 PM, Lucas Brutschy wrote:
> > > Hi Sophie,
> > >
> > > Thanks for your detailed comments - much appreciated! I think you read
> > > a version of the KIP that did not yet include the admin command-line
> > > tool and the Admin API extensions, so some of the comments are already
> > > addressed in the KIP.
> > >
> > > S1. StreamsGroupHeartbeat and StreamsGroupInitialize are called in the
> > > consumer background thread. Note that in the new consumer threading
> > > model, all RPCs are run by the background thread. Check out this:
> > > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+threading+refactor+design
> > > for more information. Both our RPCs are just part of the group
> > > membership management and do not need to be invoked explicitly by the
> > > application thread. You could imagine calling the initialize RPC
> > > explicitly (to implement explicit initialization), but this would
> > > still mean sending an event to the background thread, and the
> > > background thread in turn invokes the RPC. However, explicit
> > > initialization would require some additional public interfaces that we
> > > are not including in this KIP. StreamsGroupDescribe is called by the
> > > AdminClient, and used by the command-line tool
> > > kafka-streams-groups.sh.
> > >
> > > S2. I think the max.warmup.replicas=100 suggested by Nick was intended
> > > as the upper limit for setting the group configuration on the broker.
> > > Just to make sure that this was not a misunderstanding. By default,
> > > values above 100 should be rejected when setting a specific value for
> > > group. Are you suggesting 20 or 30 for the default value for groups,
> > > or the default upper limit for the group configuration?
> > >
> > > S3. Yes, it's supposed to be used like SHUTDOWN_APPLICATION. The
> > > MemberEpoch=-1 is a left-over from an earlier discussion. It means
> > > that the member is leaving the group, so the intention was that the
> > > member must leave the group when it asks the other members to shut
> > > down. We later reconsidered this and decided that all applications
> > > should just react to the shutdown application signal that is returned
> > > by the broker, so the client first sets the ShutdownApplication and
> > > later leaves the group. Thanks for spotting this, I removed it.
> > >
> > > S4. Not sure if this refers to the latest version of the KIP. We added
> > > an extension of the admin API and a kafka-streams-groups.sh
> > > command-line tool to the KIP already.
> > >
> > > S5. All RPCs for dealing with offsets will keep working with streams
> > > groups. The extension of the admin API is rather cosmetic, since the
> > > method names use "consumer group". The RPCs, however, are generic and
> > > do not need to be changed.
> > >
> > > S6. Yes, you can use the DeleteGroup RPC with any group ID, whether
> > > streams group or not.
> > >
> > > S7. See the admin API section.
> > >
> > > S8. I guess for both A and B, I am not sure what you are suggesting.
> > > Do you want to make the broker-side topology immutable and not include
> > > any information about the topology, like the topology ID in the RPC?
> > > It would seem that this would be a massive food-gun for people, if
> > > they start changing their topology and don't notice that the broker is
> > > looking at a completely different version of the topology. Or did you
> > > mean that there is some kind of topology ID, so that at least we can
> > > detect inconsistencies between broker and client-side topologies, and
> > > we fence out any member with an incorrect topology ID? Then we seem to
> > > end up with mostly the same RPCs and the same questions (how is the
> > > topology ID generated?). I agree t

[jira] [Created] (KAFKA-18394) Formalize durable and in-memory election state changes

[Alyssa Huang (Jira)]

Alyssa Huang created KAFKA-18394:


 Summary: Formalize durable and in-memory election state changes
 Key: KAFKA-18394
 URL: https://issues.apache.org/jira/browse/KAFKA-18394
 Project: Kafka
  Issue Type: Improvement
Reporter: Alyssa Huang


We essentially have two paths of epoch state transitions - 
1. one which involves updating on-disk election state and then in-memory epoch 
state (which includes in-memory election state changes)
2. on which does not involve any changed election state and just needs an 
in-memory epoch state change 

 

For the second case, we should explicitly check no persisted or in-memory 
election state has changed. For the first case, we should have the in-memory 
election state derive from the persisted election state change. (It should be 
impossible for the two to differ)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)