Advice on how to comprehend Kafka source code

Hi community, After being a Kafka user for several years, I want to know Kafka better maybe on a code level. I am wondering if anyone can give me any advice. To be more specific, I am thinking about the following things. 0. Is reading source code the best way to learn more about Kafka ? 1. Which

Re: [DISCUSS] KIP-1091: Improved Kafka Streams operator metrics

Thanks for the KIP Bill. Very useful addition to be able to collect all metrics broker side. The KIP proposes to add a new thread-level metric for the thread state similar to the existing client "state" metric. The KIP lacks the corresponding definition though -- it only list the three new GAU

Re: [DISCUSS] Require KIPs to include "How to teach this section"

The KIP already has a section "Documentation Plan" https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=50859709#KIPTemplate-DocumentationPlan If it's not used properly, committers reviewing (and voting) KIPs must be reminded to pay more attention and emphasis on good documentation,

Jenkins build is still unstable: Kafka » Kafka PowerPC Daily » test-powerpc #105

See

[jira] [Created] (KAFKA-17921) Support SASL_PLAINTEXT protocol with java.security.auth.login.config

Chia-Ping Tsai created KAFKA-17921: -- Summary: Support SASL_PLAINTEXT protocol with java.security.auth.login.config Key: KAFKA-17921 URL: https://issues.apache.org/jira/browse/KAFKA-17921 Project: Kaf

[jira] [Created] (KAFKA-17922) add helper to ClusterInstance to create client component

Chia-Ping Tsai created KAFKA-17922: -- Summary: add helper to ClusterInstance to create client component Key: KAFKA-17922 URL: https://issues.apache.org/jira/browse/KAFKA-17922 Project: Kafka

[jira] [Resolved] (KAFKA-6780) log cleaner shouldn't clean messages beyond high watermark

[ https://issues.apache.org/jira/browse/KAFKA-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-6780. Resolution: Duplicate This is already fixed by https://issues.apache.org/jira/browse/KAFKA-5196 > log clea

Re: [VOTE] 3.9.0 RC5

Argh. That should read "5 +1 votes". :) Thanks, everyone. Colin On Fri, Nov 1, 2024, at 14:58, Colin McCabe wrote: > Hi all, > > This vote passes with 7 +1 votes (3 bindings) and no 0 or -1 votes. > > +1 votes > PMC Members: > * Justine Olshan > * Bill Bejeck > * David Arthur > * Luke Chen > > C

Re: [VOTE] 3.9.0 RC5

Hi all, This vote passes with 7 +1 votes (3 bindings) and no 0 or -1 votes. +1 votes PMC Members: * Justine Olshan * Bill Bejeck * David Arthur * Luke Chen Community: * Federico Valeri 0 votes * No votes -1 votes * No votes Vote thread: https://lists.apache.org/thread/og8l24z8oj87x0zsoqddfxo5

[jira] [Created] (KAFKA-17920) GroupType.parse should return Optional to represent the null or empty

Chia-Ping Tsai created KAFKA-17920: -- Summary: GroupType.parse should return Optional to represent the null or empty Key: KAFKA-17920 URL: https://issues.apache.org/jira/browse/KAFKA-17920 Project: Ka

Re: [DISCUSS] Require KIPs to include "How to teach this section"

On Fri, Nov 1, 2024, at 07:08, Claude Warren, Jr wrote: > I like this idea. I'm not sure what the section should be called but It > should spell out what changes from a customer (I don't like the term user, > drug dealers have users -- we should have customers) point of view and from > a developer

Re: [DISCUSS] KIP-1103: Additional metrics for cooperative consumption

Thanks Jun and Andrew for reviewing and feedback. J1: I was thinking of having metrics as per the APIs in SharePartitionManager but I think you are right. This is something not beneficial to track as the release should only happen when a client sends a leave group request. I have removed this metr

[jira] [Resolved] (KAFKA-17803) Reconcile Differences in MockLog and KafkaMetadataLog `read` Implementation

[ https://issues.apache.org/jira/browse/KAFKA-17803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-17803. - Fix Version/s: 4.0.0 Resolution: Fixed merged the PR to trunk > Reconcile Differences in MockLog

Re: [DISCUSS] Require KIPs to include "How to teach this section"

I like this idea. I'm not sure what the section should be called but It should spell out what changes from a customer (I don't like the term user, drug dealers have users -- we should have customers) point of view and from a developer point of view. I can see cases where the change is not visible

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

Thanks for the review! > 5 Updated the KIP to include the `isEmpty` method in the transaction index > 6 You're right. The offset parameter will be equal to the next-segment-to-consider base offset. But, the API introduced in RLMM is for *one* epoch. The next epoch start-offset may not be the base

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

Last few things - # 5 About setting the TrxIndexEmpty field, could we introduce an isEmpty() function in TransactionIndex which has the following implementation: public boolean isEmpty() { return !iterable().iterator().hasNext(); } The advantages of this approach is: 1. It works for both ca

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

Hi Divij, Thanks for the detailed review! > 1, 2, 3, 4 Updated the KIP-1058 with the feedback received and also opened a draft PR for #17659

Re: [DISCUSS] Minimum constraint for segment.ms

Hi, Thanks for taking this initiative forward. I'd like to propose formatting of that table to also include current unchanged default values and unchanged min/max constraints. In my opinion it would improve the readability of the whole document. Having those values listed even unchanged makes it p

Re: [DISCUSS] KIP-1101: Trigger rebalance on rack topology changes

hi PoAn Thanks for for this KIP! Q0: Could you add more details about `A topic partition has rack change`? IIRC, the "rack change" includes both follower and leader, right? Q1: Could you please add the 'concerns' we discussed to the Motivation section? This should include topics like 'computatio

[jira] [Resolved] (KAFKA-17912) Align string conversion of SharePartitionKey

[ https://issues.apache.org/jira/browse/KAFKA-17912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-17912. Resolution: Fixed > Align string conversion of SharePartitionKey > ---

[DISCUSS] KIP-1101: Trigger rebalance on rack topology changes

Hi all, I would like to start a discussion thread on KIP-1101. Trigger rebalance on rack topology changes. In this KIP, we aim to use less memory / disk resources to detect rack changes in the new coordinator. https://cwiki.apache.org/confluence/display/KAFKA/KIP-1101%3A+Trigger+rebalance+on+ra

[jira] [Created] (KAFKA-17919) enable back the failing testShareGroups test

Chia-Ping Tsai created KAFKA-17919: -- Summary: enable back the failing testShareGroups test Key: KAFKA-17919 URL: https://issues.apache.org/jira/browse/KAFKA-17919 Project: Kafka Issue Type: