Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

Hello Andrew Thank you for your thoughtful suggestions and getting the discussion going. To AS1: In the current scenario where the server generates the UUID, if the client shuts down before receiving the memberId generated by the GC (regardless of whether it’s a graceful shutdown or not), the GC w

[jira] [Resolved] (KAFKA-17300) documentation for disablement feature

[ https://issues.apache.org/jira/browse/KAFKA-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Chen resolved KAFKA-17300. --- Fix Version/s: 3.9.0 Resolution: Fixed > documentation for disablement feature >

Re: [DISCUSS] KIP-1074: Make the replication of internal topics configurable

Hi Omina, We have not considered this path yet that you proposed and I like it as well and I would also be in favor of a solution that doesn't try to introduce a new config. Since this KIP would only make it into 4.0 anyway and your proposal seems like a smaller compatibility break that can be doc

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

Hi TengYao, Thanks for the KIP! I have a couple of comments. 1. In the motivation section, I would really start from the fundamental issue which is that the initial heartbeat is not idempotent. Then, we could describe the undesired side effects (e.g. ghost members, cannot leave without receiving

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

In my opinion, the main downside of this approach is that if you leave after receiving the first HB and the member id, the server will respond with an unknown member id error because the member is not really in the group yet. I don't want to be a hater of idempotent HB, but having a "RPC" used to

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

Hi TengYao, Thanks for your response. I’ll have just one more try to persuade. I feel that I will need to follow the approach with KIP-932 when we’ve made a decision, so I do have more than a passing interest in this. A group member in the lobby is in the group, but it does not have any assignment

[DISCUSS] KIP-1083: Increase default value of task.shutdown.graceful.timeout.ms in Connect

Hey everyone, I created KIP-1083 to increase the default value of task.shutdown.graceful.timeout.ms config in Connect. Please review. Thanks! Sagar.

[jira] [Resolved] (KAFKA-16714) kafka-share-groups.sh supporting list and describe

[ https://issues.apache.org/jira/browse/KAFKA-16714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16714. -- Resolution: Fixed > kafka-share-groups.sh supporting list and describe > -

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

Hi all, > I saw we added some new configs/metrics. I have removed the recent changes to the public interfaces to limit the scope of the KIP to minimum. PTAL. Thanks, Kamal On Wed, Aug 14, 2024 at 9:58 AM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi Luke, > > > LC5 > Agree

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

Hi Kamal, Thanks for the update. LGTM. Luke On Wed, Aug 14, 2024 at 7:25 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi all, > > > I saw we added some new configs/metrics. > > I have removed the recent changes to the public interfaces to limit the > scope of the KIP to mi

[jira] [Created] (KAFKA-17340) Kafka Java sdk ConsumerConfig describe doc is incorrectly

shilin Lu created KAFKA-17340: - Summary: Kafka Java sdk ConsumerConfig describe doc is incorrectly Key: KAFKA-17340 URL: https://issues.apache.org/jira/browse/KAFKA-17340 Project: Kafka Issue Ty

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

> I don't want to be a hater of idempotent HB, but having a "RPC" used to generate UUID is unnecessary to me. I actually agree with you. I am just trying to argue for it. > I'm not sure whether it is worth requiring the UUID format for member id. In the protocol, we declare the field "memberId" a

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

Hi Andrew, Personally, I don't like the lobby approach. It makes things more complicated and it would require changing the records on the server too. This is why I initially suggested the rejected alternative #2 which is pretty close but also not perfect. I'd like to clarify one thing. The Consum

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

> I think that this should be defined. Otherwise, folks may start using incorrect things. In the java client, we will generate a Kafka UUID. The javadoc of the Uuid class has a good definition for it. That makes sense to me. However, there is no binding agreement when we only put the recommendatio

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

Hi TengYao, Thanks for the KIP. Continuing on the point which Andrew mentioned as AS1. As the memberId is now known to the client, and client might send the leave group heartbeat on shutdown prior to receiving the initial heartbeat response. If that's true then how do we guarantee that the 2 reque

[jira] [Resolved] (KAFKA-14510) Extend DescribeConfigs API to support group configs

[ https://issues.apache.org/jira/browse/KAFKA-14510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-14510. - Fix Version/s: 4.0.0 Resolution: Fixed > Extend DescribeConfigs API to support group conf

[jira] [Resolved] (KAFKA-16576) New consumer fails with assert in consumer_test.py’s test_consumer_failure system test

[ https://issues.apache.org/jira/browse/KAFKA-16576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16576. - Assignee: Dongnuo Lyu Resolution: Fixed Fixed by https://github.com/apache/kafka/pull/1684

[jira] [Resolved] (KAFKA-17219) Adjust system test framework for new protocol consumer

[ https://issues.apache.org/jira/browse/KAFKA-17219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-17219. - Assignee: Dongnuo Lyu Resolution: Fixed Fixed by https://github.com/apache/kafka/pull/1684

[jira] [Resolved] (KAFKA-17295) New consumer fails with assert in consumer_test.py’s test_fencing_static_consumer system test

[ https://issues.apache.org/jira/browse/KAFKA-17295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-17295. - Assignee: Dongnuo Lyu Resolution: Fixed Fixed by https://github.com/apache/kafka/pull/1684

[jira] [Created] (KAFKA-17341) Refactor consumer heartbeat managers

Andrew Schofield created KAFKA-17341: Summary: Refactor consumer heartbeat managers Key: KAFKA-17341 URL: https://issues.apache.org/jira/browse/KAFKA-17341 Project: Kafka Issue Type: Impr

[jira] [Created] (KAFKA-17342) Move common coordinator code from group-coordinator into separate module coordinator-common

Sushant Mahajan created KAFKA-17342: --- Summary: Move common coordinator code from group-coordinator into separate module coordinator-common Key: KAFKA-17342 URL: https://issues.apache.org/jira/browse/KAFKA-17342

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

hi Apoorv > As the memberId is now known to the client, and client might send the leave group heartbeat on shutdown prior to receiving the initial heartbeat response. If that's true then how do we guarantee that the 2 requests to join and leave will be processed in order, which could still leave s

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

Hi David, I really appreciate your review and suggestions. As I am still gaining experience in writing KIPs, your input has been incredibly helpful. I am currently applying your suggestions to the KIP and will complete it as soon as possible. Regarding the UUID part, I think we haven’t reached a c

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

Hello Apoorv, Thank you for your feedback. Regarding the questions you raised, unfortunately, this KIP cannot guarantee the order of heartbeats. As with many classic distributed system challenges, what we can do is make our best effort to ensure that there are no idle members or stale assignments u

[jira] [Created] (KAFKA-17343) Fix failed benchmark_test.py

Chia-Ping Tsai created KAFKA-17343: -- Summary: Fix failed benchmark_test.py Key: KAFKA-17343 URL: https://issues.apache.org/jira/browse/KAFKA-17343 Project: Kafka Issue Type: Bug

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #3200

See

[jira] [Reopened] (KAFKA-17208) replica_scale_test.py fails for new consumer

[ https://issues.apache.org/jira/browse/KAFKA-17208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True reopened KAFKA-17208: --- > replica_scale_test.py fails for new consumer > > >

[VOTE] KIP-1052: Enable warmup in producer performance test

Hi all,   It seems discussion has been quiet for a couple of weeks so I'd like to call a vote on KIP-1052 https://cwiki.apache.org/confluence/display/KAFKA/KIP-1052%3A+Enable+warmup+in+producer+performance+test   Thanks, Matt Welch

Contributing to Kafka

Hi, My Jira id is maxfortun, I do not have a wiki id. I submitted a request(https://issues.apache.org/jira/browse/INFRA-25451) for it last week, but received no response. The company I work for, Dow Jones, is trying to adopt Kafka, but one of the features we need is missing. I am trying to submit

Re: Contributing to Kafka

> I submitted a request(https://issues.apache.org/jira/browse/INFRA-25451) for it last week, but received no response. that is a known issue :( Maybe you can file a jira first and then create a google doc (or other online doc) to describe your idea. Best, Chia-Ping

[jira] [Created] (KAFKA-17344) Add Unit tests for empty follower fetch scenarios

Abhijeet Kumar created KAFKA-17344: -- Summary: Add Unit tests for empty follower fetch scenarios Key: KAFKA-17344 URL: https://issues.apache.org/jira/browse/KAFKA-17344 Project: Kafka Issue T

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.9 #33

See