Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

cessary?) bridge between (1) and > (2): (3) is semantically sound (we ignore errors via passing a flag into > commitTx() instead of flush()), and at the same time safe (we force users to > explicitly flush() and [hopefully] do proper error handling, and don't rely > in am implicit flu

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

ion to ignore or not during an implicit flush... >>>>>>> >>>>>>> >>>>>>> >>>>>>>> We shouldn't be relying on errors in the callback unless we are >>>>>>>> calling flush, which we can still do. It s

Re: [VOTE] KIP-1059: Enable the Producer flush() method to clear the latest send() error

Hi Alieh, Thanks for the KIP. I think we’ve settled on a good solution. +1 (non-binding) Thanks, Andrew > On 25 Jun 2024, at 13:17, Alieh Saeedi wrote: > > Hi all, > > I would like to open voting for KIP-1059: Enable the Producer flush() > method to clear the latest send() error >

Re: [DISCUSS] KIP-1062: Introduce Pagination for some requests used by Admin API

Hi Omnia, Thanks for the KIP. This is a really nice improvement for administering large clusters. AS1: Besides topics, the most numerous resources in Kafka clusters in my experience are consumer groups. Would it be possible to extend the KIP to cover ListGroups while you’re in here? I’ve heard

Re: [DISCUSS] KIP-1062: Introduce Pagination for some requests used by Admin API

d, like DescribeTopicPartitionsResponse. > > Updates the KIP to reflect this now. > > Let me know if you have any more feedback on this. > > Best > Omnia > >> On 27 Jun 2024, at 17:53, Andrew Schofield wrote: >> >> Hi Omnia, >> Thanks for the KIP.

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

Hi Chris, As it stands, the error handling for transactions in KafkaProducer is not ideal. There’s no reason why a failed operation should fail a transaction provided that the application can tell that the operation was not included in the transaction and then make its own decision whether to conti

Re: [DISCUSS] KIP-1068: KIP-1068: New JMX Metrics for AsyncKafkaConsumer

Hi Brenden, Thanks for the KIP. It fills a gap in the metrics for the new consumer nicely. AS1. If using the CLASSIC protocol, and thus the LegacyKafkaConsumer, I would expect that the metrics do not exist at all. Maybe say something like “These metrics are for the new consumer implementation usin

Re: [DISCUSS] KIP-1043: Administration of groups

ocolType field of RPC response (shown in the cmd line output as > PROTOCOL). We would have the filters aligned with the output for all cases, > which seems more consistent. > > Thanks! > > Lianet > > > > On Thu, Jun 6, 2024 at 8:16 AM Andrew Schofield > wrote: &g

Re: [DISCUSS] KIP-1068: KIP-1068: New JMX Metrics for AsyncKafkaConsumer

be set to null or 0." → which >>>> one will be set to null and which ones will be set to 0, and why? >>>> >>>> nit: "The current number of unsent requests in the consumer network" → >>>> Seems to be missing something? >>>> >

Re: [DISCUSS] KIP-1043: Administration of groups

s expecting a "successfully created" kind of output) > > $ bin/kafka-share-groups.sh --bootstrap-server localhost:9092 > --create --group NewShareGroup > Share group 'NewShareGroup' exists. > > Thanks! > > Lianet > > On Mon, Jul 15,

Re: [DISCUSS] KIP-1071: Streams Rebalance Protocol

Hi Lucas and Bruno, Thanks for the great KIP. I've read through the document and have some initial comments. AS1: I suppose that there is a new o.a.k.common.GroupType.STREAMS enumeration constant. This is a change to the public interface and should be called out. AS2: Since streams groups are n

Re: [DISCUSS] KIP-1043: Administration of groups

;>> Makes sense? Not sure if I’m missing other readability issues with the >>> javadocs you’re seeing. >>> >>> LM7. Looks better to me now with the added filter on the kafka-group.sh >> for >>> the protocol. But then, the new –consumer filter achiev

Re: [DISCUSS] KIP-1071: Streams Rebalance Protocol

; differentiate different schema types with the same content. So the > keys are not versioned, but the version field is "abused" as a type > tag. This is the same in KIP-848, we followed it for consistency. > > Cheers, > Lucas > > > On Thu

Re: [DISCUSS] KIP-1071: Streams Rebalance Protocol

ical as in KIP-848. > > AS10: In the consumer offset topic, the version in the key is used to > differentiate different schema types with the same content. So the > keys are not versioned, but the version field is "abused" as a type > tag. This is the same in KIP-848, we followed i

Re: [DISCUSS] KIP-1071: Streams Rebalance Protocol

story significantly. >> >> AS8: "streams.group.assignor.name" sounds good to me to distinguish >> the config from class names. Not sure if I like the "default". To be >> consistent, we'd then have to call it >> `group.streams.default.session.tim

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

1 hour, I expect 3600MB of additional heap needed per >>>>>>>>>> producer". We can then discuss whether we expect higher or lower >>>>>>>>>> throughput, commit intervals, or heap usage to determine what the >>>>>

Re: [DISCUSS] KIP-1068: KIP-1068: New JMX Metrics for AsyncKafkaConsumer

which >>>>> captures the time spent in the buffer. Do you think we should have a >>>>> similar name for `application-event-queue-age-avg` i.e. change to ` >>>>> application-event-queue-time-avg`? Moreover other than similar naming, >>>>> `time` anyways seems more suitable tha

Re: [VOTE] KIP-1068: New metrics for the new KafkaConsumer

HI Brenden, Thanks for the KIP. +1 (non-binding) Thanks, Andrew > On 25 Jul 2024, at 23:26, Apoorv Mittal wrote: > > Hi Brenden, > The metrics look good. Thanks for the KIP. > > +1 (Non-Binding) > > Regards, > Apoorv Mittal > > > On Thu, Jul 25, 2024 at 5:04 PM Philip Nee > wrote: > >> Hi Bren

Re: [DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

Hi Rich, Thanks for resurrecting this KIP. It seems like a useful idea to me and I’d be interested in seeing the proposed public interfaces. I note that you specifically called out the ProducerInterceptor.onAcknowledgement method, as opposed to the producer Callback.onCompletion method. Thanks,

Re: [DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

available%20in%20onAcknowledgement>, > I added FAQ#2 to explain that. If I have missed any documentation regarding > the design of RecordMetadata, please let me know. > > Regards, > Rich > > > On Fri, Jul 26, 2024 at 4:00 PM Andrew Schofield > wrote: > >> Hi

Re: [DISCUSS] Graduation steps for Features

Hi Josep, I think it’s high time that this was tackled. I suggest that it would be best handled as a KIP because then we have a document which can be discussed and improved, followed by a formal vote. A standard set of terms with agreed meanings would be very helpful for some of the larger KIPs wh

Re: [DISCUSS] Graduation steps for Features

Hi, I would at least like to see agreement on the various stages. I’m used to Early Access, then Preview, and finally General Availability, but I notice that at least one other person on this thread had the order different. Also, General Availability sounds a bit wrong for an open-source project, s

Re: [DISCUSS] KIP-1071: Streams Rebalance Protocol

11:13 AM Lucas Brutschy >> wrote: >>> Hi Andrew, >>> >>> AS2: I added a note for now. If others feel strongly about it, we can >>> still add more administrative tools to the KIP - it should not change >>> the overall story significantly. >>

Re: [DISCUSS] KIP-1043: Administration of groups

ol but also > when validating the list of assignors provided by a consumer in a > JoinGroupRequest. Seems a bit confusing to me already, so maybe better not > to add more to it? Just first thoughts. What do you think? > > Thanks, > Lianet > > On Fri, Jul 19, 2024 at 5:00 

Re: [DISCUSS] KIP-1079: Deprecate `delete-config` of TopicCommand

Hi, Thanks for the KIP. Personally, I think we are being a bit too timid here. The problem is that the `--delete-config` option is already broken and it cannot be the case that anyone is using it successfully. By removing it in 5.0, we are essentially going to have an option whose help message

Re: [DISCUSS] KIP-1079: Deprecate `delete-config` of TopicCommand

Hi, I’m happy to go with Chia-Ping’s position here. I’m just aware that this option is already entirely broken. Either way, it’s a very small point and it can certainly remain until its final removal in AK 5.0. Thanks, Andrew > On 5 Aug 2024, at 14:29, TengYao Chi wrote: > > Hi Andrew, Chia-Pin

Re: [DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

that timestamp may be overwritten (and this is the root >>>> cause of the gap), but that could be resolved just by keeping the >>> original >>>> time and making it available. >>>> RecordMetadata would keep a timestamp (passed from the record creation, >

Re: [DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

public void onAcknowledgement(RecordMetadata metadata, Exception > exception, Headers headers) { > // implement my logic here >} > } > ``` > > Regards, > Rich > > > On Fri, Aug 9, 2024 at 4:16 AM Andrew Schofield > wrote: > >> Hi Rich, >> Than

Re: [VOTE] KIP-512: make Record Headers available in onAcknowledgement and onComplete

Hi Rich, Thanks for the KIP. Looks good to me. +1 (non-binding) Thanks, Andrew > On 10 Aug 2024, at 18:53, Rich C. wrote: > > Sorry, forgot the link to KIP-512 > https://cwiki.apache.org/confluence/display/KAFKA/KIP-512%3A+make+Record+Headers+available+in+onAcknowledgement+and+onComplete > > Re

Re: [DISCUSS] KIP-1071: Streams Rebalance Protocol

n idea you mentioned, >>> for instance). Actually, I may be missing something about the HB, but if we >>> are at the point where HB requires that the topology has been initialized, >>> and the topology init requires the group, why is it the heartbeat RPC the >>> on

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

Hi TengYao, Thanks for the KIP. I wonder if there’s a different way to close what is quite a small window. AS1: It is true that the initial heartbeat is not idempotent, but this remains true with this KIP. It’s just differently not idempotent. If the client makes its own member ID, sends a reques

Re: [VOTE] KIP-932: Queues for Kafka

accordingly and the revised code has been merged. If anyone has any comments, please let me know. Thanks, Andrew > On 21 May 2024, at 15:35, Andrew Schofield wrote: > > Hi Jun, > All the client metrics are standard. None are required. > > I’ve updated the KIP accordingly. &g

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

tional complexity for > compatibility, as the new server would not immediately add the member to > the group, while the old server would. This requires clients to > differentiate whether their memberId has been added to the group or not, > which could result in unexpected logs. > &

Re: [VOTE] KIP-932: Queues for Kafka

tionMetadataValue: Is NumPartitions redundant given > PartitionMetadata? > > 2. I assume that ShareGroupPartitionMetadataValue contains initialized > partitions. Does that > make ShareGroupStatePartitionMetadataValue.InitializedTopics redundant? > > Jun > > On Tue, Au

Re: [VOTE] KIP-714: Client metrics and observability

configuration resources”. The subscriptions are derived from the configuration resources, but they are not the same thing. Thanks, Andrew > On 16 Oct 2023, at 09:18, Andrew Schofield > wrote: > > The vote for KIP-714 has now concluded and the KIP is APPROVED. > > The votes are: > B

Re: [VOTE] KIP-993: Allow restricting files accessed by File and Directory ConfigProviders

Thanks for the KIP. +1 (non-binding) Thanks, Andrew > On 14 Dec 2023, at 13:41, Mickael Maison wrote: > > +1 (binding) > Thanks for the KIP > > Mickael > > On Wed, Dec 13, 2023 at 4:01 PM Chris Egerton wrote: >> >> Hi Tina, >> >> Thanks for the KIP! +1 (binding) >> >> Cheers, >> >> Chris >> >>

Re: [DISCUSS] Kafka distributed transaction support

Hi Rowland, KIP-939 https://cwiki.apache.org/confluence/display/KAFKA/KIP-939%3A+Support+Participation+in+2PC introduces two-phase commit, but not XA support. I believe the KIP is expected to be delivered in Kafka 4.0 later this year. The way that it works and the use cases that it’s intended fo

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

I also agree with 2 & 2 with reasoning along the same lines as Artem. Thanks, Andrew > On 12 Jan 2024, at 09:15, Federico Valeri wrote: > > On Thu, Jan 11, 2024 at 10:43 PM Artem Livshits > wrote: >> >> Hi Proven, >> >> I'd say that we should do 2 & 2. The idea is that for small features th

Re: [VOTE] KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

+1 (non-binding) Thanks, Andrew > On 22 Jan 2024, at 07:29, Federico Valeri wrote: > > +1 (non binding) > > Thanks. > > On Mon, Jan 22, 2024 at 7:03 AM Luke Chen wrote: >> >> Hi Ziming, >> >> +1(binding) from me. >> >> Thanks. >> Luke >> >> On Mon, Jan 22, 2024 at 11:50 AM Kamal Chandraprakash

Re: [DISCUSS] KIP-932: Queues for Kafka

of a share-partition, it must read > the most recent SHARE_CHECKPOINT": How does a broker find this efficiently > on restart? > > 25. AcknowledgeCommitCallback: How would an application use it? It doesn't > indicate which record's acknowledgement has failed. >

Re: [DISCUSS] KIP-932: Queues for Kafka

t; 28. I noticed that the "*--state*" and "*--timeout*" options are not > mentioned for the kafka-share-groups.sh tool. Was this omission > intentional, or is it possibly an oversight in the KIP? > Thanks, > Chirag > > On Mon, Feb 12, 2024 at 5:25 PM Andrew Scho

Re: [Discuss] KIP-1019: Expose method to determine Metric Measurability

Hi Apoorv, Thanks for the KIP. Looks like a useful change to tidy up the metrics code. Thanks, Andrew > On 14 Feb 2024, at 14:55, Apoorv Mittal wrote: > > Hi, > I would like to start discussion of a small KIP which fills a gap in > determining Kafka Metric measurability. > > KIP-1019: Expose met

Re: [VOTE] 3.7.0 RC4

+1 (non-binding). I used the staged binaries with Scala 2.13. I tried the new group coordinator and consumer group protocol which is included with the Early Access release of KIP-848. Also verified the availability of the new APIs. All working as expected. Thanks, Andrew > On 15 Feb 2024, at 05

Re: [VOTE] KIP-1019: Expose method to determine Metric Measurability

Hi Apoorv, Thanks for the KIP. +1 (non-binding) Thanks, Andrew > On 19 Feb 2024, at 22:31, Apoorv Mittal wrote: > > Hi, > I’d like to start the voting for KIP-1019: Expose method to determine > Metric Measurability. > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1019%3A+Expose+method

Re: [DISCUSS] KIP-932: Queues for Kafka

erGroupsOptions supports filtering by state. Should we > support it here too? > > 31. ConsumerGroupListing includes state. Should we include state in > GroupListing to be consistent? > > Jun > > On Mon, Feb 12, 2024 at 3:55 AM Andrew Schofield < > andrew_schofield_

Re: [DISCUSS] KIP-932: Queues for Kafka

Hi Manikumar, Thanks for your comments. 1. I believe that in general, there are not situations in which a dynamic config change is prevented because of the existence of a resource. So, if we prevented setting config `group.type=consumer` on resource G of GROUP type if there was a share group G in

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

This is an interesting problem. While it would be possible to use a Consumer, it can’t handle exceptions really. A java.util.Stream has a similar problem. I wonder whether an interface which looks like java.util.concurrent.Flow.Subscriber would be good. Something like: public interface AdminResu

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

Hi Justine, Thanks for the KIP. This area of Kafka is complicated and making it easier is good. When I use the `kafka-features.sh` tool to describe the features on my cluster, I find that there’s a single feature called “metadata.version”. I think this KIP does a handful of things: 1) It intro

Re: [DISCUSS] KIP-1021: Allow to get last stable offset (LSO) in kafka-get-offsets.sh

Hi Ahmed, Thanks for the KIP. It looks like a useful addition. The use of negative timestamps, and in particular letting the user use `--time -1` or the equivalent `--time latest` is a bit peculiar in this tool already. The negative timestamps come from org.apache.kafka.common.requests.ListOffs

Re: [DISCUSS] KIP-1021: Allow to get last stable offset (LSO) in kafka-get-offsets.sh

et by time? or do you think having a special > argument for this case is better? > > On Tue, Feb 27, 2024 at 2:19 PM Andrew Schofield < > andrew_schofield_j...@outlook.com> wrote: > >> Hi Ahmed, >> Thanks for the KIP. It looks like a useful addition. >> >> The

Re: [DISCUSS] KIP-939: Support Participation in 2PC

Hi Artem, I totally agree that a timeout for the 2PC case is a bad idea. It does abandon the 2PC guarantee. Thanks, Andrew > On 28 Feb 2024, at 00:44, Artem Livshits > wrote: > > Hi Jun, > > Thank you for the discussion. > >> For 3b, it would be useful to understand the reason why an admin does

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

Hi David, Thanks for the summary. It seems to me that the Flow-like option is best because it can easily handle cancellations and exceptions, returning the topic partition information and signalling when the last of the results have been returned. I think it’s also equally applicable to any of th

Re: [DISCUSS] KIP-1021: Allow to get last stable offset (LSO) in kafka-get-offsets.sh

lag. >>> >>> Thanks, >>> Andrew >>> >>>> On 28 Feb 2024, at 09:22, Ahmed Sobeh >>> wrote: >>>> >>>> Hi Andrew, >>>> >>>> Thanks for the hint! That sounds reasonable, do you think adding a >>

Re: [DISCUSS] KIP-932: Queues for Kafka

re groups, Do we need to clean any in-memory state from > share-partition leaders? > 3. Any metrics for the share-coordinator? > > Thanks > Manikumar > > On Wed, Feb 21, 2024 at 12:11 AM Andrew Schofield < > andrew_schofield_j...@outlook.com> wrote: > >> Hi Mani

[VOTE] KIP-932: Queues for Kafka

Hi, I’ve been working to complete KIP-932 over the past few months and discussions have quietened down. I’d like to open the voting for KIP-932: https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka Thanks, Andrew

Re: [DISCUSS] KIP-932: Queues for Kafka

e group members or share groups? Are they any ways we try to > handle this or mitigate risks with respect to memory usage and client > connections (wrt to rebalances for example) > > Thanks, > > Justine > > On Fri, Mar 8, 2024 at 12:51 AM Andrew Schofield < > andrew_sch

Re: [DISCUSS] KIP-1008: ParKa - the Marriage of Parquet and Kafka

Hi Xinli, Thanks for the KIP. I see that the discussion thread has died down, which is often a tricky situation with a KIP. I’ve been thinking about this KIP for a while and it was really good to be able to attend the Kafka Summit London session to get a proper understanding of it. I think it’s

Re: [DISCUSS] KIP-1021: Allow to get last stable offset (LSO) in kafka-get-offsets.sh

:56, Ahmed Sobeh wrote: > > Hi Andrew, > > Apologies for disappearing, I had to undergo knee surgery, all good now! > > I adjusted the KIP as you suggested, do you think it's ready for the voting > stage? > > On Wed, Mar 6, 2024 at 4:02 PM Andrew Schofield < >

Re: [DISCUSS] KIP-932: Queues for Kafka

also mentioned in the KIP how there are a few inter-broker hops to the > share coordinator, etc for a given read operation of a partition. Are we > concerned about performance here? My work in transactions and trying to > optimize performance made me realize how expensive these inter-

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

Hi David, I agree that we should use the same mechanism to gate KIP-932 once that feature reaches production readiness. The precise details of the values will depend upon the current state of all these flags when that release comes. Thanks, Andrew > On 28 Mar 2024, at 07:11, David Jacot wrote: >

Re: [DISCUSS] KIP-932: Queues for Kafka

uming from large number of topics with large number of partitions what > will be the impact on coordinator memory? > > Thanks > Omnia > > >> On 25 Mar 2024, at 10:23, Andrew Schofield >> wrote: >> >> Hi Justine, >> Thanks for your questions. >

Re: [VOTE] 3.6.2 RC2

Hi, I downloaded the binary release and upgraded a broker from 3.6.0 to 3.6.2. Documentation and protocol links look fine. +1 (non-binding) Thanks, Andrew > On 3 Apr 2024, at 07:17, Manikumar wrote: > > Gentle reminder. Please download, test and vote for the release. > > Thanks, > > On Fri, Ma

Re: [DISCUSS] KIP-932: Queues for Kafka

un > > On Wed, Apr 3, 2024 at 3:32 PM Tom Bentley wrote: > >> Hi Andrew (and Omnia), >> >> Thanks for the KIP. I hope to provide some feedback on this KIP soon, but I >> had a thought on the specific subject of group configs and MM2. If brokers >> validate f

Re: [DISCUSS] KIP-932: Queues for Kafka

roup config > parameter? A similar issue arises wrt topic configs, but this is the first > instance (that I'm aware of) of a config being added during the MM2 era, so > perhaps this is a minor problem worth thinking about. > > Cheers, > > Tom > > On Wed, 3 Apr 2024

Re: [DISCUSS] KIP-932: Queues for Kafka

Hi David, Thanks for your questions. 70. The Group Coordinator communicates with the Share Coordinator over RPCs. In the general case, it’s an inter-broker call. It is probably possible to optimise for the situation in which the appropriate GC and SC shards are co-located, but the KIP does not d

Re: [VOTE] KIP-1022 Formatting and Updating Features

Hi Justine, Thanks for the KIP. +1 (non-binding) Thanks, Andrew > On 8 Apr 2024, at 18:07, Justine Olshan wrote: > > Hello all, > I would like to start a vote for KIP-1022 Formatting and Updating Features >

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

Hi Ivan, Thanks for the KIP. I can see situations in which this would be helpful. I have one question. The KIP says the client checks the cluster ID when it re-bootstraps and that it will fail if the cluster ID doesn’t match the previously known one. How does it fail? Which exception does it th

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

I should remove the mention of this > cluster ID checks from the KIP. > > Best, > Ivan > > [1] > https://github.com/apache/kafka/blob/ff90f78c700c582f9800013faad827c36b45ceb7/clients/src/main/java/org/apache/kafka/clients/Metadata.java#L355 > > On Tue, Apr 9, 2024, at 09

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

I should remove the mention of this > cluster ID checks from the KIP. > > Best, > Ivan > > [1] > https://github.com/apache/kafka/blob/ff90f78c700c582f9800013faad827c36b45ceb7/clients/src/main/java/org/apache/kafka/clients/Metadata.java#L355 > > On Tue, Apr 9, 2024, at 09

Re: [DISCUSS] KIP-932: Queues for Kafka

of just SPSO? > > 109. DeleteShareGroupOffsetsResult exposes > public KafkaFuture partitionResult(final TopicPartition partition) > DeleteShareGroupsResult exposes > public Map> deletedGroups() > Should we make them more consistent? > > 110. Should ShareGroupDescription includ

Re: [DISCUSS] KIP-932: Queues for Kafka

hanged the ambiguous “State” to “DeliveryState” in RPCs and records. And I added a clarification about how the “group.type” configuration should be used. Thanks, Andrew > On 10 Apr 2024, at 15:33, Andrew Schofield > wrote: > > Hi Jun, > Thanks for your questions. > > 41. &

Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams

Hi Damien, Sebastien and Loic, Thanks for the KIP. The DLQ pattern is well established and bringing this to Kafka Streams is a good improvement. I do plan to add DLQ support to share groups in the future, once KIP-932 is complete. Having broad support in Kafka for DLQs is great. I have a few comme

Re: [VOTE] KIP-899: Allow producer and consumer clients to rebootstrap

Thanks for the KIP +1 (non-binding) Andrew > On 15 Apr 2024, at 14:16, Chris Egerton wrote: > > Hi Ivan, > > Thanks for the KIP. After the recent changes, this LGTM. +1 (binding) > > Cheers, > > Chris > > On Wed, Aug 2, 2023 at 12:15 AM Ivan Yurchenko > wrote: > >> Hello, >> >> The discussion

Re: [DISCUSS] KIP-932: Queues for Kafka

wledgement` if the name is already > scope to share groups? > > 019: Should we also scope the name of `record-acknowledgement` to follow > `share-acknowledgement`? > > 020: I suppose that the SPEO is always bounded by the HWM. It may be good > to call it out. Is it also bou

Re: [DISCUSS] KIP-1037: Allow WriteTxnMarkers API with Alter Cluster Permission

Hi Nikhil, I agree with Christo. This is a good improvement and I think your choice of Alter permission on the cluster is the best available. Thanks, Andrew > On 15 Apr 2024, at 12:33, Christo Lolov wrote: > > Heya Nikhil, > > Thank you for raising this KIP! > > Your proposal makes sense to me.

Re: [DISCUSS] KIP-932: Queues for Kafka

Is there a way for a user to > optimize this? > > 116. For each new RPC, could we list the associated acls? > > 117. Since this KIP changes internal records and RPCs, it would be useful > to document the upgrade process. > > Jun > > On Wed, Apr 10, 2024 at 7:35 

Re: [DISCUSS] KIP-1036: Extend RecordDeserializationException exception

Hi, Thanks for the KIP. I think it’s an interesting idea and it seems to work nicely with how the clients work today. Recently, I’ve been digging in to the record deserialization code in the consumer as part of implementing KIP-932. It’s pretty nasty in there. There are essentially two kinds of

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

Hi Alieh, Thanks for the KIP. Exception handling in the Kafka producer and consumer is really not ideal. It’s even harder working out what’s going on with the consumer. I’m a bit nervous about this KIP and I agree with Chris that it could do with additional motivation. This would be an expert-le

Re: [DISCUSS] KIP-932: Queues for Kafka

he client. Ditto for > group.share.heartbeat.interval.ms. > > 133. group.share.max.size: Would group.share.max.members.per.group be a > more intuitive name? > > 134. group.share.assignors: Why does it need to be a list? > > 135. share.coordinator.threads: Is that per share coo

Re: [VOTE] KIP-1037: Allow WriteTxnMarkers API with Alter Cluster Permission

Hi Nikhil, Thanks for the KIP. Looks good to me. +1 (non-binding) Thanks, Andrew > On 22 Apr 2024, at 09:17, Christo Lolov wrote: > > Heya Nikhil, > > Thanks for the proposal, as mentioned before it makes sense to me! > > +1 (binding) > > Best, > Christo > > On Sat, 20 Apr 2024 at 00:25, Justin

Re: [DISCUSS] KIP-1036: Extend RecordDeserializationException exception

Hi Fred, Just reviewing the KIP again now that the discussion has quietened down a little. It will soon be ready for a vote I think. I have a few comments about details. A1) The signature of the new constructor for RecordDeserializationException needs to be updated accord to the discussion. I see

Re: [DISCUSS] KIP-932: Queues for Kafka

taken in milliseconds to load the share-group state from > the share-group state partitions loaded in the last 30 seconds." > The window depends on metrics.num.samples and metrics.sample.window.ms > and is not always 30 seconds, right? > 137.4 Could you explain write/write-latency a bi

Re: [DISCUSS] KIP-932: Queues for Kafka

n > still keep the ShareGroupState as long as the topic still exists. The same > optimization could be applied to DeletingTopics too. > > Jun > > > On Tue, Apr 23, 2024 at 3:57 AM Andrew Schofield > wrote: > >> Hi Jun, >> Thanks for the reply. >>

Re: [DISCUSS] KIP-932: Queues for Kafka

gt; 148. records-per-request-avg: Should we pair it with > records-per-request-max? > > 149. If the shareGroupState is not available, what error code is used in > WriteShareGroupStateResponse? > > Jun > > On Wed, Apr 24, 2024 at 7:12 AM Andrew Schofield > wrote: >

Re: [DISCUSS] KIP-932: Queues for Kafka

of view to not fetch from the share leaders? Or do we > assume that clients will do the right thing in this case? By this, I meant > that they will stop fetching immediately. This part is not clear in the KIP > or I missed it. > > Best, > David > > On Thu, Apr 25, 2024 at 1:19

Re: [DISCUSS] KIP-1081: Graduation Steps for Features

Hi Josep, Thanks for creating this KIP. It looks like a good proposal. A few comments. AS1: I don’t think features should be able to progress between the levels in patch releases. Yes, there may be some bug fixes which mean that the usability of a feature has progressed markedly, but given that Ka

Re: [DISCUSS] KIP-1076: Metrics for client applications a KIP-714 extension

Hi Bill (and Matthias), Thanks for the KIP. This looks like a valuable extension to KIP-714. AS1: Personally, I think that registerMetricsForSubscription should return void and not throw an exception if metrics push is not enabled. Otherwise, you end up with an application whose behaviour is marke

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

e: >>>> >>>>> Hi Andrew, >>>>> >>>>> Personally, I don't like the lobby approach. It makes things more >>>>> complicated and it would require changing the records on the server >>> too. >>>>

Re: [VOTE] KIP-1079: Deprecate `delete-config` of TopicCommand

Thanks for the KIP. +1 (non-binding) Andrew > On 27 Aug 2024, at 10:31, Federico Valeri wrote: > > +1 non binding > > Should we also create a Jira to track future removal? > > On Tue, Aug 27, 2024 at 11:02 AM Chia-Ping Tsai wrote: >> >> +1 (binding) >> >> On 2024/08/27 06:15:57 TengYao Chi wro

Re: [ANNOUNCE] New committer: Lianet Magrans

Awesome news. Well done, Lianet. Andrew > On 28 Aug 2024, at 17:47, Bruno Cadonna wrote: > > Well deserved! > > Best, > Bruno > > On 8/28/24 6:37 PM, Bill Bejeck wrote: >> Congrats Lianet! >> On Wed, Aug 28, 2024 at 12:32 PM Matthias J. Sax wrote: >>> Congrats! Very well deserved! >>> >>> On 8/

Re: [DISCUSS] KIP-1043: Administration of groups

, Andrew From: Andrew Schofield Sent: 02 August 2024 15:00 To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-1043: Administration of groups Hi Lianet, Thanks for your comment. I’ve been digging more into the situation with describing groups in a broker with

Re: [VOTE] KIP-1076: Metrics for client applications a KIP-714 extension

Thanks for the KIP. +1 (non-binding) Andrew From: Lucas Brutschy Sent: 03 September 2024 16:02 To: dev@kafka.apache.org Subject: Re: [VOTE] KIP-1076: Metrics for client applications a KIP-714 extension Hi Bill, thanks for the KIP! +1 (binding) Lucas On Tue,

Re: Implementing missing public method ConsumerRecords.nextOffsets

Hi Lucas, This sounds like an interesting approach and I wonder whether anyone will have the context about why this part of KIP-320 was not completed. There is one little detail that I can think of, which we can address later as the details are confirmed. KafkaShareConsumer also uses ConsumerRecor

Re: [DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

Hi Rich, Thanks for the explanation. It’s true that it breaks use of a lambda as an implementation of Callback and I would say that it’s not acceptable to do this. I don’t think I’ve seen a use of this interface which didn’t use a lambda. I believe the main focus of this KIP was the improvement

Re: [DISCUSS] KIP-1043: Administration of groups

of view, this is ready for a vote. > > Cheers, > Lucas > > > > On Tue, Sep 3, 2024 at 2:56 PM Andrew Schofield > wrote: > > > > Hi, > > I’ve spent some time working with clusters containing groups of multiple > > types, fixing problems and improving err

Re: [DISCUSS] KIP-1043: Administration of groups

in terms of the equivalent sequence of commands. From my point of view, this is ready for a vote. Cheers, Lucas On Tue, Sep 3, 2024 at 2:56 PM Andrew Schofield wrote: > > Hi, > I’ve spent some time working with clusters containing groups of multiple > types, fixing problems a

Re: [DISCUSS] KIP-585: Conditional SMT

hing that complex, it's perhaps too complex for a transformation chain and it's really a streaming application. Andrew Schofield IBM Event Streams On 2020/04/08 21:39:31, Christopher Egerton wrote: > Hi Tom, > > With regards to the potential Transformation::valida

Re: [Discuss] KIP-582 Add a "continue" option for Kafka Connect error handling

ou need to run more connectors, but in a distributed connect cluster, that's easy to achieve. Thanks, Andrew Schofield IBM On 24/04/2020, 22:00, "Zihan Li" wrote: Hi Chris, Thanks a lot for your comments. 1. The complexity comes from maintaining an additional topic

Re: [DISCUSS] KIP-585: Conditional SMT

ibute code. In order to make it simpler to configure, I suggest that the class name configured using "?type" would be assumed to be in package org.apache.kafka.connect.predicates in the absence of a package name so that shorter names such as "TopicMatches" could be used. Cheers,

Re: [DISCUSS] KIP-585: Conditional SMT

nk the Filter SMT would be useful I'm willing to include it in this KIP, since it seems straightforward. If people are happy with this approach, I'll update the KIP. Kind regards, Tom On Sun, Apr 26, 2020 at 3:00 PM Andrew Schofield wrote: > Hi,

<    1   2   3   4   5   6   7   >