Re: [VOTE] 3.8.0 RC0

Hi all, I'm cancelling the VOTE thread for 3.8.0-RC0. I submitted a PR with the backport https://github.com/apache/kafka/pull/16593 and I'll generate a new RC as soon as it's merged. Best, On Sat, Jul 13, 2024 at 7:09 PM Josep Prat wrote: > Thanks for reviewing the RC Jakub, > > If you can ope

[jira] [Created] (KAFKA-17137) Ensure Admin APIs are properly tested

Mickael Maison created KAFKA-17137: -- Summary: Ensure Admin APIs are properly tested Key: KAFKA-17137 URL: https://issues.apache.org/jira/browse/KAFKA-17137 Project: Kafka Issue Type: Improve

Re: [VOTE] 3.8.0 RC0

Hi, I'm concerned we did not have tests to catch that issue earlier. Such an essential API like describeTopics() should be properly tested. Taking a quick look, it seems a bunch of other Admin APIs also don't have integration tests. I created https://issues.apache.org/jira/browse/KAFKA-17137 to ad

Re: [VOTE] 3.8.0 RC0

Thanks Mickael! +1 for increasing the test coverage for admin clients. But I don't think this should be the blocker for v3.8.0, given the delay of v3.8.0 and we already have many releases with this state. What do you think? Thanks. Luke On Mon, Jul 15, 2024 at 4:57 PM Mickael Maison wrote: > Hi

[jira] [Resolved] (KAFKA-16661) add a lower `log.initial.task.delay.ms` value to integration test framework

[ https://issues.apache.org/jira/browse/KAFKA-16661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Chen resolved KAFKA-16661. --- Fix Version/s: 3.9.0 Resolution: Fixed > add a lower `log.initial.task.delay.ms` value to int

Re: [VOTE] 3.8.0 RC0

Yeah I've not marked it as a blocker for 3.8.0. It's just something we need to do in the background. On Mon, Jul 15, 2024 at 11:05 AM Luke Chen wrote: > > Thanks Mickael! > +1 for increasing the test coverage for admin clients. > But I don't think this should be the blocker for v3.8.0, given the

Re: [DISCUSS] KIP-1043: Administration of groups

Hi Lianet, Thanks for your comments. LM1. Admin.listGroups() in principle needs to be able to return results from any version of the ListGroups RPC. The older versions do not contain the group type, so I think it’s reasonable to have Optional. I think there’s a difference between Optional.empty (I

Re: [VOTE] 3.8.0 RC0

Sounds good! Thank you. Luke On Mon, Jul 15, 2024 at 5:11 PM Mickael Maison wrote: > Yeah I've not marked it as a blocker for 3.8.0. It's just something we > need to do in the background. > > On Mon, Jul 15, 2024 at 11:05 AM Luke Chen wrote: > > > > Thanks Mickael! > > +1 for increasing the te

[jira] [Resolved] (KAFKA-17097) Add replace.null.with.default configuration to ValueToKey and ReplaceField (KIP-1040)

[ https://issues.apache.org/jira/browse/KAFKA-17097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-17097. Fix Version/s: 3.9.0 Resolution: Fixed > Add replace.null.with.default configuratio

[jira] [Created] (KAFKA-17138) tiered storage with apache iceberg

yongzhi.shao created KAFKA-17138: Summary: tiered storage with apache iceberg Key: KAFKA-17138 URL: https://issues.apache.org/jira/browse/KAFKA-17138 Project: Kafka Issue Type: Wish

[jira] [Created] (KAFKA-17139) MirrorSourceTask will stop mirroring when get BufferUnderflowException

Yu Wang created KAFKA-17139: --- Summary: MirrorSourceTask will stop mirroring when get BufferUnderflowException Key: KAFKA-17139 URL: https://issues.apache.org/jira/browse/KAFKA-17139 Project: Kafka

Re: [VOTE] KIP-1067: Remove ReplicaVerificationTool in 4.0 (deprecate in 3.9)

Thank you very much. This KIP is now passed by +3 bindings (Chia-Ping Tsai, Justine Olshan, Luke Chen) and +1 non-binding (Federico Valeri). Greatly appreciate all who participated. Thanks, Dongjin. On Fri, Jul 12, 2024 at 9:20 AM Luke Chen wrote: > +1 (binding) from me. > > Thanks Dongjin. > L

Re: [DISCUSS] KIP-1068: KIP-1068: New JMX Metrics for AsyncKafkaConsumer

Hi Brenden, Thanks for the updates. AS4. I see that you’ve added `.ms` to a bunch of the metrics reflecting the fact that they’re measured in milliseconds. However, I observe that most metrics in Kafka that are measured in milliseconds, with some exceptions in Kafka Connect and MirrorMaker do not

[VOTE] 3.8.0 RC1

Hello Kafka users, developers and client-developers, This is the second release candidate for Apache Kafka 3.8.0. Some of the major features included in this release are: * KIP-1028: Docker Official Image for Apache Kafka * KIP-974: Docker Image for GraalVM based Native Kafka Broker * KIP-1036: E

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

Hey Greg, thanks for the feedback. I can understand your concern about atomicity, but what does a user do after a transaction is failed due to a `too-large-record `exception? They will submit the same batch without the problematic record again. What we are providing is actually a better/more conv

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

Hi, Christo, Thanks for the KIP and sorry for the late reply. Since this KIP changes the format of TopicRecord, it would be useful to add a section on upgrade based on bumped metadata.version. Jun On Wed, May 15, 2024 at 2:47 AM Luke Chen wrote: > Hi Christo, > > Thanks for the explanation. >

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #3108

See

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

Hi Greg, What you say makes a lot of sense. I just wanted to clarify a couple of subtle points. AL1. There is a functional reason to handle errors that happen on send (oginate in the producer logic in the client) vs. errors that are returned from the broker. The problem is that RecordTooLargeEx

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

Hi Alieh, Thanks for your response. > what does a user do > after a transaction is failed due to a `too-large-record `exception? They > will submit the same batch without the problematic record again. If they re-submit the same record, they are indicating that this record is an integral part of

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

Hi Artem, Thank you for clarifying as I'm joining the conversation late and may have some misconceptions. > Because of this, a more "complete" solution that > allows ignoring RecordTooLargeException regardless of its origin is > actually incorrect, while a "partial" solution that allows ignoring

[jira] [Resolved] (KAFKA-17102) FetchRequest#forgottenTopics would return incorrect data

[ https://issues.apache.org/jira/browse/KAFKA-17102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-17102. Fix Version/s: 3.9.0 Resolution: Fixed > FetchRequest#forgottenTopics would return

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

Hi Greg, > This makes me think that this IGNORE_SEND_ERRORS covers an arbitrary set of error conditions that may be expanded in the future, possibly to cover the broker side RecordTooLargeException. I don't think it contradicts what I said (the keyword here is "in the future") -- with the current

[jira] [Created] (KAFKA-17140) 'make tools' should check the version of tools

Chia-Ping Tsai created KAFKA-17140: -- Summary: 'make tools' should check the version of tools Key: KAFKA-17140 URL: https://issues.apache.org/jira/browse/KAFKA-17140 Project: Kafka Issue Type

Re: [VOTE] KIP-1059: Enable the Producer flush() method to clear the latest send() error

+0 Alieh, Matthias, Andrew--I know this isn't what you were hoping for, and I want to acknowledge the significant time and effort you've put into this KIP. I do believe it solves a real problem for Kafka Streams, but I don't believe the solution as presented is worth the fine print and potential f

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

I agree with Alieh and Artem -- in the end, why buffer records twice? We effectively want to allow to push some error handling (which I btw consider "business logic") into the producer. IMHO, there is nothing wrong with it. Dropping a poison pill record is no really a violation of atomicity fro

Re: [VOTE] KIP-1059: Enable the Producer flush() method to clear the latest send() error

Thanks Chris. That's fair. -Matthias On 7/15/24 4:03 PM, Chris Egerton wrote: +0 Alieh, Matthias, Andrew--I know this isn't what you were hoping for, and I want to acknowledge the significant time and effort you've put into this KIP. I do believe it solves a real problem for Kafka Streams, but

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

Matthias, Thank you for rejecting my suggested alternatives. Your responses are the sorts of things I expected to see summarized in the text of the KIP. I agree with most of your rejections, except this one: > "Estimation" is not sufficient, but we would need to know it exactly. > And that's an

[jira] [Resolved] (KAFKA-14401) Connector/Tasks reading offsets can get stuck if underneath WorkThread dies

[ https://issues.apache.org/jira/browse/KAFKA-14401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-14401. --- Fix Version/s: 3.9.0 Resolution: Fixed > Connector/Tasks reading offsets can get stuc

[jira] [Created] (KAFKA-17141) "DescribeTopicPartitions API is not supported" warning message confuses users

Luke Chen created KAFKA-17141: - Summary: "DescribeTopicPartitions API is not supported" warning message confuses users Key: KAFKA-17141 URL: https://issues.apache.org/jira/browse/KAFKA-17141 Project: Kafk

Re: [VOTE] KIP-1070: Deprecate MockProcessorContext

Makes sense to me -- seems like an oversight since we did correctly deprecate the old Processor, ProcessorSupplier, etc (not to mention the #transform, #transformValues methods). Still a +1 (binding) from me On Fri, Jul 12, 2024 at 4:41 PM Matthias J. Sax wrote: > I just realized, that there is

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #3110

See