Jenkins build is back to normal : kafka-trunk-jdk11 #1624

See

NotEnoughReplicasException: The size of the current ISR Set(2) is insufficient to satisfy the min.isr requirement of 3

I had the following setup Brokers : 3 - all are up and running with min.insync.replicas=3. I created a topic with the following configuration bin\windows\kafka-topics --zookeeper 127.0.0.1:2181 --topic topic-ack-all --create --partitions 4 --replication-factor 3 I triggered the producer with "ac

[jira] [Created] (KAFKA-10243) ConcurrentModificationException while processing connection setup timeouts

Rajini Sivaram created KAFKA-10243: -- Summary: ConcurrentModificationException while processing connection setup timeouts Key: KAFKA-10243 URL: https://issues.apache.org/jira/browse/KAFKA-10243 Projec

Build failed in Jenkins: kafka-2.5-jdk8 #163

See Changes: [boyang] KAFKA-10239: Make GroupInstanceId ignorable in DescribeGroups (#8989) -- [...truncated 5.92 MB...] org.apache.kafka.streams.TopologyTestDriverTest > s

[VOTE] KIP-632: Add DirectoryConfigProvider

Hi, I'd like to start a vote on KIP-632, which is about making the config provider mechanism more ergonomic on Kubernetes: https://cwiki.apache.org/confluence/display/KAFKA/KIP-632%3A+Add+DirectoryConfigProvider Please take a look if you have time. Many thanks, Tom

Re: Feedback: Print schemaId using bin/kafka-dump-log.sh

Hi Mohanraj While I see the usefulness of your suggestion, the main issue is that you're using the Confluent schema registry's conventions and hardwiring them into Kafka core. Given that Confluent's standards are not part of Kafka's official standards, I do not think you will get approval to submi

[DISCUSS] KIP-638: Deprecate DescribeLogDirsResponse.[LogDirInfo, ReplicaInfo]

Hi devs, I hope to start the discussion of KIP-638, which aims to fix a glitch in Admin#describeLogDirs method. - KIP: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866169 - Jira: https://issues.apache.org/jira/browse/KAFKA-8794 All kinds of feedback will be greatly appreci

Re: Untrimmed Index Files resulting in data loss

Hi John, Thanks for reporting the issue. Let's continue the discussion in the PR. Ismael On Mon, Jul 6, 2020 at 7:13 PM John Malizia wrote: > Hi there, about a week ago I submitted an issue report and an associated PR > > https://issues.apache.org/jira/browse/KAFKA-10207 > https://github.com/a

Re: [DISCUSS] KIP-638: Deprecate DescribeLogDirsResponse.[LogDirInfo, ReplicaInfo]

Hi Dongjin, It looks like this KIP is addressing the same issue as KIP-621: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158862109 On Tue, Jul 7, 2020 at 2:29 PM Dongjin Lee wrote: > > Hi devs, > > I hope to start the discussion of KIP-638, which aims to fix a glitch in > Adm

Re: [DISCUSS] Include min.insync.replicas in MetadataResponse to make Producer smarter in partitioning events

Hi Arvin, Thanks for the KIP. Unfortunately, I don't think this makes sense since it would increase the amount of data we send back in the metadata response, which is pretty bad for scalability. In general we probably want to avoid the case where we don't have the appropriate number of in-syn

Re: [VOTE] KIP-621: Deprecate and replace DescribeLogDirsResult.all() and .values()

Thanks, Tom. I tried to think of a better way to do this, but I think you're right that we probably just need different methods. +1 (binding). best, Colin On Mon, Jul 6, 2020, at 01:14, Tom Bentley wrote: > Hi, > > I'd like to start a vote on KIP-621 which is about deprecating methods in > De

Re: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter

Hi all After resurrecting the discussion thread [1] for KIP-431 and have not received any further feedback for 2 weeks, I would like to resurrect the voting thread [2] for KIP-431. I have updated KIP-431 wiki page (https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+a

Re: [DISCUSS] KIP-363

Hi Tom, Thanks for this. I think the tough part is probably the few messages that are still using manual serialization, which can't be easily converted to using this. So we will probably have to upgrade them to using automatic generation, or accept a little inconsistency for a while until the

[jira] [Created] (KAFKA-10244) An new java interface to replace 'kafka.common.MessageReader'

Chia-Ping Tsai created KAFKA-10244: -- Summary: An new java interface to replace 'kafka.common.MessageReader' Key: KAFKA-10244 URL: https://issues.apache.org/jira/browse/KAFKA-10244 Project: Kafka

[jira] [Resolved] (KAFKA-10243) ConcurrentModificationException while processing connection setup timeouts

[ https://issues.apache.org/jira/browse/KAFKA-10243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram resolved KAFKA-10243. Reviewer: Rajini Sivaram Resolution: Fixed > ConcurrentModificationException while p

[DISCUSS] KIP-641 An new java interface to replace 'kafka.common.MessageReader'

hi all, I would like to start the discussion for KIP-641. https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866569 Many thanks, Chia-Ping

Re: [VOTE] KIP-620 Deprecate ConsumerConfig#addDeserializerToConfig(Properties, Deserializer, Deserializer) and ProducerConfig#addSerializerToConfig(Properties, Serializer, Serializer)

> do we just suggest they no longer have any production use case? yep > KafkaProducer internal only. Do we also want to deprecate this public API as > well? We have to make sure users' code can keep working beyond recompilation when migrating to "next" release. Hence, deprecation cycle is nece

Re: [VOTE] KIP-620 Deprecate ConsumerConfig#addDeserializerToConfig(Properties, Deserializer, Deserializer) and ProducerConfig#addSerializerToConfig(Properties, Serializer, Serializer)

Ok, after a second thought, keeping a function which still has production reference is ok. We probably should not make it public in the first place, but this is not high priority either. On Tue, Jul 7, 2020 at 9:03 AM Chia-Ping Tsai wrote: > > do we just suggest they no longer have any productio

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

Hi, Satish, Harsha, Any new updates on the KIP? This feature is one of the most important and most requested features in Apache Kafka right now. It would be helpful if we can make sustained progress on this. Could you share how far along is the design/implementation right now? Is there anything th

Re: [VOTE] KIP-632: Add DirectoryConfigProvider

+1 (non-binding). Thanks for the KIP! On Tue, Jul 7, 2020 at 12:54 PM Tom Bentley wrote: > Hi, > > I'd like to start a vote on KIP-632, which is about making the config > provider mechanism more ergonomic on Kubernetes: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-632%3A+Add+Direc

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

Hi Jun, Satish and I have added more design details in the KIP, including how to keep consistency between replicas (especially when there is leadership changes / log truncations) and new metrics. We also made some other minor changes in the doc. We will finish the KIP changes in the next couple of

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

Hi, Ying, Thanks for the update. It's good to see the progress on this. Please let us know when you are done updating the KIP wiki. Jun On Tue, Jul 7, 2020 at 10:13 AM Ying Zheng wrote: > Hi Jun, > > Satish and I have added more design details in the KIP, including how to > keep consistency be

Re: [DISCUSSION] KIP-619: Add internal topic creation support

Hi Colin, Thanks for the comments. I’ve modified the KIP accordingly. > I think we need to understand which of these limitations we will carry > forward and which we will not. We also have the option of putting > limitations just on consumer offsets, but not on other internal topics. In the

[GitHub] [kafka-site] mjsax merged pull request #272: KAFKA-10222:Incorrect methods show up in 0.10 Kafka Streams docs

mjsax merged pull request #272: URL: https://github.com/apache/kafka-site/pull/272 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [kafka-site] mjsax commented on pull request #272: KAFKA-10222:Incorrect methods show up in 0.10 Kafka Streams docs

mjsax commented on pull request #272: URL: https://github.com/apache/kafka-site/pull/272#issuecomment-655021969 We can still merge this PR, because otherwise the web-page would not be updated. This is an automated message fr

[GitHub] [kafka-site] mjsax commented on pull request #271: MINOR: Fix table of contents in protocol page

mjsax commented on pull request #271: URL: https://github.com/apache/kafka-site/pull/271#issuecomment-655024302 Thanks @wkodate This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [kafka-site] mjsax merged pull request #271: MINOR: Fix table of contents in protocol page

mjsax merged pull request #271: URL: https://github.com/apache/kafka-site/pull/271 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Build failed in Jenkins: kafka-trunk-jdk8 #4699

See Changes: [github] KAFKA-10243; ConcurrentModificationException while processing connection -- [...truncated 3.16 MB...] org.apache.kafka.streams.internals.WindowSto

[jira] [Created] (KAFKA-10245) Using vulnerable log4j version

Pavel Kuznetsov created KAFKA-10245: --- Summary: Using vulnerable log4j version Key: KAFKA-10245 URL: https://issues.apache.org/jira/browse/KAFKA-10245 Project: Kafka Issue Type: Bug

[jira] [Resolved] (KAFKA-10191) fix flaky StreamsOptimizedTest

[ https://issues.apache.org/jira/browse/KAFKA-10191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sophie Blee-Goldman resolved KAFKA-10191. - Resolution: Fixed > fix flaky StreamsOptimizedTest > ---

[jira] [Resolved] (KAFKA-10222) Incorrect methods show up in 0.10 Kafka Streams docs

[ https://issues.apache.org/jira/browse/KAFKA-10222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-10222. - Resolution: Fixed > Incorrect methods show up in 0.10 Kafka Streams docs > -

[VOTE] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext

Hi everyone, I'd like to start a vote for adding two new time API's to ProcessorContext. Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext Thanks

Re: [DISCUSS] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext

Hey John, since ProcessorContext is a public API, I couldn't be sure that people won't try to extend it. Without a default implementation, user code compilation will break. William and Piotr, it seems that we haven't added any example usage of the new API, could we try to address that? It should

Jenkins build is back to normal : kafka-trunk-jdk14 #274

See

Re: [DISCUSS] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext

Sure, I would appreciate help from Piotr creating an example. On Tue, Jul 7, 2020 at 12:03 PM Boyang Chen wrote: > Hey John, > > since ProcessorContext is a public API, I couldn't be sure that people > won't try to extend it. Without a default implementation, user code > compilation will break.

Jenkins build is back to normal : kafka-trunk-jdk8 #4700

See

[DISCUSS] KIP-631: The Quorum-based Kafka Controller

Hi all, I posted a KIP about how the quorum-based controller envisioned in KIP-500 will work. Please take a look here: https://cwiki.apache.org/confluence/x/4RV4CQ best, Colin

Build failed in Jenkins: kafka-trunk-jdk14 #275

See Changes: [github] KAFKA-10191 fix flaky StreamsOptimizedTest (#8913) -- [...truncated 6.38 MB...] org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcesso

Build failed in Jenkins: kafka-trunk-jdk11 #1626

See Changes: [github] KAFKA-10191 fix flaky StreamsOptimizedTest (#8913) -- [...truncated 6.38 MB...] org.apache.kafka.streams.test.OutputVerifierTest > shouldFailIfKey

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

HI Colin. Thanks for the KIP. Here is some feedback and various questions. "*Controller processes will listen on a separate port from brokers. This will be true even when the broker and controller are co-located in the same JVM*". I assume it is possible that the port numbers could be the same

Re: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter

Hi Badai, Thanks for picking this up. I've reviewed the KIP document and the threads you linked. I think we may want to make more improvements in the future to the printing of headers in particular, but this KIP seems like a clear benefit already. I think we can take it incrementally. I'm +1 (bi

[jira] [Created] (KAFKA-10246) AbstractProcessorContext topic() throws NullPointerException when modifying a state store within the DSL from a punctuator

Peter Pringle created KAFKA-10246: - Summary: AbstractProcessorContext topic() throws NullPointerException when modifying a state store within the DSL from a punctuator Key: KAFKA-10246 URL: https://issues.apache.

[jira] [Resolved] (KAFKA-10221) Backport fix for KAFKA-9603 to 2.5

[ https://issues.apache.org/jira/browse/KAFKA-10221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler resolved KAFKA-10221. -- Resolution: Fixed > Backport fix for KAFKA-9603 to 2.5 > ---

Re: [DISCUSS] KIP-637 Include min.insync.replicas in MetadataResponse to make Producer smarter in partitioning events

Thanks for the comment. Yes, it's indeed a downside that this will increase the size of the metadata response, I was thinking if it's worth providing this information to the Producer conditionally, e.g. add a config to Producer to allow people choose whether to include this information in the Metad

Re: [DISCUSS] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext

I think, we don't need a default implementation for the new methods. What would be the use-case to implement the `ProcessorContext` interface? In contract to, for example, `KeyValueStore`, `ProcessorContext` is a use-only interface because it's never passed into Kafka Streams, but only handed out

Re: [VOTE] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext

Thanks for the KIP. +1 (binding) -Matthias On 7/7/20 11:48 AM, William Bottrell wrote: > Hi everyone, > > I'd like to start a vote for adding two new time API's to ProcessorContext. > > Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext >

Re: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter

+1 (binding) -Matthias On 7/7/20 7:16 PM, John Roesler wrote: > Hi Badai, > > Thanks for picking this up. I've reviewed the KIP document and > the threads you linked. I think we may want to make more > improvements in the future to the printing of headers in particular, > but this KIP seems lik