Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

Bumping this as I'd like to see it land... It's one of the "features" that tends to catch Kafka n00bs unawares and typically results in message skippage/loss, vs the proposed solution is much more intuitive behavior. Plus it's more wire efficient because consumers no longer need to commit offsets

Re: [VOTE] KIP-232: Detect outdated metadata using leaderEpoch and partitionEpoch

"If consumer wants to consume message with offset 16, then consumer must have already fetched message with offset 15" --> this may not be always true right? What if consumer just call seek(16) after construction and then poll without committed offset ever stored before? Admittedly it is rare but w

Re: [VOTE] KIP-212: Enforce set of legal characters for connector names

Thanks, Sönke. This KIP passes with three binding +1s from Ewen, Gwen, and Jason. I've updated the KIP pages accordingly. On Tue, Jan 23, 2018 at 10:14 PM, Jason Gustafson wrote: > +1 > > On Tue, Jan 23, 2018 at 2:01 PM, Gwen Shapira wrote: > > > +1 (binding) > > > > On Tue, Jan 23, 2018, 1:17

Re: KIP-250 Add Support for Quorum-based Producer Acknowledgment

I agree with Dong, we should see if it's possible to change the default behavior so that as soon as min.insync.replicas brokers respond than the broker acknowledges the message back to the client without waiting for additional brokers who are in the in-sync replica list to respond. (I actually thou

答复: KIP-250 Add Support for Quorum-based Producer Acknowledgment

Currently, with holding the assigned replicas(AR) for all partitions, controller is now able to elect new leaders by selecting the first replica of AR which occurs in both live replica set and ISR. If switching to the LEO-based strategy, controller context might need to be enriched or augmented

Re: [VOTE] KIP-145: Expose Record Headers in Kafka Connect

Thanks for the votes, everyone. This KIP passes with three binding +1 votes and 4 non-binding +1s (including mine). Randall On Tue, Jan 23, 2018 at 3:19 PM, Michael Pearce wrote: > +1 (non-binding) > > I personally think the current proposed Converters described in the KIP > are good, as Randal

Re: [VOTE] KIP-232: Detect outdated metadata using leaderEpoch and partitionEpoch

Hey Guozhang, Thanks much for reviewing the KIP! In the scenario you described, let's assume that broker A has messages with offset up to 10, and broker B has messages with offset up to 20. If consumer wants to consume message with offset 9, it will not receive OffsetOutOfRangeException from brok

Re: KIP-250 Add Support for Quorum-based Producer Acknowledgment

Hey Litao, Thanks for the KIP. I have one quick comment before you provide more detail on how to select the leader with the largest LEO. Do you think it would make sense to change the default behavior of acks=-1, such that broker will acknowledge the message once the message has been replicated t

Re: [VOTE] KIP-227: Introduce Fetch Requests that are Incremental to Increase Partition Scalability

With 3 +1 votes from Ismael, Jun, and Jason, the vote passes. Thanks, Colin On Tue, Jan 23, 2018, at 22:17, Colin McCabe wrote: > On Tue, Jan 23, 2018, at 21:47, Ismael Juma wrote: > > Colin, > > > > You get a cumulative count for rates since we added > > https://cwiki.apache.org/confluence/disp

Re: [VOTE] KIP-227: Introduce Fetch Requests that are Incremental to Increase Partition Scalability

On Tue, Jan 23, 2018, at 21:47, Ismael Juma wrote: > Colin, > > You get a cumulative count for rates since we added > https://cwiki.apache.org/confluence/display/KAFKA/KIP-187+-+Add+cumulative+count+metric+for+all+Kafka+rate+metrics> Oh, good point. C. > Ismael > > On Tue, Jan 23, 2018 at 4:21

KIP-250 Add Support for Quorum-based Producer Acknowledgment

Hey folks. I would like to add a feature to support the quorum-based acknowledgment for the producer request. We have been running a modified version of Kafka on our testing cluster for weeks, the improvement of P999 is significant with very stable latency. Additionally, I have a proposal to achiev

[DISCUSS] KIP-250 Add Support for Quorum-based Producer Acknowledgment

Hey folks. I would like to add a feature to support the quorum-based acknowledgment for the producer request. We have been running a modified version of Kafka on our testing cluster for weeks, the improvement of P999 is significant with very stable latency. Additionally, I have a proposal to achiev

[DISCUSS] KIP-250 Add Support for Quorum-based Producer Acknowledgment

Hey folks. I would like to add a feature to support the quorum-based acknowledgment for the producer request. We have been running a modified version of Kafka on our testing cluster for weeks, the improvement of P999 is significant with very stable latency. Additionally, I have a proposal to achiev

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

Hi Rajini, I think the proposal makes sense. One suggestion: can we just allow the config to be passed? That is, leave out the properties config for now. On Tue, Jan 23, 2018 at 3:01 PM, Rajini Sivaram wrote: > Since we are running out of time to get the whole ConfigCommand converted > to using

Re: [VOTE] KIP-227: Introduce Fetch Requests that are Incremental to Increase Partition Scalability

Colin, You get a cumulative count for rates since we added https://cwiki.apache.org/confluence/display/KAFKA/KIP-187+-+Add+cumulative+count+metric+for+all+Kafka+rate+metrics Ismael On Tue, Jan 23, 2018 at 4:21 PM, Colin McCabe wrote: > On Tue, Jan 23, 2018, at 11:57, Jun Rao wrote: > > Hi, Col

Build failed in Jenkins: kafka-trunk-jdk8 #2347

See Changes: [github] KAFKA-6287; Consumer group command should list simple consumer groups -- [...truncated 3.40 MB...] kafka.utils.CoreUtilsTest > testReadInt PASSED k

Re: [VOTE] KIP-227: Introduce Fetch Requests that are Incremental to Increase Partition Scalability

Thanks for the KIP, +1 (binding). On Tue, Dec 19, 2017 at 7:28 PM, Colin McCabe wrote: > Hi all, > > I'd like to start the vote on KIP-227: Incremental Fetch Requests. > > The KIP is here: https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 227%3A+Introduce+Incremental+FetchRequests+to+Incr

Re: [VOTE] KIP-232: Detect outdated metadata using leaderEpoch and partitionEpoch

Thanks Dong, I made a pass over the wiki and it lgtm. Just a quick question: can we completely eliminate the OffsetOutOfRangeException with this approach? Say if there is consecutive leader changes such that the cached metadata's partition epoch is 1, and the metadata fetch response returns with

Build failed in Jenkins: kafka-trunk-jdk9 #327

See Changes: [github] KAFKA-6287; Consumer group command should list simple consumer groups -- [...truncated 1.87 MB...] org.apache.kafka.streams.tools.StreamsResetterTest

Re: Regarding : Store stream for infinite time

Thanks Matthias. This is really helpful. Regards, Aman On Wed, Jan 24, 2018 at 2:47 AM, Matthias J. Sax wrote: > It's simpler to set retention time to Long.MAX_VALUE instead of log > compaction and generating unique keys. This effectively disable > retention because the value is huge. > > The a

[jira] [Created] (KAFKA-6477) Add Support for Quorum-based Producer Acknowledge

Litao Deng created KAFKA-6477: - Summary: Add Support for Quorum-based Producer Acknowledge Key: KAFKA-6477 URL: https://issues.apache.org/jira/browse/KAFKA-6477 Project: Kafka Issue Type: New Fea

[jira] [Resolved] (KAFKA-6336) when using assign() with kafka consumer the KafkaConsumerGroup command doesnt show those consumers

[ https://issues.apache.org/jira/browse/KAFKA-6336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-6336. Resolution: Fixed > when using assign() with kafka consumer the KafkaConsumerGroup command d

[jira] [Resolved] (KAFKA-6421) consumer group can't show when use kafka-consumer-groups tool if the leader of __consumer_offsets partition for this group changed

[ https://issues.apache.org/jira/browse/KAFKA-6421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-6421. Resolution: Fixed > consumer group can't show when use kafka-consumer-groups tool if the lea

[jira] [Resolved] (KAFKA-6287) Inconsistent protocol type for empty consumer groups

[ https://issues.apache.org/jira/browse/KAFKA-6287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-6287. Resolution: Fixed > Inconsistent protocol type for empty consumer groups > -

Re: [VOTE] KIP-212: Enforce set of legal characters for connector names

+1 On Tue, Jan 23, 2018 at 2:01 PM, Gwen Shapira wrote: > +1 (binding) > > On Tue, Jan 23, 2018, 1:17 PM Randall Hauch wrote: > > > +1 (non-binding) > > > > On Mon, Jan 22, 2018 at 6:35 PM, Sönke Liebau < > > soenke.lie...@opencore.com.invalid> wrote: > > > > > All, > > > > > > this KIP has bee

Build failed in Jenkins: kafka-trunk-jdk8 #2346

See Changes: [wangguoz] KAFKA-6461 TableTableJoinIntegrationTest is unstable if caching is -- [...truncated 1.85 MB...] org.apache.kafka.streams.kstream.internals.KTableA

Jenkins build is back to normal : kafka-trunk-jdk7 #3112

See

Jenkins build is back to normal : kafka-1.0-jdk7 #130

See

Streams - retry configuration publishing changelog updates?

I'm seeing this timeout in my Kafka Streams application logs: Caused by: org.apache.kafka.common.errors.TimeoutException: Expiring 7 record(s) for my-table-changelog-18: 30011 ms has passed since last append What kind of retries to I get by default publishing to changelogs? I have these producer

Build failed in Jenkins: kafka-trunk-jdk9 #326

See Changes: [wangguoz] KAFKA-6461 TableTableJoinIntegrationTest is unstable if caching is -- [...truncated 1.87 MB...] org.apache.kafka.streams.kstream.WindowTest > shoul

Re: [VOTE] KIP-227: Introduce Fetch Requests that are Incremental to Increase Partition Scalability

On Tue, Jan 23, 2018, at 11:57, Jun Rao wrote: > Hi, Collin, > > Thanks for the updated KIP. +1. Just a minor comment. It seems that it's > better for TotalIncrementalFetchSessionsEvicted to be a rate, instead of > just an ever-growing count. Thanks. Perhaps we can add the rate in addition to th

[VOTE] KIP-232: Detect outdated metadata using leaderEpoch and partitionEpoch

Hi all, I would like to start the voting process for KIP-232: https://cwiki.apache.org/confluence/display/KAFKA/KIP-232%3A+Detect+outdated+metadata+using+leaderEpoch+and+partitionEpoch The KIP will help fix a concurrency issue in Kafka which currently can cause message loss or message duplicatio

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Hey Jun, Jason, Thanks much for all the review! I will open the voting thread. Regards, Dong On Tue, Jan 23, 2018 at 3:37 PM, Jun Rao wrote: > Hi, Dong, > > The current KIP looks good to me. > > Thanks, > > Jun > > On Tue, Jan 23, 2018 at 12:29 PM, Dong Lin wrote: > > > Hey Jun, > > > > Do yo

[jira] [Created] (KAFKA-6476) Document dynamic config update

Rajini Sivaram created KAFKA-6476: - Summary: Document dynamic config update Key: KAFKA-6476 URL: https://issues.apache.org/jira/browse/KAFKA-6476 Project: Kafka Issue Type: Sub-task

Re: [VOTE] KIP-249: Add Delegation Token Operations to Kafka Admin Client

Hi, Mani, Thanks for the KIP. +1 Jun On Sun, Jan 21, 2018 at 7:44 AM, Manikumar wrote: > Hi All, > > I would like to start a vote on KIP-249 which would add delegation token > operations > to Java Admin Client. > > We have merged DelegationToken API PR recently. We want to include admin > clie

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Hi, Dong, The current KIP looks good to me. Thanks, Jun On Tue, Jan 23, 2018 at 12:29 PM, Dong Lin wrote: > Hey Jun, > > Do you think the current KIP looks OK? I am wondering if we can open the > voting thread. > > Thanks! > Dong > > On Fri, Jan 19, 2018 at 3:08 PM, Dong Lin wrote: > > > Hey

[jira] [Resolved] (KAFKA-6461) TableTableJoinIntegrationTest is unstable if caching is enabled

[ https://issues.apache.org/jira/browse/KAFKA-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-6461. Resolution: Fixed > TableTableJoinIntegrationTest is unstable if caching is enabled > --

Re: [VOTE] KIP-86: Configurable SASL callback handlers

Hi, Rajini, Thanks for the KIP. +1 from me. Jun On Thu, Jan 18, 2018 at 8:58 AM, tao xiao wrote: > +1 (non-binding) > > On Fri, 19 Jan 2018 at 00:47 Rajini Sivaram > wrote: > > > Hi all, > > > > I would like to restart the vote for KIP-86: > >https://cwiki.apache.org/confluence/display/K

Re: 1.1 KIPs

Hi, Today is the KIP deadline. Tomorrow I'll update the release page with any KIPS that have recently been voted on and accepted according to the process that can be found here: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-Process Thanks,

Re: [VOTE] KIP-247: Add public test utils for Kafka Streams

One minor change to the KIP. The class TopologyTestDriver will be in package `org.apache.kafka.streams` (instead of `o.a.k.streams.test`). +1 (binding). I am closing this vote as accepted with 3 binding votes (Damian, Guozhang, Matthias) and 2 non-binding votes (Bill, James). Thanks for the di

Build failed in Jenkins: kafka-trunk-jdk8 #2345

See Changes: [jason] KAFKA-6241; Enable dynamic updates of broker SSL keystore (#4263) -- [...truncated 123.92 KB...] kafka.tools.ConsoleProducerTest > testValidConfigsOld

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

Since we are running out of time to get the whole ConfigCommand converted to using the new AdminClient for 1.1.0 (KIP-248), we need a way to enable ConfigCommand to handle broker config updates (implemented by KIP-226). As a simple first step, it would make sense to use the existing ConfigCommand t

Re: [VOTE] KIP-212: Enforce set of legal characters for connector names

+1 (binding) On Tue, Jan 23, 2018, 1:17 PM Randall Hauch wrote: > +1 (non-binding) > > On Mon, Jan 22, 2018 at 6:35 PM, Sönke Liebau < > soenke.lie...@opencore.com.invalid> wrote: > > > All, > > > > this KIP has been discussed for quite some time now and I believe we > > addressed all major conc

[jira] [Created] (KAFKA-6475) ConfigException on the broker results in UnknownServerException in the admin client

Xavier Léauté created KAFKA-6475: Summary: ConfigException on the broker results in UnknownServerException in the admin client Key: KAFKA-6475 URL: https://issues.apache.org/jira/browse/KAFKA-6475 Pro

RE: [VOTE] KIP-145: Expose Record Headers in Kafka Connect

+1 (non-binding) I personally think the current proposed Converters described in the KIP are good, as Randall states it keeps it more in line with the pattern of the Converter methods, I think this is a good reason. -Original Message- From: Jason Gustafson [mailto:ja...@confluent.io]

Re: [VOTE] KIP-212: Enforce set of legal characters for connector names

+1 (non-binding) On Mon, Jan 22, 2018 at 6:35 PM, Sönke Liebau < soenke.lie...@opencore.com.invalid> wrote: > All, > > this KIP has been discussed for quite some time now and I believe we > addressed all major concerns in the current revision, so I'd like to > start a vote. > > KIP can be found h

Re: Regarding : Store stream for infinite time

It's simpler to set retention time to Long.MAX_VALUE instead of log compaction and generating unique keys. This effectively disable retention because the value is huge. The advantage is, that you don't need to worry about generating unique keys. -Matthias On 1/23/18 10:39 AM, Aman Rastogi wrote:

Re: [VOTE] KIP-231: Improve the Required ACL of ListGroups API

Hi Jason, Thanks for the vote and feedback. You are correct, I missed the possibility of using the wildcard permission. I'll update the rejected alternatives section accordingly. Thanks again. --Vahid From: Jason Gustafson To: dev@kafka.apache.org Date: 01/23/2018 08:27 AM Subject:

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Hey Jun, Do you think the current KIP looks OK? I am wondering if we can open the voting thread. Thanks! Dong On Fri, Jan 19, 2018 at 3:08 PM, Dong Lin wrote: > Hey Jun, > > I think we can probably have a static method in Util class to decode the > byte[]. Both KafkaConsumer implementation and

[jira] [Created] (KAFKA-6474) Rewrite test to use new public TopologyTestDriver

Matthias J. Sax created KAFKA-6474: -- Summary: Rewrite test to use new public TopologyTestDriver Key: KAFKA-6474 URL: https://issues.apache.org/jira/browse/KAFKA-6474 Project: Kafka Issue Typ

Re: [VOTE] KIP-145: Expose Record Headers in Kafka Connect

Hey Randall, It seemed a bit cleaner to me, but I'm not sure if whether there is any advantage to keeping the interfaces decoupled. I'd probably suggest going for the simpler API unless you can think of a good reason not to. -Jason On Tue, Jan 23, 2018 at 8:47 AM, Randall Hauch wrote: > I most

[jira] [Created] (KAFKA-6473) Add MockProcessorContext to public test-utils

Matthias J. Sax created KAFKA-6473: -- Summary: Add MockProcessorContext to public test-utils Key: KAFKA-6473 URL: https://issues.apache.org/jira/browse/KAFKA-6473 Project: Kafka Issue Type: I

Re: Wiki Page Request Permission

Hi, Litao, Thanks for your interest. Just gave you the wiki permission. Jun On Tue, Jan 23, 2018 at 9:33 AM, Litao Deng wrote: > Hey. > > I would like to add a KIP; however, I even can NOT see the 'Create' button > on the header of the KIP home page. > > Could you please grant me the permissio

Re: [VOTE] KIP-227: Introduce Fetch Requests that are Incremental to Increase Partition Scalability

Hi, Collin, Thanks for the updated KIP. +1. Just a minor comment. It seems that it's better for TotalIncrementalFetchSessionsEvicted to be a rate, instead of just an ever-growing count. Jun On Mon, Jan 22, 2018 at 4:35 PM, Jason Gustafson wrote: > > > > What if we want to have fetch sessions f

Build failed in Jenkins: kafka-trunk-jdk7 #3111

See Changes: [jason] KAFKA-6241; Enable dynamic updates of broker SSL keystore (#4263) -- [...truncated 1.90 MB...] org.apache.kafka.streams.tools.StreamsResetterTest > s

Re: [VOTE] KIP-212: Enforce set of legal characters for connector names

Thanks for the KIP! +1 (non-binding) Konstantine On Tue, Jan 23, 2018 at 10:48 AM, Ewen Cheslack-Postava wrote: > +1 (binding) > > On Tue, Jan 23, 2018 at 7:28 AM, Matt Farmer wrote: > > > +1 from me (non-binding) =) > > > > > On Jan 22, 2018, at 7:35 PM, Sönke Liebau INVALID> > > wrote: > >

Build failed in Jenkins: kafka-trunk-jdk9 #325

See Changes: [jason] KAFKA-6241; Enable dynamic updates of broker SSL keystore (#4263) -- [...truncated 1.45 MB...] kafka.server.DelayedOperationTest > shouldCancelForKeyR

Re: Official Kafka Gitter Room?

Hi, Gitter is used on apache spark project : Spark with Scala/Lobby - A place to discuss and ask questions about using Scala for Spark programming. Regards, JL

Wiki Page Request Permission

Hey. I would like to add a KIP; however, I even can NOT see the 'Create' button on the header of the KIP home page. Could you please grant me the permission? My username is "bryan.deng" Thanks a lot!

Re: [VOTE] KIP-212: Enforce set of legal characters for connector names

+1 (binding) On Tue, Jan 23, 2018 at 7:28 AM, Matt Farmer wrote: > +1 from me (non-binding) =) > > > On Jan 22, 2018, at 7:35 PM, Sönke Liebau > > > wrote: > > > > All, > > > > this KIP has been discussed for quite some time now and I believe we > > addressed all major concerns in the current

Re: Regarding : Store stream for infinite time

Thanks Svante. Regards, Aman On Tue, Jan 23, 2018 at 11:38 PM, Svante Karlsson wrote: > Yes, it will store the last value for each key > > 2018-01-23 18:30 GMT+01:00 Aman Rastogi : > > > Hi All, > > > > We have a use case to store stream for infinite time (given we have > enough > > storage). >

Re: [VOTE] KIP-229: DeleteGroups API

Thanks to those who provided feedback and/or voted on this KIP. The KIP passes with the following vote count: * 3 binding +1's by Rajini, Guozhang, Jason * 5 non-binding +1's by Jeff, Ted, Edo, Mickael, Colin Regards. --Vahid From: "Vahid S Hashemian" To: dev@kafka.apache.org Date: 01

Re: Regarding : Store stream for infinite time

Yes, it will store the last value for each key 2018-01-23 18:30 GMT+01:00 Aman Rastogi : > Hi All, > > We have a use case to store stream for infinite time (given we have enough > storage). > > We are planning to solve this by Log Compaction. If each message key is > unique and Log compaction is

Regarding : Store stream for infinite time

Hi All, We have a use case to store stream for infinite time (given we have enough storage). We are planning to solve this by Log Compaction. If each message key is unique and Log compaction is enabled, it will store whole stream for infinite time. Just wanted to check if my assumption is correct

Re: [VOTE] KIP-145: Expose Record Headers in Kafka Connect

I mostly just followed the pattern of the Converter methods, which also take the individual components. Really, the only advantage of the current approach is that the HeaderConverter implementations are a bit more decoupled as they are not aware of the Header/Headers API nor the implementation clas

Re: [VOTE] KIP-231: Improve the Required ACL of ListGroups API

+1. It's a little odd that we are still left with the anomalous cluster permission for ListGroups. You mention in the rejected alternatives that reducing permissions would make it impossible for admins to list groups, but I'm not sure why that would be the case. Wouldn't a wildcard group describe p

Re: [VOTE] KIP-212: Enforce set of legal characters for connector names

+1 from me (non-binding) =) > On Jan 22, 2018, at 7:35 PM, Sönke Liebau > wrote: > > All, > > this KIP has been discussed for quite some time now and I believe we > addressed all major concerns in the current revision, so I'd like to > start a vote. > > KIP can be found here: > https://cwiki.

[jira] [Created] (KAFKA-6472) WordCount example code error

JONYhao created KAFKA-6472: -- Summary: WordCount example code error Key: KAFKA-6472 URL: https://issues.apache.org/jira/browse/KAFKA-6472 Project: Kafka Issue Type: Bug Components: document

[jira] [Created] (KAFKA-6471) seekToEnd and seek give unclear results for Consumer with read_committed isolation level

Coen Damen created KAFKA-6471: - Summary: seekToEnd and seek give unclear results for Consumer with read_committed isolation level Key: KAFKA-6471 URL: https://issues.apache.org/jira/browse/KAFKA-6471 Proj