Re: [VOTE] KIP-280: Enhanced log compaction

Hi, So, is everyone OK using the approach with 2 properties? E.g.: Scenario 1:     compaction.strategy: offset     :- Behaviour is the same as what currently exists, where the compaction is done only via the 'offset' Scenario 2:     compaction.strategy: timestamp     :- Similar to 'offset'

[jira] [Created] (KAFKA-7094) Variate should unify code style in one method, and use camel name

Matt Wang created KAFKA-7094: Summary: Variate should unify code style in one method, and use camel name Key: KAFKA-7094 URL: https://issues.apache.org/jira/browse/KAFKA-7094 Project: Kafka Iss

[VOTE] KIP-293: Add new metrics for consumer/replication fetch requests

Hello, In the absence of additional feedback on this KIP I'd like to start a vote. To summarize, the KIP simply proposes to add a consumer metric to track the number of fetch requests made by (real) client consumers and not other replicating brokers. KIP link: https://cwiki.apache.org/confluenc

Build failed in Jenkins: kafka-2.0-jdk8 #56

See Changes: [rajinisivaram] MINOR: Fix timing issue in advertised listener update test (#5256) -- [...truncated 432.85 KB...] kafka.zookeeper.ZooKeeperClientTest > testBlo

[VOTE] KIP-312: Add Overloaded StreamsBuilder Build Method to Accept java.util.Properties

All, I'd like to start a vote for this KIP now. Thanks, Bill

Re: [VOTE] KIP-312: Add Overloaded StreamsBuilder Build Method to Accept java.util.Properties

+1 (binding) On 6/25/18 6:11 AM, Bill Bejeck wrote: > All, > I'd like to start a vote for this KIP now. > > Thanks, > Bill > signature.asc Description: OpenPGP digital signature

Re: [VOTE] KIP-280: Enhanced log compaction

+1 from my side for using `compaction.strategy` with values "offset", "timestamp" and "header" and `compaction.strategy.header` -Matthias On 6/25/18 1:25 AM, Luís Cabral wrote: > Hi, > > So, is everyone OK using the approach with 2 properties? > > E.g.: > > Scenario 1: >     compaction.strate

[jira] [Created] (KAFKA-7095) Low traffic consumer is not consuming messages after the offsets is deleted by Kafka

Aldo Sinanaj created KAFKA-7095: --- Summary: Low traffic consumer is not consuming messages after the offsets is deleted by Kafka Key: KAFKA-7095 URL: https://issues.apache.org/jira/browse/KAFKA-7095 Proj

[DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

Hey All, I wrote up a KIP to handle one more edge case in the replication protocol and to support better handling of truncation in the consumer when unclean leader election is enabled. Let me know what you think. https://cwiki.apache.org/confluence/display/KAFKA/KIP-320%3A+Allow+fetchers+to+detec

Request permission to assign JIRA

Hi Team, I am trying to claim a bug in Jira, Could you please help me gain the required permissions. my JIRA usernane is lambdaliu. thanks, lambdaliu.

Re: Request permission to assign JIRA

Added. Thanks for contributing! -Jason On Mon, Jun 25, 2018 at 9:27 AM, lambdaliu(刘少波) wrote: > Hi Team, > > I am trying to claim a bug in Jira, Could you please help me gain the > required permissions. > my JIRA usernane is lambdaliu. > > thanks, > lambdaliu. >

Re: [VOTE] KIP-280: Enhanced log compaction

+1 from me as well. On Mon, Jun 25, 2018 at 8:16 AM, Matthias J. Sax wrote: > +1 from my side for using `compaction.strategy` with values "offset", > "timestamp" and "header" and `compaction.strategy.header` > > -Matthias > > On 6/25/18 1:25 AM, Luís Cabral wrote: > > Hi, > > > > So, is everyon

Re: [VOTE] KIP-312: Add Overloaded StreamsBuilder Build Method to Accept java.util.Properties

+1. On Mon, Jun 25, 2018 at 8:12 AM, Matthias J. Sax wrote: > +1 (binding) > > On 6/25/18 6:11 AM, Bill Bejeck wrote: > > All, > > I'd like to start a vote for this KIP now. > > > > Thanks, > > Bill > > > > -- -- Guozhang

Re: [VOTE] KIP-312: Add Overloaded StreamsBuilder Build Method to Accept java.util.Properties

+1 On Mon, Jun 25, 2018 at 9:45 AM, Guozhang Wang wrote: > +1. > > On Mon, Jun 25, 2018 at 8:12 AM, Matthias J. Sax > wrote: > > > +1 (binding) > > > > On 6/25/18 6:11 AM, Bill Bejeck wrote: > > > All, > > > I'd like to start a vote for this KIP now. > > > > > > Thanks, > > > Bill > > > > > > >

[DISCUSS] KIP-323: Schedulable KTable as Graph source

Hey, guys, I've just started a KIP discussion here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-323%3A+Schedulable+KTable+as+Graph+source

Re: [DISCUSS] KIP-308: Support dynamic update of max.connections.per.ip/max.connections.per.ip.overrides configs

Hey Manikumar, Thanks for the KIP. This seems useful. -Jason On Thu, Jun 21, 2018 at 11:38 PM, Manikumar wrote: > Hi all, > > I have created a KIP to add support for dynamic update of > max.connections.per.ip/max.connections.per.ip.overrides configs > > *https://cwiki.apache.org/confluence/pag

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

I'm wondering if StreamThreadNotStarted could be merged into StreamThreadNotRunning, because I think users' handling logic for the third case would be likely the same as the second. Do you have some scenarios where users may want to handle them differently? Guozhang On Sun, Jun 24, 2018 at 5:25 P

Re: [DISCUSS] KIP-308: Support dynamic update of max.connections.per.ip/max.connections.per.ip.overrides configs

Hey Manikumar, Thanks much for the KIP. It looks pretty good. Thanks, Dong On Thu, Jun 21, 2018 at 11:38 PM, Manikumar wrote: > Hi all, > > I have created a KIP to add support for dynamic update of > max.connections.per.ip/max.connections.per.ip.overrides configs > > *https://cwiki.apache.org/

Re: [DISCUSS] KIP-319: Replace segments with segmentSize in WindowBytesStoreSupplier

Re `segmentInterval` parameter in Windows: currently it is used in two places, the windowed stream aggregation, and the stream-stream joins. For the former, we can potentially move the parameter from windowedBy() to Materialized, but for the latter we currently do not expose a Materialized object y

Re: [Discuss] KIP-321: Add method to get TopicNameExtractor in TopologyDescription

Yup I agree, generally speaking the `toString()` output is not recommended to be relied on programmatically in user's code, but we've observed convenience-beats-any-other-reasons again and again in development unfortunately. I think we should still not claiming it is part of the public APIs that wo

Re: [kafka-clients] Re: [VOTE] 1.1.1 RC1

+1 (non-binding) Ran tests, Verified quick start, producer/consumer perf tests On Sat, Jun 23, 2018 at 8:11 AM Dong Lin wrote: > Thank you for testing and voting the release! > > I noticed that the date for 1.1.1-rc1 is wrong. Please kindly test and > vote by Tuesday, June 26, 12 pm PT. > >

Re: [VOTE] KIP-312: Add Overloaded StreamsBuilder Build Method to Accept java.util.Properties

Thanks Bill! +1 On Mon, 25 Jun 2018 at 18:57 Ted Yu wrote: > +1 > > On Mon, Jun 25, 2018 at 9:45 AM, Guozhang Wang wrote: > > > +1. > > > > On Mon, Jun 25, 2018 at 8:12 AM, Matthias J. Sax > > wrote: > > > > > +1 (binding) > > > > > > On 6/25/18 6:11 AM, Bill Bejeck wrote: > > > > All, > > > >

[jira] [Created] (KAFKA-7096) Consumer should drop the data for unassigned topic partitions

Mayuresh Gharat created KAFKA-7096: -- Summary: Consumer should drop the data for unassigned topic partitions Key: KAFKA-7096 URL: https://issues.apache.org/jira/browse/KAFKA-7096 Project: Kafka

Request access to create KIP

Hi, my wiki id is gyishun. Thanks! - Yishun

Re: [DISCUSS] - KIP-314: KTable to GlobalKTable Bi-directional Join

Thanks for your help so far guys. While I do think that I have a fairly reasonable way forward for restructuring the topologies and threads, there is, unfortunately, what I believe is a fatal flaw that cannot be easily resolved. I have updated the page ( https://cwiki.apache.org/confluence/display

Re: [VOTE] 2.0.0 RC0

+1 (non-binding) Heroku has run our usual set of upgrade and performance tests, and we haven't found any notable issues through that. On Sat, Jun 23, 2018 at 12:30 AM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > +1 (non-binding) > > Built from source and ran quickstart successfully on

Build failed in Jenkins: kafka-trunk-jdk8 #2766

See Changes: [mjsax] Minor: add exception to debug log for -- [...truncated 877.51 KB...] kafka.utils.SchedulerTest > testRestart STARTED kafka.utils.SchedulerTest > tes

[jira] [Created] (KAFKA-7097) VerifiableProducer does not work properly with --message-create-time argument

Jasper Knulst created KAFKA-7097: Summary: VerifiableProducer does not work properly with --message-create-time argument Key: KAFKA-7097 URL: https://issues.apache.org/jira/browse/KAFKA-7097 Project:

Re: Request access to create KIP

Hi, could someone give me access to create KIP? Thanks! - Yishun On Mon, Jun 25, 2018, 10:44 AM Yishun Guan wrote: > Hi, my wiki id is gyishun. Thanks! - Yishun >

SASL Unit test failing

Hello, I did a fresh clone of the kafka src code, and the following SASL unit tests have been failing consistently: - testMechanismPluggability - testMechanismPluggability - testMultipleServerMechanisms All three tests have similar stack trace: at org.junit.Assert.assertTrue(Assert.java:52) at or

Re: Request access to create KIP

Done. Thanks for contributing! -Jason On Mon, Jun 25, 2018 at 12:49 PM, Yishun Guan wrote: > Hi, could someone give me access to create KIP? Thanks! - Yishun > > On Mon, Jun 25, 2018, 10:44 AM Yishun Guan wrote: > > > Hi, my wiki id is gyishun. Thanks! - Yishun > > >

Re: Request access to create KIP

Thank you! On Mon, Jun 25, 2018, 1:03 PM Jason Gustafson wrote: > Done. Thanks for contributing! > > -Jason > > On Mon, Jun 25, 2018 at 12:49 PM, Yishun Guan wrote: > > > Hi, could someone give me access to create KIP? Thanks! - Yishun > > > > On Mon, Jun 25, 2018, 10:44 AM Yishun Guan wrote:

Re: [DISCUSS] KIP-319: Replace segments with segmentSize in WindowBytesStoreSupplier

Thanks Matthias and Guozhang, About deprecating the "segments" field instead of making it private. Yes, I just took another look at the code, and that is correct. I'll update the KIP. I do agree that in the long run, it makes more sense as a parameter to the store somehow than as a parameter to t

Re: [DISCUSS] KIP-319: Replace segments with segmentSize in WindowBytesStoreSupplier

I agree that it makes sense to have segmentInterval as a parameter to a store, but I also agree with Guozhang's point about not moving as part of this KIP. Thanks, Bill On Mon, Jun 25, 2018 at 4:17 PM John Roesler wrote: > Thanks Matthias and Guozhang, > > About deprecating the "segments" field

Re: [Discuss] KIP-321: Add method to get TopicNameExtractor in TopologyDescription

Thanks for the KIP! Overall I'm +1 on the KIP. I have one question. The KIP states that the method "topicNameExtractor()" is added to the InternalTopologyBuilder.java. It could be that I'm missing something, but wow does this work if a user has provided different TopicNameExtractor instances t

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

The scenario I had I mind was, that KS is started in one thread while a second thread has a reference to the object to issue queries. If a query is issue before the "main thread" started KS, and the "query thread" knows that it will eventually get started, it can retry. On the other hand, if KS is

Re: [DISCUSS] KIP-319: Replace segments with segmentSize in WindowBytesStoreSupplier

Ok. @John: can you create a JIRA to track this? I think KAFKA-4730 is related, but actually an own ticket (that is blocked by not having Materialized for stream-stream joins). -Matthias On 6/25/18 2:10 PM, Bill Bejeck wrote: > I agree that it makes sense to have segmentInterval as a parameter t

Re: [DISCUSS] KIP-319: Replace segments with segmentSize in WindowBytesStoreSupplier

Matthias, That's a good idea. I'm not sure why I didn't... Thanks, -John On Mon, Jun 25, 2018 at 4:35 PM Matthias J. Sax wrote: > Ok. > > @John: can you create a JIRA to track this? I think KAFKA-4730 is > related, but actually an own ticket (that is blocked by not having > Materialized for st

Re: [DISCUSS] KIP-319: Replace segments with segmentSize in WindowBytesStoreSupplier

Ah, it turns out I did create a ticket: it's KAFKA-7080: https://issues.apache.org/jira/browse/KAFKA-7080 -John On Mon, Jun 25, 2018 at 4:44 PM John Roesler wrote: > Matthias, > > That's a good idea. I'm not sure why I didn't... > > Thanks, > -John > > On Mon, Jun 25, 2018 at 4:35 PM Matthias J

Re: SASL Unit test failing

I ran the test alone which passed. Can you include -i on the command line to see if there is some clue from the output ? Here is my environment: Java version: 1.8.0_151, vendor: Oracle Corporation Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/jre Default locale: en_

[jira] [Resolved] (KAFKA-6978) Make Streams Window retention time strict

[ https://issues.apache.org/jira/browse/KAFKA-6978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler resolved KAFKA-6978. - Resolution: Fixed Fix Version/s: 2.1.0 This feature was merged in  https://github.com/apac

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

Hi Harsha, If I understand correctly, the replication quota mechanism proposed in KIP-73 can be helpful in that scenario. Have you tried it out? Thanks, Lucas On Sun, Jun 24, 2018 at 8:28 AM, Harsha wrote: > Hi Lucas, > One more question, any thoughts on making this configurable >

[VOTE] KIP-319: Replace numSegments to segmentInterval in Streams window configurations

Hello All, Thanks for the discussion on KIP-319. I'd now like to start the voting. As a reminder, KIP-319 proposes a fix to an issue I identified in KAFKA-7080. Specifically, the issue is that we're creating CachingWindowStore with the *number of segments* instead of the *segment size*. Here's t

Re: [VOTE] KIP-319: Replace numSegments to segmentInterval in Streams window configurations

+1 On Mon, Jun 25, 2018 at 2:56 PM, John Roesler wrote: > Hello All, > > Thanks for the discussion on KIP-319. I'd now like to start the voting. > > As a reminder, KIP-319 proposes a fix to an issue I identified in > KAFKA-7080. Specifically, the issue is that we're creating > CachingWindowStore

Re: [VOTE] KIP-319: Replace numSegments to segmentInterval in Streams window configurations

+1 On Mon, Jun 25, 2018 at 2:58 PM, Ted Yu wrote: > +1 > > On Mon, Jun 25, 2018 at 2:56 PM, John Roesler wrote: > > > Hello All, > > > > Thanks for the discussion on KIP-319. I'd now like to start the voting. > > > > As a reminder, KIP-319 proposes a fix to an issue I identified in > > KAFKA-70

Re: [DISCUSS] KIP-323: Schedulable KTable as Graph source

This would be useful feature. In the Public Interfaces section, the new method lacks a closing parenthesis. In the Proposed Changes section, if the order of the 3 bullets can match the order of the parameters of the new method, it would be easier to read. For Rejected Alternatives #2, can you ad

Re: [DISCUSS] KIP-323: Schedulable KTable as Graph source

Flávio, thanks for creating this KIP. I think this "single-aggregation" use case is common enough that we should consider how to efficiently supports it: for example, for KSQL that's built on top of Streams, we've seen lots of query statements whose return is expected a single row indicating the "

Re: [Discuss] KIP-321: Add method to get TopicNameExtractor in TopologyDescription

Good catch. I think the proposed change is to add that function in InternalTopologyBuilder#Sink class. Guozhang On Mon, Jun 25, 2018 at 2:23 PM, Bill Bejeck wrote: > Thanks for the KIP! > > Overall I'm +1 on the KIP. I have one question. > > The KIP states that the method "topicNameExtracto

Re: [Discuss] KIP-321: Add method to get TopicNameExtractor in TopologyDescription

If there are different TopicNameExtractor classes from multiple sink nodes, the new method should throw exception alerting user of such scenario. On Mon, Jun 25, 2018 at 2:23 PM, Bill Bejeck wrote: > Thanks for the KIP! > > Overall I'm +1 on the KIP. I have one question. > > The KIP states th

Re: SASL Unit test failing

I ran test with -i option as follows - "./gradlew -i test". The same set of three tests failed. My environment: $ java -version java version "1.8.0_121" Java(TM) SE Runtime Environment (build 1.8.0_121-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode) $ cat /etc/redhat-relea

Re: SASL Unit test failing

Here was the command I used: ./gradlew -Dtest.single=SaslAuthenticatorTest clients:test On Mon, Jun 25, 2018 at 5:39 PM, Ahmed A wrote: > I ran test with -i option as follows - "./gradlew -i test". The same set > of three tests failed. > > My environment: > $ java -version > java version "1.8

Re: SASL Unit test failing

I ran the test on Linux as well. cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) Java version: 1.8.0_161, vendor: Oracle Corporation Java home: /jdk1.8.0_161/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-327.28.3.el7.x86_64", arch: "amd64", fa

[jira] [Created] (KAFKA-7098) Improve accuracy of the log cleaner throttle rate

Dong Lin created KAFKA-7098: --- Summary: Improve accuracy of the log cleaner throttle rate Key: KAFKA-7098 URL: https://issues.apache.org/jira/browse/KAFKA-7098 Project: Kafka Issue Type: Improvement

Re: [Discuss] KIP-321: Add method to get TopicNameExtractor in TopologyDescription

Ted, Why? Each sink can have a different TopicNameExtractor. -Matthias On 6/25/18 5:19 PM, Ted Yu wrote: > If there are different TopicNameExtractor classes from multiple sink nodes, > the new method should throw exception alerting user of such scenario. > > > On Mon, Jun 25, 2018 at 2:23 PM,

Build failed in Jenkins: kafka-trunk-jdk8 #2767

See Changes: [jason] KAFKA-6591; Move super user check before ACL matching (#4618) -- [...truncated 1.94 MB...] org.apache.kafka.common.acl.AclOperationTest > testName S

Re: [VOTE] 1.0.2 RC0

+1 (non-binding) Verified tests, quick start, producer/consumer perf tests. On Sat, Jun 23, 2018 at 2:25 AM Ted Yu wrote: > +1 > > Ran test suite. > > Checked signatures. > > On Fri, Jun 22, 2018 at 11:42 AM, Vahid S Hashemian < > vahidhashem...@us.ibm.com> wrote: > > > +1 (non-binding) > > > >

Re: [DISCUSS] KIP-319: Replace segments with segmentSize in WindowBytesStoreSupplier

KAFKA-7080 is for this KIP. I meant to create a JIRA to add `segmentInterval` to `Materialized` and a JIRA to add `Materialized` to `KStream#join(KStream)`. Thx. -Matthias On 6/25/18 2:46 PM, John Roesler wrote: > Ah, it turns out I did create a ticket: it's KAFKA-7080: > https://issues.apache

Re: [VOTE] KIP-319: Replace numSegments to segmentInterval in Streams window configurations

+1 (binding) On 6/25/18 3:00 PM, Guozhang Wang wrote: > +1 > > On Mon, Jun 25, 2018 at 2:58 PM, Ted Yu wrote: > >> +1 >> >> On Mon, Jun 25, 2018 at 2:56 PM, John Roesler wrote: >> >>> Hello All, >>> >>> Thanks for the discussion on KIP-319. I'd now like to start the voting. >>> >>> As a remind

Re: [Discuss] KIP-321: Add method to get TopicNameExtractor in TopologyDescription

My previous response was talking about the new method in InternalTopologyBuilder. The exception just means there is no uniform extractor for all the sinks. On Mon, Jun 25, 2018 at 8:02 PM, Matthias J. Sax wrote: > Ted, > > Why? Each sink can have a different TopicNameExtractor. > > > -Matthias >