Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

2017-02-14 Thread Guozhang Wang
Some thoughts on the mixture usage of DSL / PAPI: There were some suggestions on mixing the usage of DSL and PAPI: https://issues.apache.org/jira/browse/KAFKA-3455, and after thinking it a bit more carefully, I'd rather not recommend users following this pattern, since in DSL this can always be ac

Re: Potential memory leak in rocksdb

2017-02-14 Thread Pierre Coquentin
I have checked from my previous test, and in each partition, there is between 2 and 4 sst files. I am open to any test which could pinpoint what I am missing :) Off topic question: what's the best solution to clean the rocksdb logs ? logrotate or is there a configuration directly in rocksdb ? Or ar

Re: Kafka streams: Getting a state store associated to a processor

2017-02-14 Thread Adam Warski
Hello, > Can you try this out with 0.10.2 branch or current trunk? With 0.10.2 this works fine! The state-store changelog is created with 3 partitions if the source topic has 3 partitions. I checked with client from https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka_2.1

Purpose of the metadata field of Offset Commit and Fetch

2017-02-14 Thread Henning Røigaard-Petersen
According to the API docs (https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol) Offset Commit and Offset Response sports a string field metadata, but does not explain its usage. Can anyone explain what it is for? Is it used by Kafka or is it wholly consumer-specific?

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

2017-02-14 Thread Mathieu Fenniak
On Tue, Feb 14, 2017 at 1:14 AM, Guozhang Wang wrote: > Some thoughts on the mixture usage of DSL / PAPI: > > There were some suggestions on mixing the usage of DSL and PAPI: > https://issues.apache.org/jira/browse/KAFKA-3455, and after thinking it a > bit more carefully, I'd rather not recommend

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

2017-02-14 Thread Damian Guy
> And about printing the topology for debuggability: I agrees this is a > > potential drawback, and I'd suggest maintain some functionality to build > a > > "dry topology" as Mathieu suggested; the difficulty is that, internally > we > > need a different "copy" of the topology for each thread so th

get function

2017-02-14 Thread Samy CHBINOU
In kafka there is a Subscribe API to be notified of incoming message values from a topic. I didn't see a Get(key, topic) function. I mean a Get function to retreive a value from a key inside a topic. Isn't this feature implemented? (Like in Redis). If it doesn't exists, do you think is is easy

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

2017-02-14 Thread Mathieu Fenniak
On Tue, Feb 14, 2017 at 9:37 AM, Damian Guy wrote: > > And about printing the topology for debuggability: I agrees this is a > > > potential drawback, and I'd suggest maintain some functionality to > build > > a > > > "dry topology" as Mathieu suggested; the difficulty is that, internally > > we

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

2017-02-14 Thread Guozhang Wang
> and you can't output multiple records or branching logic from a transform(); For output multiple records in transform, we are currently working on https://issues.apache.org/jira/browse/KAFKA-4217, I think that should cover this use case. For branching the output in transform, I agree this is no

Request for JIRA access

2017-02-14 Thread Thomas Dutta
Dear Team, I am a Computer Science graduate student at the University of Illinois at Chicago. I am a newbie and I want to contribute to Apache Kafka project. Could you please add me to the list so that I can assign JIRA ticket to myself. Please let me know if you need additional information. Reg

Request for JIRA access

2017-02-14 Thread Thomas Dutta
Dear Team, I am a Computer Science graduate student at the University of Illinois at Chicago. I am a newbie and I want to contribute to Apache Kafka project. Could you please add me to the list so that I can assign JIRA ticket to myself. Please let me know if you need additional information. Reg

Re: Request for JIRA access

2017-02-14 Thread Guozhang Wang
Thomas, Do you already have an apache id? I cannot find your name in apache wiki space. Guozhang On Tue, Feb 14, 2017 at 12:31 AM, Thomas Dutta wrote: > Dear Team, > > I am a Computer Science graduate student at the University of Illinois at > Chicago. I am a newbie and I want to contribute

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

2017-02-14 Thread Matthias J. Sax
You can already output any number of record within .transform() using the provided Context object from init()... -Matthias On 2/14/17 9:16 AM, Guozhang Wang wrote: >> and you can't output multiple records or branching logic from a > transform(); > > For output multiple records in transform, we

[VOTE] 0.10.2.0 RC2

2017-02-14 Thread Ewen Cheslack-Postava
Hello Kafka users, developers and client-developers, This is the third candidate for release of Apache Kafka 0.10.2.0. This is a minor version release of Apache Kafka. It includes 19 new KIPs. See the release notes and release plan (https://cwiki.apache.org/conf luence/display/KAFKA/Release+Plan+

Re: KIP-121 [VOTE]: Add KStream peek method

2017-02-14 Thread Steven Schlansker
Hi, it looks like I have 2 of the 3 minimum votes, can a third voter please consider this KIP? Thanks. (PS - new revision on GitHub PR with hopefully the last round of improvements) > On Feb 8, 2017, at 9:06 PM, Matthias J. Sax wrote: > > +1 > > On 2/8/17 4:51 PM, Gwen Shapira wrote: >> +1 (b

Re: KIP-121 [VOTE]: Add KStream peek method

2017-02-14 Thread Jay Kreps
+1 Nice improvement. -Jay On Tue, Feb 14, 2017 at 1:22 PM, Steven Schlansker < sschlans...@opentable.com> wrote: > Hi, it looks like I have 2 of the 3 minimum votes, can a third voter > please consider this KIP? > Thanks. > > (PS - new revision on GitHub PR with hopefully the last round of > im

Re: KIP-121 [VOTE]: Add KStream peek method

2017-02-14 Thread Zakee
+1 -Zakee > On Feb 14, 2017, at 1:56 PM, Jay Kreps wrote: > > +1 > > Nice improvement. > > -Jay > > On Tue, Feb 14, 2017 at 1:22 PM, Steven Schlansker < > sschlans...@opentable.com> wrote: > >> Hi, it looks like I have 2 of the 3 minimum votes, can a third voter >> please consider this KIP?

Auto creation of Topic and Security

2017-02-14 Thread Stephane Maarek
Hi, If I authorise applications to create their own topics (with auto create setting), will the topics automatically inherit some SASL ACLs? I’m wondering this because of for example Kafka Streams applications…. I know they can create their own temp topics and such. Just wondering what a good ACL

Re: Kafka streams: Getting a state store associated to a processor

2017-02-14 Thread Michael Noll
> By the way - do I understand correctly that when a state store is persistent, it is logged by default? Yes. > So enableLogging(Map) only is a way to provide default configuration to the default logging? Yes. That is, any configs that should be applied to the state store's changelog topic. >