[GitHub] kafka-site pull request #16: Update design.html

2016-07-07 Thread nihed
Github user nihed closed the pull request at: https://github.com/apache/kafka-site/pull/16 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is en

[jira] [Commented] (KAFKA-1067) the default partitioner should be randomizing messages and a new partition for the meta refresh requirements created

2016-07-07 Thread Asaf Mesika (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365726#comment-15365726 ] Asaf Mesika commented on KAFKA-1067: After hitting this issue in production and search

Pros and cons of dockerizing kafka brokers?

2016-07-07 Thread Krish
Hi, I am currently testing a custom docker volume driver plugin for AWS EFS/EBS access and mounting. So, running kafka broker inside a container makes will ease up a lot of configuration issues wrt storage for me. Are there any pros and cons of dockerizing kafka broker? Off the top of my head, sin

Re: [VOTE] KIP-67: Queryable state for Kafka Streams

2016-07-07 Thread Damian Guy
Thanks Henry - we've updated the KIP with an example and the new config parameter required. FWIW the user doesn't register a listener, they provide a host:port in config. It is expected they will start a service running on that host:port that they can use to connect to the running KafkaStreams Inst

__consumer_offsets leader is -1

2016-07-07 Thread Kafka
Hi, __consumer_offsets ’s partition 7 and partition 27 leader is -1, and isr is null,who can tell me how to recover it,thank you. Topic: __consumer_offsets Partition: 0Leader: 3 Replicas: 3,4,5 Isr: 4,5,3 Topic: __consumer_offsets Partition: 1Leader: 4

[jira] [Created] (KAFKA-3933) Kafka OOM During Log Recovery Due to Leaked Native Memory

2016-07-07 Thread Tom Crayford (JIRA)
Tom Crayford created KAFKA-3933: --- Summary: Kafka OOM During Log Recovery Due to Leaked Native Memory Key: KAFKA-3933 URL: https://issues.apache.org/jira/browse/KAFKA-3933 Project: Kafka Issue T

[GitHub] kafka pull request #1421: KAFKA-3680: Enable Kafka clients to run in any cla...

2016-07-07 Thread rajinisivaram
GitHub user rajinisivaram reopened a pull request: https://github.com/apache/kafka/pull/1421 KAFKA-3680: Enable Kafka clients to run in any classloader env Configure default classes using class objects instead of class names, enable configurable lists of classes to be specified as c

[GitHub] kafka pull request #1421: KAFKA-3680: Enable Kafka clients to run in any cla...

2016-07-07 Thread rajinisivaram
Github user rajinisivaram closed the pull request at: https://github.com/apache/kafka/pull/1421 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[jira] [Commented] (KAFKA-3680) Make Java client classloading more flexible

2016-07-07 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366109#comment-15366109 ] ASF GitHub Bot commented on KAFKA-3680: --- GitHub user rajinisivaram reopened a pull r

[jira] [Commented] (KAFKA-3680) Make Java client classloading more flexible

2016-07-07 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366108#comment-15366108 ] ASF GitHub Bot commented on KAFKA-3680: --- Github user rajinisivaram closed the pull r

[jira] [Updated] (KAFKA-3680) Make Java client classloading more flexible

2016-07-07 Thread Rajini Sivaram (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram updated KAFKA-3680: -- Fix Version/s: 0.10.1.0 Status: Patch Available (was: Open) > Make Java client class

Re: [VOTE] KIP-67: Queryable state for Kafka Streams

2016-07-07 Thread Michael Noll
+1 (non-binding) On Thu, Jul 7, 2016 at 10:24 AM, Damian Guy wrote: > Thanks Henry - we've updated the KIP with an example and the new config > parameter required. FWIW the user doesn't register a listener, they provide > a host:port in config. It is expected they will start a service running on

[jira] [Commented] (KAFKA-3101) Optimize Aggregation Outputs

2016-07-07 Thread Eno Thereska (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366124#comment-15366124 ] Eno Thereska commented on KAFKA-3101: - [~bbejeck] Sounds great. So the KIP is KIP-63 a

[jira] [Commented] (KAFKA-3101) Optimize Aggregation Outputs

2016-07-07 Thread Bill Bejeck (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366148#comment-15366148 ] Bill Bejeck commented on KAFKA-3101: [~enothereska] ok I get it now, KAFKA-3101 is bei

[VOTE] KIP-55: Secure quotas for authenticated users

2016-07-07 Thread Rajini Sivaram
I would like to initiate voting for KIP-55 ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users). Since the KIP has changed quite a lot since the last vote, we will discard the previous vote and start this new voting thread. KIP-55 extends the existin

[jira] [Commented] (KAFKA-3933) Kafka OOM During Log Recovery Due to Leaked Native Memory

2016-07-07 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366174#comment-15366174 ] Ismael Juma commented on KAFKA-3933: Nice catch. The suggested solution sounds good to

[jira] [Updated] (KAFKA-3933) Kafka OOM During Log Recovery Due to Leaked Native Memory

2016-07-07 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3933: --- Priority: Critical (was: Major) > Kafka OOM During Log Recovery Due to Leaked Native Memory > ---

[jira] [Updated] (KAFKA-3933) Kafka OOM During Log Recovery Due to Leaked Native Memory

2016-07-07 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3933: --- Fix Version/s: 0.10.0.1 > Kafka OOM During Log Recovery Due to Leaked Native Memory >

[jira] [Commented] (KAFKA-3101) Optimize Aggregation Outputs

2016-07-07 Thread Eno Thereska (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366230#comment-15366230 ] Eno Thereska commented on KAFKA-3101: - Yup. Thanks. > Optimize Aggregation Outputs >

[jira] [Commented] (KAFKA-2729) Cached zkVersion not equal to that in zookeeper, broker not recovering.

2016-07-07 Thread Chris Rodier (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366237#comment-15366237 ] Chris Rodier commented on KAFKA-2729: - We also observed this identical issue, on 0.9.0

[jira] [Created] (KAFKA-3934) kafka-server-start.sh enables GC by default with no way to disable

2016-07-07 Thread Grant Henke (JIRA)
Grant Henke created KAFKA-3934: -- Summary: kafka-server-start.sh enables GC by default with no way to disable Key: KAFKA-3934 URL: https://issues.apache.org/jira/browse/KAFKA-3934 Project: Kafka

join in the dev group

2016-07-07 Thread caizhiqi...@kingsoft.com
hello, i wanna join and learn more kafka. thanks so much . caizhiqi...@kingsoft.com

Using KIP Process in Apache Flink

2016-07-07 Thread Aljoscha Krettek
Hi Kafka Community, I'm a member of the Flink community and I would like to ask if you're okay with us using your KIP process for Flink. We are at a stage where we have to be more careful in planning big feature additions to Flink and are looking for ways to formalize our processes there. I feel we

Re: Using KIP Process in Apache Flink

2016-07-07 Thread Ismael Juma
Hi Aljoscha, Thanks for sharing your intention to use a process similar to our KIP process. You are more than welcome to copy the structures and docs that we have for the KIP process. :) Ismael On Thu, Jul 7, 2016 at 4:16 PM, Aljoscha Krettek wrote: > Hi Kafka Community, > I'm a member of the

Re: join in the dev group

2016-07-07 Thread Matthias J. Sax
Hi, if you want to subscribe to dev list, you need to send an email to dev-subscr...@kafka.apache.org See: https://kafka.apache.org/contact.html -Matthias On 07/07/2016 06:04 AM, caizhiqi...@kingsoft.com wrote: > > hello, > i wanna join and learn more kafka. thanks so much . > > >

[jira] [Commented] (KAFKA-3933) Kafka OOM During Log Recovery Due to Leaked Native Memory

2016-07-07 Thread Tom Crayford (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366329#comment-15366329 ] Tom Crayford commented on KAFKA-3933: - That makes sense to me. Thanks for the pointer.

[jira] [Commented] (KAFKA-3894) Log Cleaner thread crashes and never restarts

2016-07-07 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366339#comment-15366339 ] Jun Rao commented on KAFKA-3894: [~wushujames], this is slightly different from KAFKA-3810

[jira] [Commented] (KAFKA-2985) Consumer group stuck in rebalancing state

2016-07-07 Thread Rajneesh Mitharwal (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366378#comment-15366378 ] Rajneesh Mitharwal commented on KAFKA-2985: --- I am also facing this problem. Brok

Re: [VOTE] KIP-67: Queryable state for Kafka Streams

2016-07-07 Thread Henry Cai
+1 On Thu, Jul 7, 2016 at 6:48 AM, Michael Noll wrote: > +1 (non-binding) > > On Thu, Jul 7, 2016 at 10:24 AM, Damian Guy wrote: > > > Thanks Henry - we've updated the KIP with an example and the new config > > parameter required. FWIW the user doesn't register a listener, they > provide > > a

[jira] [Commented] (KAFKA-3932) Consumer fails to consume in a round robin fashion

2016-07-07 Thread Vahid Hashemian (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366402#comment-15366402 ] Vahid Hashemian commented on KAFKA-3932: Can what you suggest be implemented with

Re: KStream: KTable-KTable leftJoin with key only on RHS of join generates null in joined table

2016-07-07 Thread Guozhang Wang
Hello, The problem is that, in stream processing when you have a new record coming from the big table tB and does not find a match with the small table (i.e. only a sub set of keys in practice) in tA, you cannot tell which case of the following is true: 1. there was a matching record in tA, which

[jira] [Commented] (KAFKA-3101) Optimize Aggregation Outputs

2016-07-07 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366418#comment-15366418 ] Guozhang Wang commented on KAFKA-3101: -- [~bbejeck] Regarding KAFKA-3776, there is sti

Re: [VOTE] KIP-67: Queryable state for Kafka Streams

2016-07-07 Thread Sriram Subramanian
+1 On Thu, Jul 7, 2016 at 9:53 AM, Henry Cai wrote: > +1 > > On Thu, Jul 7, 2016 at 6:48 AM, Michael Noll wrote: > > > +1 (non-binding) > > > > On Thu, Jul 7, 2016 at 10:24 AM, Damian Guy > wrote: > > > > > Thanks Henry - we've updated the KIP with an example and the new config > > > parameter

Re: KStream: KTable-KTable leftJoin with key only on RHS of join generates null in joined table

2016-07-07 Thread Philippe Derome
Oh ok, I had assumed that distinguishing cases 1 and 2 was possible. Thanks. On 7 Jul 2016 1:02 p.m., "Guozhang Wang" wrote: > Hello, > > The problem is that, in stream processing when you have a new record coming > from the big table tB and does not find a match with the small table (i.e. > onl

[jira] [Created] (KAFKA-3935) ConnectDistributedTest.test_restart_failed_task.connector_type=sink system test failing

2016-07-07 Thread Ewen Cheslack-Postava (JIRA)
Ewen Cheslack-Postava created KAFKA-3935: Summary: ConnectDistributedTest.test_restart_failed_task.connector_type=sink system test failing Key: KAFKA-3935 URL: https://issues.apache.org/jira/browse/KAFKA-

[jira] [Commented] (KAFKA-3933) Kafka OOM During Log Recovery Due to Leaked Native Memory

2016-07-07 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366526#comment-15366526 ] Ismael Juma commented on KAFKA-3933: Shall I assign the JIRA to you? > Kafka OOM Duri

Re: [DISCUSS] Client Side Auto Topic Creation

2016-07-07 Thread Guozhang Wang
I agree with Grant and Ismael regarding 1 / 2 / 3, and for 2) I would prefer having the default configs on the client side (i.e. like what we did in the kafka-topics script today) than on the cluster side, where the AdminClient will auto-set the configs if they are not specified by the user trigger

[jira] [Commented] (KAFKA-1754) KOYA - Kafka on YARN

2016-07-07 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366584#comment-15366584 ] Thomas Weise commented on KAFKA-1754: - KOYA is now part of Slider: https://github.com

[GitHub] kafka pull request #1597: KAFKA-3817 Follow-up: Avoid forwarding old value i...

2016-07-07 Thread guozhangwang
GitHub user guozhangwang opened a pull request: https://github.com/apache/kafka/pull/1597 KAFKA-3817 Follow-up: Avoid forwarding old value if it is null in KTableRepartition Also handle Null value in SmokeTestUtil. You can merge this pull request into a Git repository by running:

[jira] [Commented] (KAFKA-3817) KTableRepartitionMap should handle null inputs

2016-07-07 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366589#comment-15366589 ] ASF GitHub Bot commented on KAFKA-3817: --- GitHub user guozhangwang opened a pull requ

consumer.subscribe(Pattern p , ..) method fails with Authorizer

2016-07-07 Thread Manikumar Reddy
Hi, consumer.subscribe(Pattern p , ..) method implementation tries to get metadata of all the topics. This will throw TopicAuthorizationException on internal topics and other unauthorized topics. We may need to move the pattern matching to sever side. Is this know issue?. If not, I will raise JIR

[jira] [Commented] (KAFKA-3101) Optimize Aggregation Outputs

2016-07-07 Thread Bill Bejeck (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366591#comment-15366591 ] Bill Bejeck commented on KAFKA-3101: [~guozhang] ok will do. > Optimize Aggregation

[jira] [Assigned] (KAFKA-3933) Kafka OOM During Log Recovery Due to Leaked Native Memory

2016-07-07 Thread Tom Crayford (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom Crayford reassigned KAFKA-3933: --- Assignee: Tom Crayford > Kafka OOM During Log Recovery Due to Leaked Native Memory >

[jira] [Commented] (KAFKA-3933) Kafka OOM During Log Recovery Due to Leaked Native Memory

2016-07-07 Thread Tom Crayford (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366721#comment-15366721 ] Tom Crayford commented on KAFKA-3933: - Done. I hope to work on a fix tomorrow. > Kafk

Re: [DISCUSS] Client Side Auto Topic Creation

2016-07-07 Thread Jun Rao
It seems that it makes sense for the writer to trigger auto topic creation, but not the reader. So, my preference is Jay's option #1: add a new configuration to enable topic creation on the producer side and defaults to true. If the topic doesn't exist, the producer will send a createTopicRequest a

[jira] [Commented] (KAFKA-3689) Exception when attempting to decrease connection count for address with no connections

2016-07-07 Thread James Cheng (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366744#comment-15366744 ] James Cheng commented on KAFKA-3689: We are seeing this too. Kafka 0.10.0.0 I attache

[jira] [Updated] (KAFKA-3689) Exception when attempting to decrease connection count for address with no connections

2016-07-07 Thread James Cheng (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Cheng updated KAFKA-3689: --- Attachment: KAFKA-3689.log.redacted > Exception when attempting to decrease connection count for addre

Re: [VOTE] KIP-55: Secure quotas for authenticated users

2016-07-07 Thread Tom Crayford
+1 (non-binding) On Thursday, 7 July 2016, Rajini Sivaram wrote: > I would like to initiate voting for KIP-55 ( > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users > ). > Since the KIP has changed quite a lot since the last vote, we will discard

RE: Error after restarting kafka broker

2016-07-07 Thread Subhash Agrawal
Also including kafka user mailing list. Has anybody seen this error? Any idea to fix it? Thanks Subhash Agrawal. From: Subhash Agrawal Sent: Wednesday, July 06, 2016 5:12 PM To: 'dev@kafka.apache.org' Subject: Error after restarting kafka broker Hi All, I am running kafka broker (0.10.0 version)

SSL: Broker works but consumer/producer fail

2016-07-07 Thread Vineet Kumar
Hi I followed Apache Kafka SSL instructions verbatim but my producer and consumer both hang or error out as follows. openssl s_client BTW does work fine with the server below yielding certificates etc thereby confirming that the server can talk back SSL. *Producer and Consumer*

Kafka 0.10 Java Client TimeoutException: Batch containing 1 record(s) expired

2016-07-07 Thread Armin Kadiyan
Hi all, I have a single node, multi (3) broker Zookeeper / Kafka setup. I am using the Kafka 0.10 Java client. I wrote following simple remote (on a different Server than Kafka) Producer (in the code I replaced my public IP address with MYIP): Properties config = new Properties(); try { config