[jira] [Updated] (KAFKA-1620) Make kafka api protocol implementation public

[Anton Karamanov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Karamanov updated KAFKA-1620:
---
Attachment: 0002-KAFKA-1620-Make-kafka-api-protocol-implementation-pu.patch

Not really. I just thought it might become useful at some point as well. I took 
a closer look and now realize that it's related to internal messages, so 
there's no actual need to make them public.

Here's updated and rebased 
[patch|^0002-KAFKA-1620-Make-kafka-api-protocol-implementation-pu.patch].

> Make kafka api protocol implementation public
> -
>
> Key: KAFKA-1620
> URL: https://issues.apache.org/jira/browse/KAFKA-1620
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Anton Karamanov
>Assignee: Anton Karamanov
> Attachments: 
> 0001-KAFKA-1620-Make-kafka-api-protocol-implementation-pu.patch, 
> 0002-KAFKA-1620-Make-kafka-api-protocol-implementation-pu.patch
>
>
> Some of the classes which implement Kafka api protocol, such as 
> {{RequestOrResponse}} and {{FetchRequest}} are defined as private to 
> {{kafka}} package. Those classes would be extremely usefull for writing 
> custom clients (we're using Scala with Akka and implementing one directly on 
> top of Akka TCP), and don't seem to contain any actuall internal logic of 
> Kafka. Therefore it seems like a nice idea to make them public.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-1635: Fixed incorrect java doc of makeLe...

[LantaoJin]

GitHub user LantaoJin opened a pull request:

https://github.com/apache/kafka/pull/33

KAFKA-1635: Fixed incorrect java doc of makeLeaders() in ReplicaManager



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/LantaoJin/kafka KAFKA-1635

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/33.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #33


commit 6739a8e601331ad07d9856dc351785351755a5d5
Author: LantaoJin 
Date:   2014-09-16T08:26:34Z

KAFKA-1635: Fixed incorrect java doc of makeLeaders() in ReplicaManager




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-1635) Java doc of makeLeaders in ReplicaManager is wrong

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135135#comment-14135135
 ] 

ASF GitHub Bot commented on KAFKA-1635:
---

GitHub user LantaoJin opened a pull request:

https://github.com/apache/kafka/pull/33

KAFKA-1635: Fixed incorrect java doc of makeLeaders() in ReplicaManager



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/LantaoJin/kafka KAFKA-1635

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/33.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #33


commit 6739a8e601331ad07d9856dc351785351755a5d5
Author: LantaoJin 
Date:   2014-09-16T08:26:34Z

KAFKA-1635: Fixed incorrect java doc of makeLeaders() in ReplicaManager




> Java doc of makeLeaders in ReplicaManager is wrong
> --
>
> Key: KAFKA-1635
> URL: https://issues.apache.org/jira/browse/KAFKA-1635
> Project: Kafka
>  Issue Type: Bug
>  Components: core, replication
>Reporter: Lantao Jin
>Assignee: Neha Narkhede
>Priority: Minor
>  Labels: doc, server
> Attachments: kafka-1635-1.patch
>
>
> ReplicaManager have an incorrect java doc. The overview of function  
> makeLeaders() is the same as makeFollowers().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1635) Java doc of makeLeaders in ReplicaManager is wrong

[Lantao Jin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lantao Jin updated KAFKA-1635:
--
Priority: Trivial  (was: Minor)

> Java doc of makeLeaders in ReplicaManager is wrong
> --
>
> Key: KAFKA-1635
> URL: https://issues.apache.org/jira/browse/KAFKA-1635
> Project: Kafka
>  Issue Type: Bug
>  Components: core, replication
>Reporter: Lantao Jin
>Assignee: Neha Narkhede
>Priority: Trivial
>  Labels: doc, server
> Attachments: kafka-1635-1.patch
>
>
> ReplicaManager have an incorrect java doc. The overview of function  
> makeLeaders() is the same as makeFollowers().
> Also see commit at 
> https://github.com/apache/kafka/commit/6739a8e601331ad07d9856dc351785351755a5d5



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1635) Java doc of makeLeaders in ReplicaManager is wrong

[Lantao Jin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lantao Jin updated KAFKA-1635:
--
Description: 
ReplicaManager have an incorrect java doc. The overview of function  
makeLeaders() is the same as makeFollowers().
Also see commit at 
https://github.com/apache/kafka/commit/6739a8e601331ad07d9856dc351785351755a5d5

  was:ReplicaManager have an incorrect java doc. The overview of function  
makeLeaders() is the same as makeFollowers().


> Java doc of makeLeaders in ReplicaManager is wrong
> --
>
> Key: KAFKA-1635
> URL: https://issues.apache.org/jira/browse/KAFKA-1635
> Project: Kafka
>  Issue Type: Bug
>  Components: core, replication
>Reporter: Lantao Jin
>Assignee: Neha Narkhede
>Priority: Minor
>  Labels: doc, server
> Attachments: kafka-1635-1.patch
>
>
> ReplicaManager have an incorrect java doc. The overview of function  
> makeLeaders() is the same as makeFollowers().
> Also see commit at 
> https://github.com/apache/kafka/commit/6739a8e601331ad07d9856dc351785351755a5d5



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1282) Disconnect idle socket connection in Selector

[Nicolae Marasoiu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135561#comment-14135561
 ] 

Nicolae Marasoiu commented on KAFKA-1282:
-

Hi, I have not made any patch yet, I waited for feedback from Neha too, but I 
will do the patch today, it looks ok to me the idea of closing at most one old 
connection per selector iteration.

For #1, the way Neha and me discussed, and the way you understood it works (for 
the latest patch), is that an old connection is taken into consideration for 
close only when a new connection is being opened up (or activity exists on an 
existing connection too).

> Disconnect idle socket connection in Selector
> -
>
> Key: KAFKA-1282
> URL: https://issues.apache.org/jira/browse/KAFKA-1282
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: nicu marasoiu
>  Labels: newbie++
> Fix For: 0.9.0
>
> Attachments: 
> KAFKA-1282_Disconnect_idle_socket_connection_in_Selector.patch, 
> idleDisconnect.patch
>
>
> To reduce # socket connections, it would be useful for the new producer to 
> close socket connections that are idle. We can introduce a new producer 
> config for the idle time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-1282) Disconnect idle socket connection in Selector

[Nicolae Marasoiu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135561#comment-14135561
 ] 

Nicolae Marasoiu edited comment on KAFKA-1282 at 9/16/14 3:00 PM:
--

Hi, 

I have understood what you say and I agree it is highly unintuitive and we 
should change that. I just saw you propose a solution which included a 
precomputation of the time to close, and it was bit confusion, looked like an 
attempt of micro optimization.

I have not made any patch yet, I waited for feedback from Neha too, but I will 
do the patch today: it looks ok to me the idea of closing at most one old 
connection per selector iteration.

So the solution will look more like the previous patch, but instead of 
traversing n+1 entries to close n old connections, it will just pick the oldest 
and check if it is time to close.

For #1, the way Neha and me discussed, and the way you understood it works (for 
the latest patch), is that an old connection is taken into consideration for 
close only when a new connection is being opened up (or activity exists on an 
existing connection too). But this will no longer be the case.


was (Author: nmarasoiu):
Hi, I have not made any patch yet, I waited for feedback from Neha too, but I 
will do the patch today, it looks ok to me the idea of closing at most one old 
connection per selector iteration.

For #1, the way Neha and me discussed, and the way you understood it works (for 
the latest patch), is that an old connection is taken into consideration for 
close only when a new connection is being opened up (or activity exists on an 
existing connection too).

> Disconnect idle socket connection in Selector
> -
>
> Key: KAFKA-1282
> URL: https://issues.apache.org/jira/browse/KAFKA-1282
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: nicu marasoiu
>  Labels: newbie++
> Fix For: 0.9.0
>
> Attachments: 
> KAFKA-1282_Disconnect_idle_socket_connection_in_Selector.patch, 
> idleDisconnect.patch
>
>
> To reduce # socket connections, it would be useful for the new producer to 
> close socket connections that are idle. We can introduce a new producer 
> config for the idle time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1618) Exception thrown when running console producer with no port number for the broker

[Balaji Seshadri (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135588#comment-14135588
 ] 

Balaji Seshadri commented on KAFKA-1618:


ok if user doesn't give any port i will default to 9092,if he/she does i will 
go ahead and use it.

Is that ok ?.

[~nehanarkhede] and [~gwenshap]


> Exception thrown when running console producer with no port number for the 
> broker
> -
>
> Key: KAFKA-1618
> URL: https://issues.apache.org/jira/browse/KAFKA-1618
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.1.1
>Reporter: Gwen Shapira
>Assignee: BalajiSeshadri
>  Labels: newbie
> Fix For: 0.8.2
>
> Attachments: KAFKA-1618-ALL.patch, KAFKA-1618.patch
>
>
> When running console producer with just "localhost" as the broker list, I get 
> ArrayIndexOutOfBounds exception.
> I expect either a clearer error about arguments or for the producer to 
> "guess" a default port.
> [root@shapira-1 bin]# ./kafka-console-producer.sh  --topic rufus1 
> --broker-list localhost
> java.lang.ArrayIndexOutOfBoundsException: 1
>   at 
> kafka.client.ClientUtils$$anonfun$parseBrokerList$1.apply(ClientUtils.scala:102)
>   at 
> kafka.client.ClientUtils$$anonfun$parseBrokerList$1.apply(ClientUtils.scala:97)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:105)
>   at kafka.client.ClientUtils$.parseBrokerList(ClientUtils.scala:97)
>   at 
> kafka.producer.BrokerPartitionInfo.(BrokerPartitionInfo.scala:32)
>   at 
> kafka.producer.async.DefaultEventHandler.(DefaultEventHandler.scala:41)
>   at kafka.producer.Producer.(Producer.scala:59)
>   at kafka.producer.ConsoleProducer$.main(ConsoleProducer.scala:158)
>   at kafka.producer.ConsoleProducer.main(ConsoleProducer.scala)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (KAFKA-1490) remove gradlew initial setup output from source distribution

[Ivan Lyutov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-1490 started by Ivan Lyutov.
--
> remove gradlew initial setup output from source distribution
> 
>
> Key: KAFKA-1490
> URL: https://issues.apache.org/jira/browse/KAFKA-1490
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>Assignee: Ivan Lyutov
>Priority: Blocker
> Fix For: 0.8.2
>
>
> Our current source releases contains lots of stuff in the gradle folder we do 
> not need



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 25697: Patch for KAFKA-1490

[Ivan Lyutov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25697/
---

Review request for kafka.


Bugs: KAFKA-1490
https://issues.apache.org/jira/browse/KAFKA-1490


Repository: kafka


Description
---

remove gradlew initial setup output from source distribution


Diffs
-

  gradle/wrapper/gradle-wrapper.jar a7634b071cb255e91a4572934e55b8cd8877b3e4 

Diff: https://reviews.apache.org/r/25697/diff/


Testing
---


Thanks,

Ivan Lyutov

[jira] [Updated] (KAFKA-1490) remove gradlew initial setup output from source distribution

[Ivan Lyutov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Lyutov updated KAFKA-1490:
---
Attachment: KAFKA-1490.patch

> remove gradlew initial setup output from source distribution
> 
>
> Key: KAFKA-1490
> URL: https://issues.apache.org/jira/browse/KAFKA-1490
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>Assignee: Ivan Lyutov
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: KAFKA-1490.patch
>
>
> Our current source releases contains lots of stuff in the gradle folder we do 
> not need



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1490) remove gradlew initial setup output from source distribution

[Ivan Lyutov (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135703#comment-14135703
 ] 

Ivan Lyutov commented on KAFKA-1490:


I've deleted the gradlew output jar file and left properties file for 
configuration needs.
If you want to use local gradle distribution use gradle, otherwise ./gradlew 
will download gradle with version specified in properties file

> remove gradlew initial setup output from source distribution
> 
>
> Key: KAFKA-1490
> URL: https://issues.apache.org/jira/browse/KAFKA-1490
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>Assignee: Ivan Lyutov
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: KAFKA-1490.patch
>
>
> Our current source releases contains lots of stuff in the gradle folder we do 
> not need



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Kafka Security

[Andrew Psaltis]

Hi,
I was just reading the recent changes to:
https://cwiki.apache.org/confluence/display/KAFKA/Security after getting
off a call about Kafka security and how we are jumping through hoops --
like having PGP keys on the consumers and producers to get around the lack
of SSL support. Did the meeting that Joe proposed happen for Sept 9th
happen? If not is there a plan to have it? I was also looking at:
https://issues.apache.org/jira/browse/KAFKA-1477 and it seems like there
have been no comments since 11/08/2014. I would be interested in helping
with the TLS/SSL support as we have a need for it now.

Thanks,
Andrew

Re: （info）kafka truncate data of specific topic

[Guozhang Wang]

Hi Jacky,

Could you elaborate a bit on your use cases, like why you want to manually
truncate logs? Kafka provide a set of configs for data retention based on
data size and time (for example maintaining as much as 100 GB or up to 7
days of old data), would that be sufficient to you?

Guozhang



On Mon, Sep 15, 2014 at 8:00 PM, Jacky.J.Wang (mis.cnxa01.Newegg) 43048 <
jacky.j.w...@newegg.com.invalid> wrote:

> Hello kafka
>
> I truncate data of kafka as follow
>
> 1：stop kafka service
> 2：delete zookeeper /broker/topics/topic and /consumers/group
> 3：delete kafka log dir
> 4：restart kafka service
> 5：recreate topic info
> but this way need to stop the service，so how truncate kafka topic data
> with no stopping kafka service？
>
> Eagerly awaiting your reply，thanks
>
>
> Best regards,
> Jacky.J.Wang
> Eng Software Engineer,NESC-XA
> Newegg Tech (Xian) Co., Ltd.
> 15th to 16th floor, 01 Plaza, Xi’an Software Park, No. 72 Keji 2nd Road,
> Xi’an P.R.China(710075)
> Once you know, you Newegg.
>
> -
> CONFIDENTIALITY NOTICE: This email and any files transmitted with it may
> contain privileged or otherwise confidential information.  It is intended
> only for the person or persons to whom it is addressed. If you received
> this message in error, you are not authorized to read, print, retain, copy,
> disclose, disseminate, distribute, or use this message any part thereof or
> any information contained therein. Please notify the sender immediately and
> delete all copies of this message. Thank you in advance for your
> cooperation.
>
> 保密注意：此邮件及其附随文件可能包含了保密信息。该邮件的目的是发送给指定收件人。如果您非指定收件人而错误地收到了本邮件，您将无权阅读、打印、保存、复制、泄露、传播、分发或使用此邮件全部或部分内容或者邮件中包含的任何信息。请立即通知发件人，并删除该邮件。感谢您的配合！
>
>


-- 
-- Guozhang

[jira] [Commented] (KAFKA-1490) remove gradlew initial setup output from source distribution

[Ivan Lyutov (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135760#comment-14135760
 ] 

Ivan Lyutov commented on KAFKA-1490:


Sorry guys for my previous post. I've confused ".gradle" and "gradle" 
directories.
So, my method should not work. However, I think that this issue is not related 
to Kafka project and should be resolved in Gradle instead.

Furthermore, if we eventually resolve this issue, than it could cause gradle 
version problems in future for Gradle 1.x and 2.x users, since some plugins 
work under 1.x, but not under 2.x.


> remove gradlew initial setup output from source distribution
> 
>
> Key: KAFKA-1490
> URL: https://issues.apache.org/jira/browse/KAFKA-1490
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>Assignee: Ivan Lyutov
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: KAFKA-1490.patch
>
>
> Our current source releases contains lots of stuff in the gradle folder we do 
> not need



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1490) remove gradlew initial setup output from source distribution

[Joe Stein (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135765#comment-14135765
 ] 

Joe Stein commented on KAFKA-1490:
--

it is our problem, we need to resolve this we shouldn't be shipping binary in 
source

> remove gradlew initial setup output from source distribution
> 
>
> Key: KAFKA-1490
> URL: https://issues.apache.org/jira/browse/KAFKA-1490
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>Assignee: Ivan Lyutov
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: KAFKA-1490.patch
>
>
> Our current source releases contains lots of stuff in the gradle folder we do 
> not need



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Kafka Security

[Joe Stein]

Hi Andrew, yes the meeting took place and we plan to-do it every two weeks
(same bat time, same bat channel) moving forward.

In attendance was Michael Herstine (LinkedIn), Arvind Mani (LinkedIn), Gwen
Shapira (Cloudera) and myself.

Gwen updated the wiki after our discussion.  Basically we are thinking of
using 3 ports one for plain text (so like it is now), one for SASL
(implementing username/password and kerberos at least) and one for SSL and
they will all be configurable on/off.  Some investigation is going on now
to see about how we can do this without making any wire protocol changes
(ideal) or minimal changes at least.

Let me know and I can add you to the invite if you would like to contribute
the more help and input the better for sure.

Also in regards to KAFKA-1477 I just asked Ivan to update the patch to
latest trunk and we could (demand required) make a patch that works with
0.8.1.X too for folks to use... This doesn't work yet with the new producer
(TBD) or any other clients so be aware it is not yet baked in and from
release project perspective I don't know what in that patch will survive
(hopefully all of it).

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/

On Tue, Sep 16, 2014 at 10:17 AM, Andrew Psaltis 
wrote:

> Hi,
> I was just reading the recent changes to:
> https://cwiki.apache.org/confluence/display/KAFKA/Security after getting
> off a call about Kafka security and how we are jumping through hoops --
> like having PGP keys on the consumers and producers to get around the lack
> of SSL support. Did the meeting that Joe proposed happen for Sept 9th
> happen? If not is there a plan to have it? I was also looking at:
> https://issues.apache.org/jira/browse/KAFKA-1477 and it seems like there
> have been no comments since 11/08/2014. I would be interested in helping
> with the TLS/SSL support as we have a need for it now.
>
> Thanks,
> Andrew
>

[jira] [Commented] (KAFKA-1558) AdminUtils.deleteTopic does not work

[Sriharsha Chintalapani (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135811#comment-14135811
 ] 

Sriharsha Chintalapani commented on KAFKA-1558:
---

Thanks [~junrao]
Case 2 works fine too the topic gets deleted.

Case 6 when I do kill -9 on the controller and immediately run delete topic 
from the same node or on different node
one of the brokers goes on printing these messages

[2014-09-16 16:38:17,607] INFO Reconnect due to socket error: 
java.nio.channels.ClosedChannelException (kafka.consumer.SimpleConsumer)
[2014-09-16 16:38:17,608] WARN [ReplicaFetcherThread-0-1], Error in fetch Name: 
FetchRequest; Version: 0; CorrelationId: 955460; ClientId: ReplicaFetcherThre
ad-0-1; ReplicaId: 2; MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo: 
[my-topic,1] -> PartitionFetchInfo(2558522,1048576). Possible cause: 
java.nio.channels
.ClosedChannelException (kafka.server.ReplicaFetcherThread)

I've to restart this broker and delete topic goes fine.
But if I wait after killing(kill -9) controller and fetcher is removed delete 
topic goes fine without any issues
[ReplicaFetcherManager on broker 3] Removed fetcher for partitions [my-topic,1] 
(kafka.server.ReplicaFetcherManager)
[2014-09-16 16:56:58,646] INFO [ReplicaFetcherManager on broker 3] Removed 
fetcher for partitions [my-topic,0] (kafka.server.ReplicaFetcherManager)

I am not sure if its related to delete topic though any ideas on this.



> AdminUtils.deleteTopic does not work
> 
>
> Key: KAFKA-1558
> URL: https://issues.apache.org/jira/browse/KAFKA-1558
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
>Reporter: Henning Schmiedehausen
>Assignee: Sriharsha Chintalapani
>Priority: Blocker
> Fix For: 0.8.2
>
>
> the AdminUtils:.deleteTopic method is implemented as
> {code}
> def deleteTopic(zkClient: ZkClient, topic: String) {
> ZkUtils.createPersistentPath(zkClient, 
> ZkUtils.getDeleteTopicPath(topic))
> }
> {code}
> but the DeleteTopicCommand actually does
> {code}
> zkClient = new ZkClient(zkConnect, 3, 3, ZKStringSerializer)
> zkClient.deleteRecursive(ZkUtils.getTopicPath(topic))
> {code}
> so I guess, that the 'createPersistentPath' above should actually be 
> {code}
> def deleteTopic(zkClient: ZkClient, topic: String) {
> ZkUtils.deletePathRecursive(zkClient, ZkUtils.getTopicPath(topic))
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Kafka Security

[Harsha]

Hi Joe,
I am interested in joining the efforts. I went through apache
storm security with kerberos so I can bring some of that
experience into the discussion. 
Thanks,
Harsha

On Tue, Sep 16, 2014, at 10:37 AM, Joe Stein wrote:
> Hi Andrew, yes the meeting took place and we plan to-do it every two
> weeks
> (same bat time, same bat channel) moving forward.
> 
> In attendance was Michael Herstine (LinkedIn), Arvind Mani (LinkedIn),
> Gwen
> Shapira (Cloudera) and myself.
> 
> Gwen updated the wiki after our discussion.  Basically we are thinking of
> using 3 ports one for plain text (so like it is now), one for SASL
> (implementing username/password and kerberos at least) and one for SSL
> and
> they will all be configurable on/off.  Some investigation is going on now
> to see about how we can do this without making any wire protocol changes
> (ideal) or minimal changes at least.
> 
> Let me know and I can add you to the invite if you would like to
> contribute
> the more help and input the better for sure.
> 
> Also in regards to KAFKA-1477 I just asked Ivan to update the patch to
> latest trunk and we could (demand required) make a patch that works with
> 0.8.1.X too for folks to use... This doesn't work yet with the new
> producer
> (TBD) or any other clients so be aware it is not yet baked in and from
> release project perspective I don't know what in that patch will survive
> (hopefully all of it).
> 
> /***
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop 
> /
> 
> On Tue, Sep 16, 2014 at 10:17 AM, Andrew Psaltis
> 
> wrote:
> 
> > Hi,
> > I was just reading the recent changes to:
> > https://cwiki.apache.org/confluence/display/KAFKA/Security after getting
> > off a call about Kafka security and how we are jumping through hoops --
> > like having PGP keys on the consumers and producers to get around the lack
> > of SSL support. Did the meeting that Joe proposed happen for Sept 9th
> > happen? If not is there a plan to have it? I was also looking at:
> > https://issues.apache.org/jira/browse/KAFKA-1477 and it seems like there
> > have been no comments since 11/08/2014. I would be interested in helping
> > with the TLS/SSL support as we have a need for it now.
> >
> > Thanks,
> > Andrew
> >

Re: Kafka Security

[Joe Stein]

cool, I just added you to the invite

On Tue, Sep 16, 2014 at 10:57 AM, Harsha  wrote:

> Hi Joe,
> I am interested in joining the efforts. I went through apache
> storm security with kerberos so I can bring some of that
> experience into the discussion.
> Thanks,
> Harsha
>
> On Tue, Sep 16, 2014, at 10:37 AM, Joe Stein wrote:
> > Hi Andrew, yes the meeting took place and we plan to-do it every two
> > weeks
> > (same bat time, same bat channel) moving forward.
> >
> > In attendance was Michael Herstine (LinkedIn), Arvind Mani (LinkedIn),
> > Gwen
> > Shapira (Cloudera) and myself.
> >
> > Gwen updated the wiki after our discussion.  Basically we are thinking of
> > using 3 ports one for plain text (so like it is now), one for SASL
> > (implementing username/password and kerberos at least) and one for SSL
> > and
> > they will all be configurable on/off.  Some investigation is going on now
> > to see about how we can do this without making any wire protocol changes
> > (ideal) or minimal changes at least.
> >
> > Let me know and I can add you to the invite if you would like to
> > contribute
> > the more help and input the better for sure.
> >
> > Also in regards to KAFKA-1477 I just asked Ivan to update the patch to
> > latest trunk and we could (demand required) make a patch that works with
> > 0.8.1.X too for folks to use... This doesn't work yet with the new
> > producer
> > (TBD) or any other clients so be aware it is not yet baked in and from
> > release project perspective I don't know what in that patch will survive
> > (hopefully all of it).
> >
> > /***
> >  Joe Stein
> >  Founder, Principal Consultant
> >  Big Data Open Source Security LLC
> >  http://www.stealth.ly
> >  Twitter: @allthingshadoop 
> > /
> >
> > On Tue, Sep 16, 2014 at 10:17 AM, Andrew Psaltis
> > 
> > wrote:
> >
> > > Hi,
> > > I was just reading the recent changes to:
> > > https://cwiki.apache.org/confluence/display/KAFKA/Security after
> getting
> > > off a call about Kafka security and how we are jumping through hoops --
> > > like having PGP keys on the consumers and producers to get around the
> lack
> > > of SSL support. Did the meeting that Joe proposed happen for Sept 9th
> > > happen? If not is there a plan to have it? I was also looking at:
> > > https://issues.apache.org/jira/browse/KAFKA-1477 and it seems like
> there
> > > have been no comments since 11/08/2014. I would be interested in
> helping
> > > with the TLS/SSL support as we have a need for it now.
> > >
> > > Thanks,
> > > Andrew
> > >
>

Re: Kafka Security

[Gwen Shapira]

btw. I think we barely mention support of delegation tokens to allow
accessing Kafka from MR jobs, Storm, Samza, etc.

Does it sound "in scope" for next week's agenda?

Gwen

On Tue, Sep 16, 2014 at 10:59 AM, Joe Stein  wrote:
> cool, I just added you to the invite
>
> On Tue, Sep 16, 2014 at 10:57 AM, Harsha  wrote:
>
>> Hi Joe,
>> I am interested in joining the efforts. I went through apache
>> storm security with kerberos so I can bring some of that
>> experience into the discussion.
>> Thanks,
>> Harsha
>>
>> On Tue, Sep 16, 2014, at 10:37 AM, Joe Stein wrote:
>> > Hi Andrew, yes the meeting took place and we plan to-do it every two
>> > weeks
>> > (same bat time, same bat channel) moving forward.
>> >
>> > In attendance was Michael Herstine (LinkedIn), Arvind Mani (LinkedIn),
>> > Gwen
>> > Shapira (Cloudera) and myself.
>> >
>> > Gwen updated the wiki after our discussion.  Basically we are thinking of
>> > using 3 ports one for plain text (so like it is now), one for SASL
>> > (implementing username/password and kerberos at least) and one for SSL
>> > and
>> > they will all be configurable on/off.  Some investigation is going on now
>> > to see about how we can do this without making any wire protocol changes
>> > (ideal) or minimal changes at least.
>> >
>> > Let me know and I can add you to the invite if you would like to
>> > contribute
>> > the more help and input the better for sure.
>> >
>> > Also in regards to KAFKA-1477 I just asked Ivan to update the patch to
>> > latest trunk and we could (demand required) make a patch that works with
>> > 0.8.1.X too for folks to use... This doesn't work yet with the new
>> > producer
>> > (TBD) or any other clients so be aware it is not yet baked in and from
>> > release project perspective I don't know what in that patch will survive
>> > (hopefully all of it).
>> >
>> > /***
>> >  Joe Stein
>> >  Founder, Principal Consultant
>> >  Big Data Open Source Security LLC
>> >  http://www.stealth.ly
>> >  Twitter: @allthingshadoop 
>> > /
>> >
>> > On Tue, Sep 16, 2014 at 10:17 AM, Andrew Psaltis
>> > 
>> > wrote:
>> >
>> > > Hi,
>> > > I was just reading the recent changes to:
>> > > https://cwiki.apache.org/confluence/display/KAFKA/Security after
>> getting
>> > > off a call about Kafka security and how we are jumping through hoops --
>> > > like having PGP keys on the consumers and producers to get around the
>> lack
>> > > of SSL support. Did the meeting that Joe proposed happen for Sept 9th
>> > > happen? If not is there a plan to have it? I was also looking at:
>> > > https://issues.apache.org/jira/browse/KAFKA-1477 and it seems like
>> there
>> > > have been no comments since 11/08/2014. I would be interested in
>> helping
>> > > with the TLS/SSL support as we have a need for it now.
>> > >
>> > > Thanks,
>> > > Andrew
>> > >
>>

Re: [DISCUSS] 0.8.2 release branch, "unofficial" release candidates(s), 0.8.1.2 release

[Harsha]

Hi All,
   Do we have any ballpark date on the release of 0.8.1.2 or 0.8.2.
Thanks,
Harsha


On Thu, Sep 11, 2014, at 03:53 PM, Jay Kreps wrote:
> I agree that a beta for 0.8.2 would be useful. It would also be good
> to get it in production at LinkedIn before the final version.
> 
> Sorry about stalling on the security stuff. Things have been a little
> busy on our side. Let's get 0.8.2 out and then get that broken up into
> individual JIRAs that people can take on.
> 
> -Jay
> 
> On Wed, Sep 10, 2014 at 5:11 PM, Jun Rao  wrote:
> > Joe,
> >
> > Thanks for starting the discussion.
> >
> > (1) I made a pass of the open jiras for 0.8.2 and marked a few of them as
> > blockers for now. There are currently 6 blockers. Ideally, we want to get
> > all those fixed before cutting the 0.8.2 branch. The rest of the jiras
> > don't really have to be fixed in 0.8.2. So, if anyone wants to help on
> > fixing those blocker jiras, that would be great. Perhaps we can circle back
> > in a couple of weeks and see how much progress we make on those blocker
> > jiras.
> >
> > (2) A beta 0.8.2 may not be a bad idea.
> >
> > (3) We can do 0.8.1.2. However, I'd prefer only trivial and critical
> > patches to back port. The scala 2.11 patch seems ok.
> >
> > (4) Yes, we should start updating the wiki once 0.8.2 is cut.
> >
> > (5) Yes, we can include kafka-1555 if it can be fixed in time.
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> > On Wed, Sep 3, 2014 at 6:34 PM, Joe Stein  wrote:
> >
> >> Hey, I wanted to take a quick pulse to see if we are getting closer to a
> >> branch for 0.8.2.
> >>
> >> 1) There still seems to be a lot of open issues
> >>
> >> https://issues.apache.org/jira/browse/KAFKA/fixforversion/12326167/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-issues-panel
> >> and our 30 day summary is showing issues: 51 created and *34* resolved and
> >> not
> >> sure how much of that we could really just decide to push off to 0.8.3 or
> >> 0.9.0 vs working on 0.8.2 as stable for release.  There is already so much
> >> goodness on trunk.  I appreciate the double commit pain especially as trunk
> >> and branch drift (ugh).
> >>
> >> 2) Also, I wanted to float the idea of after making the 0.8.2 branch that I
> >> would do some unofficial release candidates for folks to test prior to
> >> release and vote.  What I was thinking was I would build, upload and stage
> >> like I was preparing artifacts for vote but let the community know to go in
> >> and "have at it" well prior to the vote release.  We don't get a lot of
> >> community votes during a release but issues after (which is natural because
> >> of how things are done).  I have seen four Apache projects doing this very
> >> successfully not only have they had less iterations of RC votes (sensitive
> >> to that myself) but the community kicked back issues they saw by giving
> >> them some "pre release" time to go through their own test and staging
> >> environments as the release are coming about.
> >>
> >> 3) Checking again on "should we have a 0.8.1.2" release if folks in the
> >> community find important features (this might be best asked on the user
> >> list maybe not sure) they don't want/can't wait for which wouldn't be too
> >> much pain/dangerous to back port. Two things that spring to the top of my
> >> head are 2.11 Scala support and fixing the source jars.  Both of these are
> >> easy to patch personally I don't mind but want to gauge more from the
> >> community on this too.  I have heard gripes ad hoc from folks in direct
> >> communication but no complains really in the public forum and wanted to
> >> open the floor if folks had a need.
> >>
> >> 4) 0.9 work I feel is being held up some (or at least resourcing it from my
> >> perspective).  We decided to hold up including SSL (even though we have a
> >> path for it). Jay did a nice update recently to the Security wiki which I
> >> think we should move forward with.  I have some more to add/change/update
> >> and want to start getting down to more details and getting specific people
> >> working on specific tasks but without knowing what we are doing when it is
> >> hard to manage.
> >>
> >> 5) I just updated https://issues.apache.org/jira/browse/KAFKA-1555 I think
> >> it is a really important feature update doesn't have to be in 0.8.2 but we
> >> need consensus (no pun intended). It fundamentally allows for data in min
> >> two rack requirement which A LOT of data requires for successful save to
> >> occur.
> >>
> >> /***
> >>  Joe Stein
> >>  Founder, Principal Consultant
> >>  Big Data Open Source Security LLC
> >>  http://www.stealth.ly
> >>  Twitter: @allthingshadoop 
> >> /
> >>

Review Request 25703: Patch for KAFKA-1490

[Ivan Lyutov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25703/
---

Review request for kafka.


Bugs: KAFKA-1490
https://issues.apache.org/jira/browse/KAFKA-1490


Repository: kafka


Description
---

Removed wrapper generated contents and added default task to generate gradle 
wrapper binary.


Diffs
-

  build.gradle 74c8c8a9e2f4d9a651181d5337d5a8f07f0cb313 
  gradle/wrapper/gradle-wrapper.jar a7634b071cb255e91a4572934e55b8cd8877b3e4 
  gradle/wrapper/gradle-wrapper.properties 
d238df326fec6d925ccecdecaac0bcedf8e68672 
  wrapper.gradle PRE-CREATION 

Diff: https://reviews.apache.org/r/25703/diff/


Testing
---


Thanks,

Ivan Lyutov

[jira] [Updated] (KAFKA-1490) remove gradlew initial setup output from source distribution

[Ivan Lyutov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Lyutov updated KAFKA-1490:
---
Attachment: KAFKA-1490-2.patch

Ok, I applied the patch which replays the functionality from Samza.

> remove gradlew initial setup output from source distribution
> 
>
> Key: KAFKA-1490
> URL: https://issues.apache.org/jira/browse/KAFKA-1490
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>Assignee: Ivan Lyutov
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: KAFKA-1490-2.patch, KAFKA-1490.patch
>
>
> Our current source releases contains lots of stuff in the gradle folder we do 
> not need



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-1636) High CPU in very active environment

[Laurie Turner (JIRA)]

Laurie Turner created KAFKA-1636:


 Summary: High CPU in very active environment
 Key: KAFKA-1636
 URL: https://issues.apache.org/jira/browse/KAFKA-1636
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 0.8.0
 Environment: Redhat 64 bit
 2.6.32-431.23.3.el6.x86_64 #1 SMP Wed Jul 16 06:12:23 EDT 2014 x86_64 x86_64 
x86_64 GNU/Linux

Reporter: Laurie Turner
Assignee: Neha Narkhede


Found the same issue on StackOverFlow below:

http://stackoverflow.com/questions/22983435/kafka-consumer-threads-are-in-waiting-state-and-lag-is-getting-increased

This is a very busy environment and the majority of the CPU seems to be busy in 
the in the await method. 


3XMTHREADINFO3   Java callstack:
4XESTACKTRACEat sun/misc/Unsafe.park(Native Method)
4XESTACKTRACEat 
java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:237(Compiled 
Code))
4XESTACKTRACEat 
java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2093(Compiled
 Code))
4XESTACKTRACEat 
java/util/concurrent/LinkedBlockingQueue.poll(LinkedBlockingQueue.java:478(Compiled
 Code))
4XESTACKTRACEat 
kafka/consumer/ConsumerIterator.makeNext(ConsumerIterator.scala:65(Compiled 
Code))
4XESTACKTRACEat 
kafka/consumer/ConsumerIterator.makeNext(ConsumerIterator.scala:33(Compiled 
Code))
4XESTACKTRACEat 
kafka/utils/IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:61(Compiled
 Code))
4XESTACKTRACEat 
kafka/utils/IteratorTemplate.hasNext(IteratorTemplate.scala:53(Compiled Code))



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1490) remove gradlew initial setup output from source distribution

[Ivan Lyutov (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135928#comment-14135928
 ] 

Ivan Lyutov commented on KAFKA-1490:


https://reviews.apache.org/r/25697/

> remove gradlew initial setup output from source distribution
> 
>
> Key: KAFKA-1490
> URL: https://issues.apache.org/jira/browse/KAFKA-1490
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>Assignee: Ivan Lyutov
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: KAFKA-1490-2.patch, KAFKA-1490.patch
>
>
> Our current source releases contains lots of stuff in the gradle folder we do 
> not need



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-1490) remove gradlew initial setup output from source distribution

[Ivan Lyutov (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135928#comment-14135928
 ] 

Ivan Lyutov edited comment on KAFKA-1490 at 9/16/14 6:41 PM:
-

https://reviews.apache.org/r/25703/


was (Author: edgefox):
https://reviews.apache.org/r/25697/

> remove gradlew initial setup output from source distribution
> 
>
> Key: KAFKA-1490
> URL: https://issues.apache.org/jira/browse/KAFKA-1490
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>Assignee: Ivan Lyutov
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: KAFKA-1490-2.patch, KAFKA-1490.patch
>
>
> Our current source releases contains lots of stuff in the gradle folder we do 
> not need



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Kafka Security

[Todd Palino]

Joe, can you add me to it as well? We obviously have some interest in the
discussion over here :)

-Todd

On 9/16/14, 10:59 AM, "Joe Stein"  wrote:

>cool, I just added you to the invite
>
>On Tue, Sep 16, 2014 at 10:57 AM, Harsha  wrote:
>
>> Hi Joe,
>> I am interested in joining the efforts. I went through apache
>> storm security with kerberos so I can bring some of that
>> experience into the discussion.
>> Thanks,
>> Harsha
>>
>> On Tue, Sep 16, 2014, at 10:37 AM, Joe Stein wrote:
>> > Hi Andrew, yes the meeting took place and we plan to-do it every two
>> > weeks
>> > (same bat time, same bat channel) moving forward.
>> >
>> > In attendance was Michael Herstine (LinkedIn), Arvind Mani (LinkedIn),
>> > Gwen
>> > Shapira (Cloudera) and myself.
>> >
>> > Gwen updated the wiki after our discussion.  Basically we are
>>thinking of
>> > using 3 ports one for plain text (so like it is now), one for SASL
>> > (implementing username/password and kerberos at least) and one for SSL
>> > and
>> > they will all be configurable on/off.  Some investigation is going on
>>now
>> > to see about how we can do this without making any wire protocol
>>changes
>> > (ideal) or minimal changes at least.
>> >
>> > Let me know and I can add you to the invite if you would like to
>> > contribute
>> > the more help and input the better for sure.
>> >
>> > Also in regards to KAFKA-1477 I just asked Ivan to update the patch to
>> > latest trunk and we could (demand required) make a patch that works
>>with
>> > 0.8.1.X too for folks to use... This doesn't work yet with the new
>> > producer
>> > (TBD) or any other clients so be aware it is not yet baked in and from
>> > release project perspective I don't know what in that patch will
>>survive
>> > (hopefully all of it).
>> >
>> > /***
>> >  Joe Stein
>> >  Founder, Principal Consultant
>> >  Big Data Open Source Security LLC
>> >  http://www.stealth.ly
>> >  Twitter: @allthingshadoop 
>> > /
>> >
>> > On Tue, Sep 16, 2014 at 10:17 AM, Andrew Psaltis
>> > 
>> > wrote:
>> >
>> > > Hi,
>> > > I was just reading the recent changes to:
>> > > https://cwiki.apache.org/confluence/display/KAFKA/Security after
>> getting
>> > > off a call about Kafka security and how we are jumping through
>>hoops --
>> > > like having PGP keys on the consumers and producers to get around
>>the
>> lack
>> > > of SSL support. Did the meeting that Joe proposed happen for Sept
>>9th
>> > > happen? If not is there a plan to have it? I was also looking at:
>> > > https://issues.apache.org/jira/browse/KAFKA-1477 and it seems like
>> there
>> > > have been no comments since 11/08/2014. I would be interested in
>> helping
>> > > with the TLS/SSL support as we have a need for it now.
>> > >
>> > > Thanks,
>> > > Andrew
>> > >
>>

Re: Kafka Security

[Joe Stein]

Yup!

On Tue, Sep 16, 2014 at 11:09 AM, Gwen Shapira 
wrote:

> btw. I think we barely mention support of delegation tokens to allow
> accessing Kafka from MR jobs, Storm, Samza, etc.
>
> Does it sound "in scope" for next week's agenda?
>
> Gwen
>
> On Tue, Sep 16, 2014 at 10:59 AM, Joe Stein  wrote:
> > cool, I just added you to the invite
> >
> > On Tue, Sep 16, 2014 at 10:57 AM, Harsha  wrote:
> >
> >> Hi Joe,
> >> I am interested in joining the efforts. I went through apache
> >> storm security with kerberos so I can bring some of that
> >> experience into the discussion.
> >> Thanks,
> >> Harsha
> >>
> >> On Tue, Sep 16, 2014, at 10:37 AM, Joe Stein wrote:
> >> > Hi Andrew, yes the meeting took place and we plan to-do it every two
> >> > weeks
> >> > (same bat time, same bat channel) moving forward.
> >> >
> >> > In attendance was Michael Herstine (LinkedIn), Arvind Mani (LinkedIn),
> >> > Gwen
> >> > Shapira (Cloudera) and myself.
> >> >
> >> > Gwen updated the wiki after our discussion.  Basically we are
> thinking of
> >> > using 3 ports one for plain text (so like it is now), one for SASL
> >> > (implementing username/password and kerberos at least) and one for SSL
> >> > and
> >> > they will all be configurable on/off.  Some investigation is going on
> now
> >> > to see about how we can do this without making any wire protocol
> changes
> >> > (ideal) or minimal changes at least.
> >> >
> >> > Let me know and I can add you to the invite if you would like to
> >> > contribute
> >> > the more help and input the better for sure.
> >> >
> >> > Also in regards to KAFKA-1477 I just asked Ivan to update the patch to
> >> > latest trunk and we could (demand required) make a patch that works
> with
> >> > 0.8.1.X too for folks to use... This doesn't work yet with the new
> >> > producer
> >> > (TBD) or any other clients so be aware it is not yet baked in and from
> >> > release project perspective I don't know what in that patch will
> survive
> >> > (hopefully all of it).
> >> >
> >> > /***
> >> >  Joe Stein
> >> >  Founder, Principal Consultant
> >> >  Big Data Open Source Security LLC
> >> >  http://www.stealth.ly
> >> >  Twitter: @allthingshadoop 
> >> > /
> >> >
> >> > On Tue, Sep 16, 2014 at 10:17 AM, Andrew Psaltis
> >> > 
> >> > wrote:
> >> >
> >> > > Hi,
> >> > > I was just reading the recent changes to:
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/Security after
> >> getting
> >> > > off a call about Kafka security and how we are jumping through
> hoops --
> >> > > like having PGP keys on the consumers and producers to get around
> the
> >> lack
> >> > > of SSL support. Did the meeting that Joe proposed happen for Sept
> 9th
> >> > > happen? If not is there a plan to have it? I was also looking at:
> >> > > https://issues.apache.org/jira/browse/KAFKA-1477 and it seems like
> >> there
> >> > > have been no comments since 11/08/2014. I would be interested in
> >> helping
> >> > > with the TLS/SSL support as we have a need for it now.
> >> > >
> >> > > Thanks,
> >> > > Andrew
> >> > >
> >>
>

Re: Kafka Security

[Joe Stein]

=8^)

done

On Tue, Sep 16, 2014 at 11:43 AM, Todd Palino 
wrote:

> Joe, can you add me to it as well? We obviously have some interest in the
> discussion over here :)
>
> -Todd
>
> On 9/16/14, 10:59 AM, "Joe Stein"  wrote:
>
> >cool, I just added you to the invite
> >
> >On Tue, Sep 16, 2014 at 10:57 AM, Harsha  wrote:
> >
> >> Hi Joe,
> >> I am interested in joining the efforts. I went through apache
> >> storm security with kerberos so I can bring some of that
> >> experience into the discussion.
> >> Thanks,
> >> Harsha
> >>
> >> On Tue, Sep 16, 2014, at 10:37 AM, Joe Stein wrote:
> >> > Hi Andrew, yes the meeting took place and we plan to-do it every two
> >> > weeks
> >> > (same bat time, same bat channel) moving forward.
> >> >
> >> > In attendance was Michael Herstine (LinkedIn), Arvind Mani (LinkedIn),
> >> > Gwen
> >> > Shapira (Cloudera) and myself.
> >> >
> >> > Gwen updated the wiki after our discussion.  Basically we are
> >>thinking of
> >> > using 3 ports one for plain text (so like it is now), one for SASL
> >> > (implementing username/password and kerberos at least) and one for SSL
> >> > and
> >> > they will all be configurable on/off.  Some investigation is going on
> >>now
> >> > to see about how we can do this without making any wire protocol
> >>changes
> >> > (ideal) or minimal changes at least.
> >> >
> >> > Let me know and I can add you to the invite if you would like to
> >> > contribute
> >> > the more help and input the better for sure.
> >> >
> >> > Also in regards to KAFKA-1477 I just asked Ivan to update the patch to
> >> > latest trunk and we could (demand required) make a patch that works
> >>with
> >> > 0.8.1.X too for folks to use... This doesn't work yet with the new
> >> > producer
> >> > (TBD) or any other clients so be aware it is not yet baked in and from
> >> > release project perspective I don't know what in that patch will
> >>survive
> >> > (hopefully all of it).
> >> >
> >> > /***
> >> >  Joe Stein
> >> >  Founder, Principal Consultant
> >> >  Big Data Open Source Security LLC
> >> >  http://www.stealth.ly
> >> >  Twitter: @allthingshadoop 
> >> > /
> >> >
> >> > On Tue, Sep 16, 2014 at 10:17 AM, Andrew Psaltis
> >> > 
> >> > wrote:
> >> >
> >> > > Hi,
> >> > > I was just reading the recent changes to:
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/Security after
> >> getting
> >> > > off a call about Kafka security and how we are jumping through
> >>hoops --
> >> > > like having PGP keys on the consumers and producers to get around
> >>the
> >> lack
> >> > > of SSL support. Did the meeting that Joe proposed happen for Sept
> >>9th
> >> > > happen? If not is there a plan to have it? I was also looking at:
> >> > > https://issues.apache.org/jira/browse/KAFKA-1477 and it seems like
> >> there
> >> > > have been no comments since 11/08/2014. I would be interested in
> >> helping
> >> > > with the TLS/SSL support as we have a need for it now.
> >> > >
> >> > > Thanks,
> >> > > Andrew
> >> > >
> >>
>
>

Re: Kafka Security

[Joel Koshy]

Same here.

Thanks,

Joel

On Tue, Sep 16, 2014 at 06:43:22PM +, Todd Palino wrote:
> Joe, can you add me to it as well? We obviously have some interest in the
> discussion over here :)
> 
> -Todd
> 
> On 9/16/14, 10:59 AM, "Joe Stein"  wrote:
> 
> >cool, I just added you to the invite
> >
> >On Tue, Sep 16, 2014 at 10:57 AM, Harsha  wrote:
> >
> >> Hi Joe,
> >> I am interested in joining the efforts. I went through apache
> >> storm security with kerberos so I can bring some of that
> >> experience into the discussion.
> >> Thanks,
> >> Harsha
> >>
> >> On Tue, Sep 16, 2014, at 10:37 AM, Joe Stein wrote:
> >> > Hi Andrew, yes the meeting took place and we plan to-do it every two
> >> > weeks
> >> > (same bat time, same bat channel) moving forward.
> >> >
> >> > In attendance was Michael Herstine (LinkedIn), Arvind Mani (LinkedIn),
> >> > Gwen
> >> > Shapira (Cloudera) and myself.
> >> >
> >> > Gwen updated the wiki after our discussion.  Basically we are
> >>thinking of
> >> > using 3 ports one for plain text (so like it is now), one for SASL
> >> > (implementing username/password and kerberos at least) and one for SSL
> >> > and
> >> > they will all be configurable on/off.  Some investigation is going on
> >>now
> >> > to see about how we can do this without making any wire protocol
> >>changes
> >> > (ideal) or minimal changes at least.
> >> >
> >> > Let me know and I can add you to the invite if you would like to
> >> > contribute
> >> > the more help and input the better for sure.
> >> >
> >> > Also in regards to KAFKA-1477 I just asked Ivan to update the patch to
> >> > latest trunk and we could (demand required) make a patch that works
> >>with
> >> > 0.8.1.X too for folks to use... This doesn't work yet with the new
> >> > producer
> >> > (TBD) or any other clients so be aware it is not yet baked in and from
> >> > release project perspective I don't know what in that patch will
> >>survive
> >> > (hopefully all of it).
> >> >
> >> > /***
> >> >  Joe Stein
> >> >  Founder, Principal Consultant
> >> >  Big Data Open Source Security LLC
> >> >  http://www.stealth.ly
> >> >  Twitter: @allthingshadoop 
> >> > /
> >> >
> >> > On Tue, Sep 16, 2014 at 10:17 AM, Andrew Psaltis
> >> > 
> >> > wrote:
> >> >
> >> > > Hi,
> >> > > I was just reading the recent changes to:
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/Security after
> >> getting
> >> > > off a call about Kafka security and how we are jumping through
> >>hoops --
> >> > > like having PGP keys on the consumers and producers to get around
> >>the
> >> lack
> >> > > of SSL support. Did the meeting that Joe proposed happen for Sept
> >>9th
> >> > > happen? If not is there a plan to have it? I was also looking at:
> >> > > https://issues.apache.org/jira/browse/KAFKA-1477 and it seems like
> >> there
> >> > > have been no comments since 11/08/2014. I would be interested in
> >> helping
> >> > > with the TLS/SSL support as we have a need for it now.
> >> > >
> >> > > Thanks,
> >> > > Andrew
> >> > >
> >>
> 

[jira] [Commented] (KAFKA-1499) Broker-side compression configuration

[Joel Koshy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135957#comment-14135957
 ] 

Joel Koshy commented on KAFKA-1499:
---

Sorry you are right - I mixed up the two. For some reason I thought this was 
reviewed and checked-in. My bad - will review this and get back to you.

> Broker-side compression configuration
> -
>
> Key: KAFKA-1499
> URL: https://issues.apache.org/jira/browse/KAFKA-1499
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Joel Koshy
>Assignee: Manikumar Reddy
>  Labels: newbie++
> Fix For: 0.8.2
>
> Attachments: KAFKA-1499.patch, KAFKA-1499.patch, 
> KAFKA-1499_2014-08-15_14:20:27.patch, KAFKA-1499_2014-08-21_21:44:27.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> A given topic can have messages in mixed compression codecs. i.e., it can
> also have a mix of uncompressed/compressed messages.
> It will be useful to support a broker-side configuration to recompress
> messages to a specific compression codec. i.e., all messages (for all
> topics) on the broker will be compressed to this codec. We could have
> per-topic overrides as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Kafka Security

[Andrew Psaltis]

Hi Joe,
I am certainly interested. Can you please add me to the invite.

Thanks,
Andrew

On Tue, Sep 16, 2014 at 11:37 AM, Joe Stein  wrote:

> Hi Andrew, yes the meeting took place and we plan to-do it every two weeks
> (same bat time, same bat channel) moving forward.
>
> In attendance was Michael Herstine (LinkedIn), Arvind Mani (LinkedIn), Gwen
> Shapira (Cloudera) and myself.
>
> Gwen updated the wiki after our discussion.  Basically we are thinking of
> using 3 ports one for plain text (so like it is now), one for SASL
> (implementing username/password and kerberos at least) and one for SSL and
> they will all be configurable on/off.  Some investigation is going on now
> to see about how we can do this without making any wire protocol changes
> (ideal) or minimal changes at least.
>
> Let me know and I can add you to the invite if you would like to contribute
> the more help and input the better for sure.
>
> Also in regards to KAFKA-1477 I just asked Ivan to update the patch to
> latest trunk and we could (demand required) make a patch that works with
> 0.8.1.X too for folks to use... This doesn't work yet with the new producer
> (TBD) or any other clients so be aware it is not yet baked in and from
> release project perspective I don't know what in that patch will survive
> (hopefully all of it).
>
> /***
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop 
> /
>
> On Tue, Sep 16, 2014 at 10:17 AM, Andrew Psaltis  >
> wrote:
>
> > Hi,
> > I was just reading the recent changes to:
> > https://cwiki.apache.org/confluence/display/KAFKA/Security after getting
> > off a call about Kafka security and how we are jumping through hoops --
> > like having PGP keys on the consumers and producers to get around the
> lack
> > of SSL support. Did the meeting that Joe proposed happen for Sept 9th
> > happen? If not is there a plan to have it? I was also looking at:
> > https://issues.apache.org/jira/browse/KAFKA-1477 and it seems like there
> > have been no comments since 11/08/2014. I would be interested in helping
> > with the TLS/SSL support as we have a need for it now.
> >
> > Thanks,
> > Andrew
> >
>

Re: Kafka Security

[Joe Stein]

done

On Tue, Sep 16, 2014 at 11:46 AM, Joel Koshy  wrote:

> Same here.
>
> Thanks,
>
> Joel
>
> On Tue, Sep 16, 2014 at 06:43:22PM +, Todd Palino wrote:
> > Joe, can you add me to it as well? We obviously have some interest in the
> > discussion over here :)
> >
> > -Todd
> >
> > On 9/16/14, 10:59 AM, "Joe Stein"  wrote:
> >
> > >cool, I just added you to the invite
> > >
> > >On Tue, Sep 16, 2014 at 10:57 AM, Harsha  wrote:
> > >
> > >> Hi Joe,
> > >> I am interested in joining the efforts. I went through apache
> > >> storm security with kerberos so I can bring some of that
> > >> experience into the discussion.
> > >> Thanks,
> > >> Harsha
> > >>
> > >> On Tue, Sep 16, 2014, at 10:37 AM, Joe Stein wrote:
> > >> > Hi Andrew, yes the meeting took place and we plan to-do it every two
> > >> > weeks
> > >> > (same bat time, same bat channel) moving forward.
> > >> >
> > >> > In attendance was Michael Herstine (LinkedIn), Arvind Mani
> (LinkedIn),
> > >> > Gwen
> > >> > Shapira (Cloudera) and myself.
> > >> >
> > >> > Gwen updated the wiki after our discussion.  Basically we are
> > >>thinking of
> > >> > using 3 ports one for plain text (so like it is now), one for SASL
> > >> > (implementing username/password and kerberos at least) and one for
> SSL
> > >> > and
> > >> > they will all be configurable on/off.  Some investigation is going
> on
> > >>now
> > >> > to see about how we can do this without making any wire protocol
> > >>changes
> > >> > (ideal) or minimal changes at least.
> > >> >
> > >> > Let me know and I can add you to the invite if you would like to
> > >> > contribute
> > >> > the more help and input the better for sure.
> > >> >
> > >> > Also in regards to KAFKA-1477 I just asked Ivan to update the patch
> to
> > >> > latest trunk and we could (demand required) make a patch that works
> > >>with
> > >> > 0.8.1.X too for folks to use... This doesn't work yet with the new
> > >> > producer
> > >> > (TBD) or any other clients so be aware it is not yet baked in and
> from
> > >> > release project perspective I don't know what in that patch will
> > >>survive
> > >> > (hopefully all of it).
> > >> >
> > >> > /***
> > >> >  Joe Stein
> > >> >  Founder, Principal Consultant
> > >> >  Big Data Open Source Security LLC
> > >> >  http://www.stealth.ly
> > >> >  Twitter: @allthingshadoop 
> > >> > /
> > >> >
> > >> > On Tue, Sep 16, 2014 at 10:17 AM, Andrew Psaltis
> > >> > 
> > >> > wrote:
> > >> >
> > >> > > Hi,
> > >> > > I was just reading the recent changes to:
> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/Security after
> > >> getting
> > >> > > off a call about Kafka security and how we are jumping through
> > >>hoops --
> > >> > > like having PGP keys on the consumers and producers to get around
> > >>the
> > >> lack
> > >> > > of SSL support. Did the meeting that Joe proposed happen for Sept
> > >>9th
> > >> > > happen? If not is there a plan to have it? I was also looking at:
> > >> > > https://issues.apache.org/jira/browse/KAFKA-1477 and it seems
> like
> > >> there
> > >> > > have been no comments since 11/08/2014. I would be interested in
> > >> helping
> > >> > > with the TLS/SSL support as we have a need for it now.
> > >> > >
> > >> > > Thanks,
> > >> > > Andrew
> > >> > >
> > >>
> >
>
>

Re: Kafka Security

[Joe Stein]

done

On Tue, Sep 16, 2014 at 11:48 AM, Andrew Psaltis 
wrote:

> Hi Joe,
> I am certainly interested. Can you please add me to the invite.
>
> Thanks,
> Andrew
>
> On Tue, Sep 16, 2014 at 11:37 AM, Joe Stein  wrote:
>
> > Hi Andrew, yes the meeting took place and we plan to-do it every two
> weeks
> > (same bat time, same bat channel) moving forward.
> >
> > In attendance was Michael Herstine (LinkedIn), Arvind Mani (LinkedIn),
> Gwen
> > Shapira (Cloudera) and myself.
> >
> > Gwen updated the wiki after our discussion.  Basically we are thinking of
> > using 3 ports one for plain text (so like it is now), one for SASL
> > (implementing username/password and kerberos at least) and one for SSL
> and
> > they will all be configurable on/off.  Some investigation is going on now
> > to see about how we can do this without making any wire protocol changes
> > (ideal) or minimal changes at least.
> >
> > Let me know and I can add you to the invite if you would like to
> contribute
> > the more help and input the better for sure.
> >
> > Also in regards to KAFKA-1477 I just asked Ivan to update the patch to
> > latest trunk and we could (demand required) make a patch that works with
> > 0.8.1.X too for folks to use... This doesn't work yet with the new
> producer
> > (TBD) or any other clients so be aware it is not yet baked in and from
> > release project perspective I don't know what in that patch will survive
> > (hopefully all of it).
> >
> > /***
> >  Joe Stein
> >  Founder, Principal Consultant
> >  Big Data Open Source Security LLC
> >  http://www.stealth.ly
> >  Twitter: @allthingshadoop 
> > /
> >
> > On Tue, Sep 16, 2014 at 10:17 AM, Andrew Psaltis <
> psaltis.and...@gmail.com
> > >
> > wrote:
> >
> > > Hi,
> > > I was just reading the recent changes to:
> > > https://cwiki.apache.org/confluence/display/KAFKA/Security after
> getting
> > > off a call about Kafka security and how we are jumping through hoops --
> > > like having PGP keys on the consumers and producers to get around the
> > lack
> > > of SSL support. Did the meeting that Joe proposed happen for Sept 9th
> > > happen? If not is there a plan to have it? I was also looking at:
> > > https://issues.apache.org/jira/browse/KAFKA-1477 and it seems like
> there
> > > have been no comments since 11/08/2014. I would be interested in
> helping
> > > with the TLS/SSL support as we have a need for it now.
> > >
> > > Thanks,
> > > Andrew
> > >
> >
>

Re: Review Request 25703: Patch for KAFKA-1490

[Ivan Lyutov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25703/
---

(Updated Сен. 16, 2014, 6:54 п.п.)


Review request for kafka.


Bugs: KAFKA-1490
https://issues.apache.org/jira/browse/KAFKA-1490


Repository: kafka


Description (updated)
---

Removed wrapper generated contents and added default task to generate gradle 
wrapper binary.


remove gradlew initial setup output from source distribution


Diffs (updated)
-

  build.gradle 74c8c8a9e2f4d9a651181d5337d5a8f07f0cb313 
  gradle/wrapper/gradle-wrapper.jar a7634b071cb255e91a4572934e55b8cd8877b3e4 
  gradle/wrapper/gradle-wrapper.properties 
d238df326fec6d925ccecdecaac0bcedf8e68672 
  wrapper.gradle PRE-CREATION 

Diff: https://reviews.apache.org/r/25703/diff/


Testing
---


Thanks,

Ivan Lyutov

[jira] [Updated] (KAFKA-1490) remove gradlew initial setup output from source distribution

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede updated KAFKA-1490:
-
Reviewer: Joe Stein

> remove gradlew initial setup output from source distribution
> 
>
> Key: KAFKA-1490
> URL: https://issues.apache.org/jira/browse/KAFKA-1490
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>Assignee: Ivan Lyutov
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: KAFKA-1490-2.patch, KAFKA-1490.patch
>
>
> Our current source releases contains lots of stuff in the gradle folder we do 
> not need



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Kafka Security

[Steve Morin]

Same here

> On Sep 16, 2014, at 11:51, Joe Stein  wrote:
> 
> done
> 
>> On Tue, Sep 16, 2014 at 11:46 AM, Joel Koshy  wrote:
>> 
>> Same here.
>> 
>> Thanks,
>> 
>> Joel
>> 
>>> On Tue, Sep 16, 2014 at 06:43:22PM +, Todd Palino wrote:
>>> Joe, can you add me to it as well? We obviously have some interest in the
>>> discussion over here :)
>>> 
>>> -Todd
>>> 
 On 9/16/14, 10:59 AM, "Joe Stein"  wrote:
 
 cool, I just added you to the invite
 
> On Tue, Sep 16, 2014 at 10:57 AM, Harsha  wrote:
> 
> Hi Joe,
>I am interested in joining the efforts. I went through apache
>storm security with kerberos so I can bring some of that
>experience into the discussion.
> Thanks,
> Harsha
> 
>> On Tue, Sep 16, 2014, at 10:37 AM, Joe Stein wrote:
>> Hi Andrew, yes the meeting took place and we plan to-do it every two
>> weeks
>> (same bat time, same bat channel) moving forward.
>> 
>> In attendance was Michael Herstine (LinkedIn), Arvind Mani
>> (LinkedIn),
>> Gwen
>> Shapira (Cloudera) and myself.
>> 
>> Gwen updated the wiki after our discussion.  Basically we are
> thinking of
>> using 3 ports one for plain text (so like it is now), one for SASL
>> (implementing username/password and kerberos at least) and one for
>> SSL
>> and
>> they will all be configurable on/off.  Some investigation is going
>> on
> now
>> to see about how we can do this without making any wire protocol
> changes
>> (ideal) or minimal changes at least.
>> 
>> Let me know and I can add you to the invite if you would like to
>> contribute
>> the more help and input the better for sure.
>> 
>> Also in regards to KAFKA-1477 I just asked Ivan to update the patch
>> to
>> latest trunk and we could (demand required) make a patch that works
> with
>> 0.8.1.X too for folks to use... This doesn't work yet with the new
>> producer
>> (TBD) or any other clients so be aware it is not yet baked in and
>> from
>> release project perspective I don't know what in that patch will
> survive
>> (hopefully all of it).
>> 
>> /***
>> Joe Stein
>> Founder, Principal Consultant
>> Big Data Open Source Security LLC
>> http://www.stealth.ly
>> Twitter: @allthingshadoop 
>> /
>> 
>> On Tue, Sep 16, 2014 at 10:17 AM, Andrew Psaltis
>> 
>> wrote:
>> 
>>> Hi,
>>> I was just reading the recent changes to:
>>> https://cwiki.apache.org/confluence/display/KAFKA/Security after
> getting
>>> off a call about Kafka security and how we are jumping through
> hoops --
>>> like having PGP keys on the consumers and producers to get around
> the
> lack
>>> of SSL support. Did the meeting that Joe proposed happen for Sept
> 9th
>>> happen? If not is there a plan to have it? I was also looking at:
>>> https://issues.apache.org/jira/browse/KAFKA-1477 and it seems
>> like
> there
>>> have been no comments since 11/08/2014. I would be interested in
> helping
>>> with the TLS/SSL support as we have a need for it now.
>>> 
>>> Thanks,
>>> Andrew
>> 
>> 

[jira] [Updated] (KAFKA-1622) project shouldn't require signing to build

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1622:
-
Reviewer: Joe Stein

> project shouldn't require signing to build
> --
>
> Key: KAFKA-1622
> URL: https://issues.apache.org/jira/browse/KAFKA-1622
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>Assignee: Ivan Lyutov
>Priority: Blocker
>  Labels: build, newbie, packaging
> Fix For: 0.8.2
>
>
> we only need signing for uploadArchives that is it
> The project trunk failed to build due to some signing/license checks (the 
> diff I used to get things to build is here: 
> https://gist.github.com/dehora/7e3c0bd75bb2b5d87557)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Kafka Security

[Robert Withers]

Same here, please.

- Rob

> On Sep 16, 2014, at 12:58 PM, Steve Morin  wrote:
> 
> Same here
> 
>> On Sep 16, 2014, at 11:51, Joe Stein  wrote:
>> 
>> done
>> 
>>> On Tue, Sep 16, 2014 at 11:46 AM, Joel Koshy  wrote:
>>> 
>>> Same here.
>>> 
>>> Thanks,
>>> 
>>> Joel
>>> 
 On Tue, Sep 16, 2014 at 06:43:22PM +, Todd Palino wrote:
 Joe, can you add me to it as well? We obviously have some interest in the
 discussion over here :)
 
 -Todd
 
> On 9/16/14, 10:59 AM, "Joe Stein"  wrote:
> 
> cool, I just added you to the invite
> 
>> On Tue, Sep 16, 2014 at 10:57 AM, Harsha  wrote:
>> 
>> Hi Joe,
>>   I am interested in joining the efforts. I went through apache
>>   storm security with kerberos so I can bring some of that
>>   experience into the discussion.
>> Thanks,
>> Harsha
>> 
>>> On Tue, Sep 16, 2014, at 10:37 AM, Joe Stein wrote:
>>> Hi Andrew, yes the meeting took place and we plan to-do it every two
>>> weeks
>>> (same bat time, same bat channel) moving forward.
>>> 
>>> In attendance was Michael Herstine (LinkedIn), Arvind Mani
>>> (LinkedIn),
>>> Gwen
>>> Shapira (Cloudera) and myself.
>>> 
>>> Gwen updated the wiki after our discussion.  Basically we are
>> thinking of
>>> using 3 ports one for plain text (so like it is now), one for SASL
>>> (implementing username/password and kerberos at least) and one for
>>> SSL
>>> and
>>> they will all be configurable on/off.  Some investigation is going
>>> on
>> now
>>> to see about how we can do this without making any wire protocol
>> changes
>>> (ideal) or minimal changes at least.
>>> 
>>> Let me know and I can add you to the invite if you would like to
>>> contribute
>>> the more help and input the better for sure.
>>> 
>>> Also in regards to KAFKA-1477 I just asked Ivan to update the patch
>>> to
>>> latest trunk and we could (demand required) make a patch that works
>> with
>>> 0.8.1.X too for folks to use... This doesn't work yet with the new
>>> producer
>>> (TBD) or any other clients so be aware it is not yet baked in and
>>> from
>>> release project perspective I don't know what in that patch will
>> survive
>>> (hopefully all of it).
>>> 
>>> /***
>>> Joe Stein
>>> Founder, Principal Consultant
>>> Big Data Open Source Security LLC
>>> http://www.stealth.ly
>>> Twitter: @allthingshadoop 
>>> /
>>> 
>>> On Tue, Sep 16, 2014 at 10:17 AM, Andrew Psaltis
>>> 
>>> wrote:
>>> 
 Hi,
 I was just reading the recent changes to:
 https://cwiki.apache.org/confluence/display/KAFKA/Security after
>> getting
 off a call about Kafka security and how we are jumping through
>> hoops --
 like having PGP keys on the consumers and producers to get around
>> the
>> lack
 of SSL support. Did the meeting that Joe proposed happen for Sept
>> 9th
 happen? If not is there a plan to have it? I was also looking at:
 https://issues.apache.org/jira/browse/KAFKA-1477 and it seems
>>> like
>> there
 have been no comments since 11/08/2014. I would be interested in
>> helping
 with the TLS/SSL support as we have a need for it now.
 
 Thanks,
 Andrew
>>> 
>>> 

Re: Kafka Security

[Joe Stein]

done, done

On Tue, Sep 16, 2014 at 12:23 PM, Robert Withers  wrote:

> Same here, please.
>
> - Rob
>
> > On Sep 16, 2014, at 12:58 PM, Steve Morin  wrote:
> >
> > Same here
> >
> >> On Sep 16, 2014, at 11:51, Joe Stein  wrote:
> >>
> >> done
> >>
> >>> On Tue, Sep 16, 2014 at 11:46 AM, Joel Koshy 
> wrote:
> >>>
> >>> Same here.
> >>>
> >>> Thanks,
> >>>
> >>> Joel
> >>>
>  On Tue, Sep 16, 2014 at 06:43:22PM +, Todd Palino wrote:
>  Joe, can you add me to it as well? We obviously have some interest in
> the
>  discussion over here :)
> 
>  -Todd
> 
> > On 9/16/14, 10:59 AM, "Joe Stein"  wrote:
> >
> > cool, I just added you to the invite
> >
> >> On Tue, Sep 16, 2014 at 10:57 AM, Harsha  wrote:
> >>
> >> Hi Joe,
> >>   I am interested in joining the efforts. I went through apache
> >>   storm security with kerberos so I can bring some of that
> >>   experience into the discussion.
> >> Thanks,
> >> Harsha
> >>
> >>> On Tue, Sep 16, 2014, at 10:37 AM, Joe Stein wrote:
> >>> Hi Andrew, yes the meeting took place and we plan to-do it every
> two
> >>> weeks
> >>> (same bat time, same bat channel) moving forward.
> >>>
> >>> In attendance was Michael Herstine (LinkedIn), Arvind Mani
> >>> (LinkedIn),
> >>> Gwen
> >>> Shapira (Cloudera) and myself.
> >>>
> >>> Gwen updated the wiki after our discussion.  Basically we are
> >> thinking of
> >>> using 3 ports one for plain text (so like it is now), one for SASL
> >>> (implementing username/password and kerberos at least) and one for
> >>> SSL
> >>> and
> >>> they will all be configurable on/off.  Some investigation is going
> >>> on
> >> now
> >>> to see about how we can do this without making any wire protocol
> >> changes
> >>> (ideal) or minimal changes at least.
> >>>
> >>> Let me know and I can add you to the invite if you would like to
> >>> contribute
> >>> the more help and input the better for sure.
> >>>
> >>> Also in regards to KAFKA-1477 I just asked Ivan to update the patch
> >>> to
> >>> latest trunk and we could (demand required) make a patch that works
> >> with
> >>> 0.8.1.X too for folks to use... This doesn't work yet with the new
> >>> producer
> >>> (TBD) or any other clients so be aware it is not yet baked in and
> >>> from
> >>> release project perspective I don't know what in that patch will
> >> survive
> >>> (hopefully all of it).
> >>>
> >>> /***
> >>> Joe Stein
> >>> Founder, Principal Consultant
> >>> Big Data Open Source Security LLC
> >>> http://www.stealth.ly
> >>> Twitter: @allthingshadoop 
> >>> /
> >>>
> >>> On Tue, Sep 16, 2014 at 10:17 AM, Andrew Psaltis
> >>> 
> >>> wrote:
> >>>
>  Hi,
>  I was just reading the recent changes to:
>  https://cwiki.apache.org/confluence/display/KAFKA/Security after
> >> getting
>  off a call about Kafka security and how we are jumping through
> >> hoops --
>  like having PGP keys on the consumers and producers to get around
> >> the
> >> lack
>  of SSL support. Did the meeting that Joe proposed happen for Sept
> >> 9th
>  happen? If not is there a plan to have it? I was also looking at:
>  https://issues.apache.org/jira/browse/KAFKA-1477 and it seems
> >>> like
> >> there
>  have been no comments since 11/08/2014. I would be interested in
> >> helping
>  with the TLS/SSL support as we have a need for it now.
> 
>  Thanks,
>  Andrew
> >>>
> >>>
>

Re: Kafka Security

[Robert Withers]

Oops, n/m for now, I'm non-local.

Thanks,
- Rob

> On Sep 16, 2014, at 1:23 PM, Robert Withers  
> wrote:
> 
> Same here, please.
> 
> - Rob
> 
>> On Sep 16, 2014, at 12:58 PM, Steve Morin  wrote:
>> 
>> Same here
>> 
>>> On Sep 16, 2014, at 11:51, Joe Stein  wrote:
>>> 
>>> done
>>> 
 On Tue, Sep 16, 2014 at 11:46 AM, Joel Koshy  wrote:
 
 Same here.
 
 Thanks,
 
 Joel
 
> On Tue, Sep 16, 2014 at 06:43:22PM +, Todd Palino wrote:
> Joe, can you add me to it as well? We obviously have some interest in the
> discussion over here :)
> 
> -Todd
> 
>> On 9/16/14, 10:59 AM, "Joe Stein"  wrote:
>> 
>> cool, I just added you to the invite
>> 
>>> On Tue, Sep 16, 2014 at 10:57 AM, Harsha  wrote:
>>> 
>>> Hi Joe,
>>>   I am interested in joining the efforts. I went through apache
>>>   storm security with kerberos so I can bring some of that
>>>   experience into the discussion.
>>> Thanks,
>>> Harsha
>>> 
 On Tue, Sep 16, 2014, at 10:37 AM, Joe Stein wrote:
 Hi Andrew, yes the meeting took place and we plan to-do it every two
 weeks
 (same bat time, same bat channel) moving forward.
 
 In attendance was Michael Herstine (LinkedIn), Arvind Mani
 (LinkedIn),
 Gwen
 Shapira (Cloudera) and myself.
 
 Gwen updated the wiki after our discussion.  Basically we are
>>> thinking of
 using 3 ports one for plain text (so like it is now), one for SASL
 (implementing username/password and kerberos at least) and one for
 SSL
 and
 they will all be configurable on/off.  Some investigation is going
 on
>>> now
 to see about how we can do this without making any wire protocol
>>> changes
 (ideal) or minimal changes at least.
 
 Let me know and I can add you to the invite if you would like to
 contribute
 the more help and input the better for sure.
 
 Also in regards to KAFKA-1477 I just asked Ivan to update the patch
 to
 latest trunk and we could (demand required) make a patch that works
>>> with
 0.8.1.X too for folks to use... This doesn't work yet with the new
 producer
 (TBD) or any other clients so be aware it is not yet baked in and
 from
 release project perspective I don't know what in that patch will
>>> survive
 (hopefully all of it).
 
 /***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
 /
 
 On Tue, Sep 16, 2014 at 10:17 AM, Andrew Psaltis
 
 wrote:
 
> Hi,
> I was just reading the recent changes to:
> https://cwiki.apache.org/confluence/display/KAFKA/Security after
>>> getting
> off a call about Kafka security and how we are jumping through
>>> hoops --
> like having PGP keys on the consumers and producers to get around
>>> the
>>> lack
> of SSL support. Did the meeting that Joe proposed happen for Sept
>>> 9th
> happen? If not is there a plan to have it? I was also looking at:
> https://issues.apache.org/jira/browse/KAFKA-1477 and it seems
 like
>>> there
> have been no comments since 11/08/2014. I would be interested in
>>> helping
> with the TLS/SSL support as we have a need for it now.
> 
> Thanks,
> Andrew
 
 

[jira] [Updated] (KAFKA-1282) Disconnect idle socket connection in Selector

[nicu marasoiu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nicu marasoiu updated KAFKA-1282:
-
Attachment: 1282_the_ultimate,_close_max_one_per_select.patch

Attached patch: every select iteration, zero or one connections are closed for 
being idle for too long.
The units pass well, but
For the moment I am blocked by:
./kafka-console-producer.sh
Error: Could not find or load main class kafka.tools.ConsoleProducer

> Disconnect idle socket connection in Selector
> -
>
> Key: KAFKA-1282
> URL: https://issues.apache.org/jira/browse/KAFKA-1282
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: nicu marasoiu
>  Labels: newbie++
> Fix For: 0.9.0
>
> Attachments: 1282_the_ultimate,_close_max_one_per_select.patch, 
> KAFKA-1282_Disconnect_idle_socket_connection_in_Selector.patch, 
> idleDisconnect.patch
>
>
> To reduce # socket connections, it would be useful for the new producer to 
> close socket connections that are idle. We can introduce a new producer 
> config for the idle time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1610) Local modifications to collections generated from mapValues will be lost

[Mayuresh Gharat (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayuresh Gharat updated KAFKA-1610:
---
Attachment: KAFKA-1610_2014-09-16_13:08:17.patch

> Local modifications to collections generated from mapValues will be lost
> 
>
> Key: KAFKA-1610
> URL: https://issues.apache.org/jira/browse/KAFKA-1610
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Mayuresh Gharat
>  Labels: newbie
> Fix For: 0.9.0
>
> Attachments: KAFKA-1610.patch, KAFKA-1610_2014-08-29_09:51:51.patch, 
> KAFKA-1610_2014-08-29_10:03:55.patch, KAFKA-1610_2014-09-03_11:27:50.patch, 
> KAFKA-1610_2014-09-16_13:08:17.patch
>
>
> In our current Scala code base we have 40+ usages of mapValues, however it 
> has an important semantic difference with map, which is that "map" creates a 
> new map collection instance, while "mapValues" just create a map view of the 
> original map, and hence any further value changes to the view will be 
> effectively lost.
> Example code:
> {code}
> scala> case class Test(i: Int, var j: Int) {}
> defined class Test
> scala> val a = collection.mutable.Map(1 -> 1)
> a: scala.collection.mutable.Map[Int,Int] = Map(1 -> 1)
> scala> val b = a.mapValues(v => Test(v, v))
> b: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> val c = a.map(v => v._1 -> Test(v._2, v._2))
> c: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> b.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> b
> res1: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> c.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> c
> res3: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> scala> a.put(1,3)
> res4: Option[Int] = Some(1)
> scala> b
> res5: scala.collection.Map[Int,Test] = Map(1 -> Test(3,3))
> scala> c
> res6: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> {code}
> We need to go through all these mapValue to see if they should be changed to 
> map



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 25136: Patch for KAFKA-1610

[Mayuresh Gharat]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25136/
---

(Updated Sept. 16, 2014, 8:08 p.m.)


Review request for kafka.


Bugs: KAFKA-1610
https://issues.apache.org/jira/browse/KAFKA-1610


Repository: kafka


Description (updated)
---

Reverting the changes and adding comments to make the usage of mapValues more 
clear


Diffs (updated)
-

  core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 
691d69a49a240f38883d2025afaec26fd61281b5 
  core/src/main/scala/kafka/controller/KafkaController.scala 
8ab4a1b8072c9dd187a9a6e94138b725d1f1b153 
  core/src/main/scala/kafka/server/DelayedFetch.scala 
e0f14e25af03e6d4344386dcabc1457ee784d345 
  core/src/main/scala/kafka/server/DelayedProduce.scala 
9481508fc2d6140b36829840c337e557f3d090da 
  core/src/main/scala/kafka/server/KafkaApis.scala 
c584b559416b3ee4bcbec5966be4891e0a03eefb 
  core/src/main/scala/kafka/server/KafkaServer.scala 
28711182aaa70eaa623de858bc063cb2613b2a4d 
  core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala 
af4783646803e58714770c21f8c3352370f26854 
  core/src/test/scala/unit/kafka/server/LeaderElectionTest.scala 
c2ba07c5fdbaf0e65ca033b2e4d88f45a8a15b2e 

Diff: https://reviews.apache.org/r/25136/diff/


Testing
---

Ran the unit tests and everything passed and the build succeeeded


Thanks,

Mayuresh Gharat

[jira] [Commented] (KAFKA-1610) Local modifications to collections generated from mapValues will be lost

[Mayuresh Gharat (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136111#comment-14136111
 ] 

Mayuresh Gharat commented on KAFKA-1610:


Updated reviewboard https://reviews.apache.org/r/25136/diff/
 against branch trunk

> Local modifications to collections generated from mapValues will be lost
> 
>
> Key: KAFKA-1610
> URL: https://issues.apache.org/jira/browse/KAFKA-1610
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Mayuresh Gharat
>  Labels: newbie
> Fix For: 0.9.0
>
> Attachments: KAFKA-1610.patch, KAFKA-1610_2014-08-29_09:51:51.patch, 
> KAFKA-1610_2014-08-29_10:03:55.patch, KAFKA-1610_2014-09-03_11:27:50.patch, 
> KAFKA-1610_2014-09-16_13:08:17.patch
>
>
> In our current Scala code base we have 40+ usages of mapValues, however it 
> has an important semantic difference with map, which is that "map" creates a 
> new map collection instance, while "mapValues" just create a map view of the 
> original map, and hence any further value changes to the view will be 
> effectively lost.
> Example code:
> {code}
> scala> case class Test(i: Int, var j: Int) {}
> defined class Test
> scala> val a = collection.mutable.Map(1 -> 1)
> a: scala.collection.mutable.Map[Int,Int] = Map(1 -> 1)
> scala> val b = a.mapValues(v => Test(v, v))
> b: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> val c = a.map(v => v._1 -> Test(v._2, v._2))
> c: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> b.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> b
> res1: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> c.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> c
> res3: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> scala> a.put(1,3)
> res4: Option[Int] = Some(1)
> scala> b
> res5: scala.collection.Map[Int,Test] = Map(1 -> Test(3,3))
> scala> c
> res6: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> {code}
> We need to go through all these mapValue to see if they should be changed to 
> map



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 25136: Patch for KAFKA-1610

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25136/#review53574
---


Looks got to me, just some minor comments below. Joel, could you take another 
look?


core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala


could you also leave a space at the beginning just for consistency with 
other comments? Ditto as below.



core/src/main/scala/kafka/server/DelayedFetch.scala


Thanks for the catch.



core/src/main/scala/kafka/server/DelayedProduce.scala


rename to responseStatusView?



core/src/main/scala/kafka/server/KafkaServer.scala


Can you rename this to configsView?



core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala


rename to leadersPerBrokerView?


- Guozhang Wang


On Sept. 16, 2014, 8:08 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25136/
> ---
> 
> (Updated Sept. 16, 2014, 8:08 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1610
> https://issues.apache.org/jira/browse/KAFKA-1610
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Reverting the changes and adding comments to make the usage of mapValues more 
> clear
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 
> 691d69a49a240f38883d2025afaec26fd61281b5 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 8ab4a1b8072c9dd187a9a6e94138b725d1f1b153 
>   core/src/main/scala/kafka/server/DelayedFetch.scala 
> e0f14e25af03e6d4344386dcabc1457ee784d345 
>   core/src/main/scala/kafka/server/DelayedProduce.scala 
> 9481508fc2d6140b36829840c337e557f3d090da 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> c584b559416b3ee4bcbec5966be4891e0a03eefb 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> 28711182aaa70eaa623de858bc063cb2613b2a4d 
>   core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala 
> af4783646803e58714770c21f8c3352370f26854 
>   core/src/test/scala/unit/kafka/server/LeaderElectionTest.scala 
> c2ba07c5fdbaf0e65ca033b2e4d88f45a8a15b2e 
> 
> Diff: https://reviews.apache.org/r/25136/diff/
> 
> 
> Testing
> ---
> 
> Ran the unit tests and everything passed and the build succeeeded
> 
> 
> Thanks,
> 
> Mayuresh Gharat
> 
>

Re: Kafka Security

[Jay Kreps]

Hey Joe,

Can you add me, Jun, and Neha too.

-Jay

On Tue, Sep 16, 2014 at 10:37 AM, Joe Stein  wrote:
> Hi Andrew, yes the meeting took place and we plan to-do it every two weeks
> (same bat time, same bat channel) moving forward.
>
> In attendance was Michael Herstine (LinkedIn), Arvind Mani (LinkedIn), Gwen
> Shapira (Cloudera) and myself.
>
> Gwen updated the wiki after our discussion.  Basically we are thinking of
> using 3 ports one for plain text (so like it is now), one for SASL
> (implementing username/password and kerberos at least) and one for SSL and
> they will all be configurable on/off.  Some investigation is going on now
> to see about how we can do this without making any wire protocol changes
> (ideal) or minimal changes at least.
>
> Let me know and I can add you to the invite if you would like to contribute
> the more help and input the better for sure.
>
> Also in regards to KAFKA-1477 I just asked Ivan to update the patch to
> latest trunk and we could (demand required) make a patch that works with
> 0.8.1.X too for folks to use... This doesn't work yet with the new producer
> (TBD) or any other clients so be aware it is not yet baked in and from
> release project perspective I don't know what in that patch will survive
> (hopefully all of it).
>
> /***
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop 
> /
>
> On Tue, Sep 16, 2014 at 10:17 AM, Andrew Psaltis 
> wrote:
>
>> Hi,
>> I was just reading the recent changes to:
>> https://cwiki.apache.org/confluence/display/KAFKA/Security after getting
>> off a call about Kafka security and how we are jumping through hoops --
>> like having PGP keys on the consumers and producers to get around the lack
>> of SSL support. Did the meeting that Joe proposed happen for Sept 9th
>> happen? If not is there a plan to have it? I was also looking at:
>> https://issues.apache.org/jira/browse/KAFKA-1477 and it seems like there
>> have been no comments since 11/08/2014. I would be interested in helping
>> with the TLS/SSL support as we have a need for it now.
>>
>> Thanks,
>> Andrew
>>

[jira] [Commented] (KAFKA-1618) Exception thrown when running console producer with no port number for the broker

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136206#comment-14136206
 ] 

Jay Kreps commented on KAFKA-1618:
--

I think the problem with a default port is that yes, in 70% of cases that will 
save your having to enter the port, but in the 30% of cases where the default 
is wrong it will be may be confusing why it isn't working. I think having 
people enter the port is not so bad and a pretty common thing for tools that 
connect to remote hosts and ports.

> Exception thrown when running console producer with no port number for the 
> broker
> -
>
> Key: KAFKA-1618
> URL: https://issues.apache.org/jira/browse/KAFKA-1618
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.1.1
>Reporter: Gwen Shapira
>Assignee: BalajiSeshadri
>  Labels: newbie
> Fix For: 0.8.2
>
> Attachments: KAFKA-1618-ALL.patch, KAFKA-1618.patch
>
>
> When running console producer with just "localhost" as the broker list, I get 
> ArrayIndexOutOfBounds exception.
> I expect either a clearer error about arguments or for the producer to 
> "guess" a default port.
> [root@shapira-1 bin]# ./kafka-console-producer.sh  --topic rufus1 
> --broker-list localhost
> java.lang.ArrayIndexOutOfBoundsException: 1
>   at 
> kafka.client.ClientUtils$$anonfun$parseBrokerList$1.apply(ClientUtils.scala:102)
>   at 
> kafka.client.ClientUtils$$anonfun$parseBrokerList$1.apply(ClientUtils.scala:97)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:105)
>   at kafka.client.ClientUtils$.parseBrokerList(ClientUtils.scala:97)
>   at 
> kafka.producer.BrokerPartitionInfo.(BrokerPartitionInfo.scala:32)
>   at 
> kafka.producer.async.DefaultEventHandler.(DefaultEventHandler.scala:41)
>   at kafka.producer.Producer.(Producer.scala:59)
>   at kafka.producer.ConsoleProducer$.main(ConsoleProducer.scala:158)
>   at kafka.producer.ConsoleProducer.main(ConsoleProducer.scala)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Kafka Security

[Joe Stein]

yup, yup, yup | done, done, done

On Tue, Sep 16, 2014 at 1:54 PM, Jay Kreps  wrote:

> Hey Joe,
>
> Can you add me, Jun, and Neha too.
>
> -Jay
>
> On Tue, Sep 16, 2014 at 10:37 AM, Joe Stein  wrote:
> > Hi Andrew, yes the meeting took place and we plan to-do it every two
> weeks
> > (same bat time, same bat channel) moving forward.
> >
> > In attendance was Michael Herstine (LinkedIn), Arvind Mani (LinkedIn),
> Gwen
> > Shapira (Cloudera) and myself.
> >
> > Gwen updated the wiki after our discussion.  Basically we are thinking of
> > using 3 ports one for plain text (so like it is now), one for SASL
> > (implementing username/password and kerberos at least) and one for SSL
> and
> > they will all be configurable on/off.  Some investigation is going on now
> > to see about how we can do this without making any wire protocol changes
> > (ideal) or minimal changes at least.
> >
> > Let me know and I can add you to the invite if you would like to
> contribute
> > the more help and input the better for sure.
> >
> > Also in regards to KAFKA-1477 I just asked Ivan to update the patch to
> > latest trunk and we could (demand required) make a patch that works with
> > 0.8.1.X too for folks to use... This doesn't work yet with the new
> producer
> > (TBD) or any other clients so be aware it is not yet baked in and from
> > release project perspective I don't know what in that patch will survive
> > (hopefully all of it).
> >
> > /***
> >  Joe Stein
> >  Founder, Principal Consultant
> >  Big Data Open Source Security LLC
> >  http://www.stealth.ly
> >  Twitter: @allthingshadoop 
> > /
> >
> > On Tue, Sep 16, 2014 at 10:17 AM, Andrew Psaltis <
> psaltis.and...@gmail.com>
> > wrote:
> >
> >> Hi,
> >> I was just reading the recent changes to:
> >> https://cwiki.apache.org/confluence/display/KAFKA/Security after
> getting
> >> off a call about Kafka security and how we are jumping through hoops --
> >> like having PGP keys on the consumers and producers to get around the
> lack
> >> of SSL support. Did the meeting that Joe proposed happen for Sept 9th
> >> happen? If not is there a plan to have it? I was also looking at:
> >> https://issues.apache.org/jira/browse/KAFKA-1477 and it seems like
> there
> >> have been no comments since 11/08/2014. I would be interested in helping
> >> with the TLS/SSL support as we have a need for it now.
> >>
> >> Thanks,
> >> Andrew
> >>
>

[jira] [Commented] (KAFKA-1618) Exception thrown when running console producer with no port number for the broker

[Balaji Seshadri (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136220#comment-14136220
 ] 

Balaji Seshadri commented on KAFKA-1618:


I agree with you [~jkreps] and [~nehanarkhede],its just ceremonial to have 
deafult one like Tomcat or any App server.
Most of the cases it will be custom port.

> Exception thrown when running console producer with no port number for the 
> broker
> -
>
> Key: KAFKA-1618
> URL: https://issues.apache.org/jira/browse/KAFKA-1618
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.1.1
>Reporter: Gwen Shapira
>Assignee: BalajiSeshadri
>  Labels: newbie
> Fix For: 0.8.2
>
> Attachments: KAFKA-1618-ALL.patch, KAFKA-1618.patch
>
>
> When running console producer with just "localhost" as the broker list, I get 
> ArrayIndexOutOfBounds exception.
> I expect either a clearer error about arguments or for the producer to 
> "guess" a default port.
> [root@shapira-1 bin]# ./kafka-console-producer.sh  --topic rufus1 
> --broker-list localhost
> java.lang.ArrayIndexOutOfBoundsException: 1
>   at 
> kafka.client.ClientUtils$$anonfun$parseBrokerList$1.apply(ClientUtils.scala:102)
>   at 
> kafka.client.ClientUtils$$anonfun$parseBrokerList$1.apply(ClientUtils.scala:97)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:105)
>   at kafka.client.ClientUtils$.parseBrokerList(ClientUtils.scala:97)
>   at 
> kafka.producer.BrokerPartitionInfo.(BrokerPartitionInfo.scala:32)
>   at 
> kafka.producer.async.DefaultEventHandler.(DefaultEventHandler.scala:41)
>   at kafka.producer.Producer.(Producer.scala:59)
>   at kafka.producer.ConsoleProducer$.main(ConsoleProducer.scala:158)
>   at kafka.producer.ConsoleProducer.main(ConsoleProducer.scala)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: MBeans, dashes, underscores, and KAFKA-1481

[Otis Gospodnetic]

Hi,

So maybe I should I should have asked the Q explicitly:
Could we commit the patch from
https://issues.apache.org/jira/browse/KAFKA-1481 now that, I hope, it's
clear what problems the current MBean names can cause?

Thanks,
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/



On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
otis.gospodne...@gmail.com> wrote:

> Hi,
>
> *Problem:*
> Some Kafka 0.8.x MBeans have names composed of things like  group>--.  Note how dashes are used as delimiters.
>  When  and  don't contain the delimiter character
> all is good if you want to extract parts of this MBean name by simply
> splitting on the delimiter character.  The problem is that dashes are
> allowed in topic and group names, so this splitting doesn't work.
> Moreover, underscores are also used as delimiters, and they can also be
> used in things like topic names.
>
> *Example*:
> This MBean's name is composed of --BytesPerSec:
>
> kafka.consumer:type="ConsumerTopicMetrics", name="*myGroup**-myTopic**-*
> BytesPerSec"
>
> Here we can actually split on "-" and extract all 3 parts from the MBean
> name::
> * consumer group ('*myGroup*')
> * topic ('*myTopic*')
> * metric (‘BytesPerSec’)
>
> All good!
>
> But imagine if I named the group: *my-Group*
> And if I named the topic: *my-Topic*
>
> Then we'd have:
> kafka.consumer:type="ConsumerTopicMetrics", name="*my-Group**-my-Topic**-*
> BytesPerSec"
>
> Now splitting on "-" would no longer work!  To extract "my-Group" and
> "my-Topic" and "BytesPerSec" parts I would have to know the specific group
> name and topic name to look for and could not use generic approach of just
> splitting the MBean name on the delimiter.
>
> *Solution*:
> The patch in https://issues.apache.org/jira/browse/KAFKA-1481 replaces
> all _ and - characters where they are used as delimiters in MBean names
> with a "|" character.  Because the "I" character is not allowed in topic
> names, consumer groups, host names, splitting on this new and unified
> delimiter works.
>
> I hope this explains the problem, the solution, and that this can make it
> in the next 0.8.x.
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>

[jira] [Commented] (KAFKA-1624) building on JDK 8 fails

[Stevo Slavic (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136314#comment-14136314
 ] 

Stevo Slavic commented on KAFKA-1624:
-

If I'm not mistaken, this error is caused by Scala 2.10 (and older) 
incompatibility with (reading) Java 8.
{{./gradlew clean test_core_2_11}} with Java 8 passes successfully for me on 
current trunk, although even Scala 2.11 just has experimental support for Java 
8.

Scala 2.12 will require Java 8 (see [Scala 2.12 
roadmap|http://www.scala-lang.org/news/2.12-roadmap]).

> building on JDK 8 fails
> ---
>
> Key: KAFKA-1624
> URL: https://issues.apache.org/jira/browse/KAFKA-1624
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: newbie
> Fix For: 0.9.0
>
>
> {code}
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; 
> support was removed in 8.0
> error: error while loading CharSequence, class file 
> '/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar(java/lang/CharSequence.class)' is 
> broken
> (class java.lang.RuntimeException/bad constant pool tag 18 at byte 10)
> error: error while loading Comparator, class file 
> '/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar(java/util/Comparator.class)' is 
> broken
> (class java.lang.RuntimeException/bad constant pool tag 18 at byte 20)
> error: error while loading AnnotatedElement, class file 
> '/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar(java/lang/reflect/AnnotatedElement.class)'
>  is broken
> (class java.lang.RuntimeException/bad constant pool tag 18 at byte 76)
> error: error while loading Arrays, class file 
> '/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar(java/util/Arrays.class)' is broken
> (class java.lang.RuntimeException/bad constant pool tag 18 at byte 765)
> /tmp/sbt_53783b12/xsbt/ExtractAPI.scala:395: error: java.util.Comparator does 
> not take type parameters
>   private[this] val sortClasses = new Comparator[Symbol] {
> ^
> 5 errors found
> :core:compileScala FAILED
> FAILURE: Build failed with an exception.
> * What went wrong:
> Execution failed for task ':core:compileScala'.
> > org.gradle.messaging.remote.internal.PlaceholderException (no error message)
> * Try:
> Run with --stacktrace option to get the stack trace. Run with --info or 
> --debug option to get more log output.
> BUILD FAILED
> Total time: 1 mins 48.298 secs
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1123) Broker IPv6 addresses parsed incorrectly

[JIRA]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Krzysztof Szafrański updated KAFKA-1123:

Attachment: KAFKA-1123_v2.patch

> Broker IPv6 addresses parsed incorrectly
> 
>
> Key: KAFKA-1123
> URL: https://issues.apache.org/jira/browse/KAFKA-1123
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.8.2
>Reporter: Andrew Otto
>  Labels: newbie
> Attachments: KAFKA-1123.patch, KAFKA-1123_v2.patch
>
>
> It seems that broker addresses are parsed incorrectly when IPv6 addresses are 
> supplied.  IPv6 addresses have colons in them, and Kafka seems to be 
> interpreting the first : as the address:port separator.
> I have only tried this with the console-producer --broker-list option, so I 
> don't know if this affects anything deeper than the CLI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1618) Exception thrown when running console producer with no port number for the broker

[Gwen Shapira (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136362#comment-14136362
 ] 

Gwen Shapira commented on KAFKA-1618:
-

I can't think of other client server system that doesn't guess default ports. 
Our users use ZooKeeper (at least), so they must be used to clients that guess 
the default port.

However, I agree that this is not critical and mostly a "nice to have". 

What's important is that the behavior has to be consistent everywhere. 
If users have to wonder "why does this URI works in X but not in Y", we are 
doing something wrong. If we decide that guessing is wrong, none of our tools 
should accept just a server name.

> Exception thrown when running console producer with no port number for the 
> broker
> -
>
> Key: KAFKA-1618
> URL: https://issues.apache.org/jira/browse/KAFKA-1618
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.1.1
>Reporter: Gwen Shapira
>Assignee: BalajiSeshadri
>  Labels: newbie
> Fix For: 0.8.2
>
> Attachments: KAFKA-1618-ALL.patch, KAFKA-1618.patch
>
>
> When running console producer with just "localhost" as the broker list, I get 
> ArrayIndexOutOfBounds exception.
> I expect either a clearer error about arguments or for the producer to 
> "guess" a default port.
> [root@shapira-1 bin]# ./kafka-console-producer.sh  --topic rufus1 
> --broker-list localhost
> java.lang.ArrayIndexOutOfBoundsException: 1
>   at 
> kafka.client.ClientUtils$$anonfun$parseBrokerList$1.apply(ClientUtils.scala:102)
>   at 
> kafka.client.ClientUtils$$anonfun$parseBrokerList$1.apply(ClientUtils.scala:97)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:105)
>   at kafka.client.ClientUtils$.parseBrokerList(ClientUtils.scala:97)
>   at 
> kafka.producer.BrokerPartitionInfo.(BrokerPartitionInfo.scala:32)
>   at 
> kafka.producer.async.DefaultEventHandler.(DefaultEventHandler.scala:41)
>   at kafka.producer.Producer.(Producer.scala:59)
>   at kafka.producer.ConsoleProducer$.main(ConsoleProducer.scala:158)
>   at kafka.producer.ConsoleProducer.main(ConsoleProducer.scala)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1610) Local modifications to collections generated from mapValues will be lost

[Mayuresh Gharat (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayuresh Gharat updated KAFKA-1610:
---
Attachment: KAFKA-1610_2014-09-16_15:23:27.patch

> Local modifications to collections generated from mapValues will be lost
> 
>
> Key: KAFKA-1610
> URL: https://issues.apache.org/jira/browse/KAFKA-1610
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Mayuresh Gharat
>  Labels: newbie
> Fix For: 0.9.0
>
> Attachments: KAFKA-1610.patch, KAFKA-1610_2014-08-29_09:51:51.patch, 
> KAFKA-1610_2014-08-29_10:03:55.patch, KAFKA-1610_2014-09-03_11:27:50.patch, 
> KAFKA-1610_2014-09-16_13:08:17.patch, KAFKA-1610_2014-09-16_15:23:27.patch
>
>
> In our current Scala code base we have 40+ usages of mapValues, however it 
> has an important semantic difference with map, which is that "map" creates a 
> new map collection instance, while "mapValues" just create a map view of the 
> original map, and hence any further value changes to the view will be 
> effectively lost.
> Example code:
> {code}
> scala> case class Test(i: Int, var j: Int) {}
> defined class Test
> scala> val a = collection.mutable.Map(1 -> 1)
> a: scala.collection.mutable.Map[Int,Int] = Map(1 -> 1)
> scala> val b = a.mapValues(v => Test(v, v))
> b: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> val c = a.map(v => v._1 -> Test(v._2, v._2))
> c: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> b.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> b
> res1: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> c.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> c
> res3: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> scala> a.put(1,3)
> res4: Option[Int] = Some(1)
> scala> b
> res5: scala.collection.Map[Int,Test] = Map(1 -> Test(3,3))
> scala> c
> res6: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> {code}
> We need to go through all these mapValue to see if they should be changed to 
> map



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1610) Local modifications to collections generated from mapValues will be lost

[Mayuresh Gharat (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136372#comment-14136372
 ] 

Mayuresh Gharat commented on KAFKA-1610:


Updated reviewboard https://reviews.apache.org/r/25136/diff/
 against branch trunk

> Local modifications to collections generated from mapValues will be lost
> 
>
> Key: KAFKA-1610
> URL: https://issues.apache.org/jira/browse/KAFKA-1610
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Mayuresh Gharat
>  Labels: newbie
> Fix For: 0.9.0
>
> Attachments: KAFKA-1610.patch, KAFKA-1610_2014-08-29_09:51:51.patch, 
> KAFKA-1610_2014-08-29_10:03:55.patch, KAFKA-1610_2014-09-03_11:27:50.patch, 
> KAFKA-1610_2014-09-16_13:08:17.patch, KAFKA-1610_2014-09-16_15:23:27.patch
>
>
> In our current Scala code base we have 40+ usages of mapValues, however it 
> has an important semantic difference with map, which is that "map" creates a 
> new map collection instance, while "mapValues" just create a map view of the 
> original map, and hence any further value changes to the view will be 
> effectively lost.
> Example code:
> {code}
> scala> case class Test(i: Int, var j: Int) {}
> defined class Test
> scala> val a = collection.mutable.Map(1 -> 1)
> a: scala.collection.mutable.Map[Int,Int] = Map(1 -> 1)
> scala> val b = a.mapValues(v => Test(v, v))
> b: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> val c = a.map(v => v._1 -> Test(v._2, v._2))
> c: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> b.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> b
> res1: scala.collection.Map[Int,Test] = Map(1 -> Test(1,1))
> scala> c.foreach(kv => kv._2.j = kv._2.j + 1)
> scala> c
> res3: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> scala> a.put(1,3)
> res4: Option[Int] = Some(1)
> scala> b
> res5: scala.collection.Map[Int,Test] = Map(1 -> Test(3,3))
> scala> c
> res6: scala.collection.mutable.Map[Int,Test] = Map(1 -> Test(1,2))
> {code}
> We need to go through all these mapValue to see if they should be changed to 
> map



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 25136: Patch for KAFKA-1610

[Mayuresh Gharat]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25136/
---

(Updated Sept. 16, 2014, 10:23 p.m.)


Review request for kafka.


Bugs: KAFKA-1610
https://issues.apache.org/jira/browse/KAFKA-1610


Repository: kafka


Description (updated)
---

Reverting the changes and adding comments to make the usage of mapValues more 
clear


Formatted the comments


Diffs (updated)
-

  core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 
691d69a49a240f38883d2025afaec26fd61281b5 
  core/src/main/scala/kafka/controller/KafkaController.scala 
8ab4a1b8072c9dd187a9a6e94138b725d1f1b153 
  core/src/main/scala/kafka/server/DelayedFetch.scala 
e0f14e25af03e6d4344386dcabc1457ee784d345 
  core/src/main/scala/kafka/server/DelayedProduce.scala 
9481508fc2d6140b36829840c337e557f3d090da 
  core/src/main/scala/kafka/server/KafkaApis.scala 
c584b559416b3ee4bcbec5966be4891e0a03eefb 
  core/src/main/scala/kafka/server/KafkaServer.scala 
28711182aaa70eaa623de858bc063cb2613b2a4d 
  core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala 
af4783646803e58714770c21f8c3352370f26854 
  core/src/test/scala/unit/kafka/server/LeaderElectionTest.scala 
c2ba07c5fdbaf0e65ca033b2e4d88f45a8a15b2e 

Diff: https://reviews.apache.org/r/25136/diff/


Testing
---

Ran the unit tests and everything passed and the build succeeeded


Thanks,

Mayuresh Gharat

Re: MBeans, dashes, underscores, and KAFKA-1481

[Bhavesh Mistry]

HI Otis,

What is migration path ?  If topic with special chars exists already(
".","-","|" etc)  in previous version of producer/consumer of Kafka, what
happens after the upgrade new producer or consumer (kafka version) ?  Also,
in new producer API (Kafka Trunk), does this enforce the rule about client
id as well ?

Thanks,

Bhavesh

On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic <
otis.gospodne...@gmail.com> wrote:

> Hi,
>
> So maybe I should I should have asked the Q explicitly:
> Could we commit the patch from
> https://issues.apache.org/jira/browse/KAFKA-1481 now that, I hope, it's
> clear what problems the current MBean names can cause?
>
> Thanks,
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
>
> On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic <
> otis.gospodne...@gmail.com> wrote:
>
> > Hi,
> >
> > *Problem:*
> > Some Kafka 0.8.x MBeans have names composed of things like  > group>--.  Note how dashes are used as delimiters.
> >  When  and  don't contain the delimiter character
> > all is good if you want to extract parts of this MBean name by simply
> > splitting on the delimiter character.  The problem is that dashes are
> > allowed in topic and group names, so this splitting doesn't work.
> > Moreover, underscores are also used as delimiters, and they can also be
> > used in things like topic names.
> >
> > *Example*:
> > This MBean's name is composed of --BytesPerSec:
> >
> > kafka.consumer:type="ConsumerTopicMetrics", name="*myGroup**-myTopic**-*
> > BytesPerSec"
> >
> > Here we can actually split on "-" and extract all 3 parts from the MBean
> > name::
> > * consumer group ('*myGroup*')
> > * topic ('*myTopic*')
> > * metric (‘BytesPerSec’)
> >
> > All good!
> >
> > But imagine if I named the group: *my-Group*
> > And if I named the topic: *my-Topic*
> >
> > Then we'd have:
> > kafka.consumer:type="ConsumerTopicMetrics",
> name="*my-Group**-my-Topic**-*
> > BytesPerSec"
> >
> > Now splitting on "-" would no longer work!  To extract "my-Group" and
> > "my-Topic" and "BytesPerSec" parts I would have to know the specific
> group
> > name and topic name to look for and could not use generic approach of
> just
> > splitting the MBean name on the delimiter.
> >
> > *Solution*:
> > The patch in https://issues.apache.org/jira/browse/KAFKA-1481 replaces
> > all _ and - characters where they are used as delimiters in MBean names
> > with a "|" character.  Because the "I" character is not allowed in topic
> > names, consumer groups, host names, splitting on this new and unified
> > delimiter works.
> >
> > I hope this explains the problem, the solution, and that this can make it
> > in the next 0.8.x.
> >
> > Otis
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
>

kafka 0.8.1: Producer.send() can block forever when a broker is down

[Jack Foy]

We observe that when a broker is down, Producer.send() can get into a state 
where it will block forever, even when using the async producer.

When a Producer first sends data, it fetches topic metadata from the broker 
cluster. To do this, it shuffles the list of hosts in the cluster, then 
iterates through the list querying each broker.

For each broker in the shuffled list, the Producer creates a SyncProducer and 
invokes SyncProducer.send(). 
SyncProducer.send() creates a BlockingChannel and invokes 
BlockingChannel.connect().
BlockingChannel.connect() retrieves a java.nio.channels.SocketChannel, sets it 
to blocking mode, and invokes SocketChannel.connect(), passing the current 
broker hostname. 

If the first broker in the list is nonresponsive, SocketChannel.connect() will 
wait forever. 

I think the correct change is as follows:

diff --git a/core/src/main/scala/kafka/network/BlockingChannel.scala 
b/core/src/main/scala/kafka/network/BlockingChannel.scala
index eb7bb14..9bb102a 100644
--- a/core/src/main/scala/kafka/network/BlockingChannel.scala
+++ b/core/src/main/scala/kafka/network/BlockingChannel.scala
@@ -55,7 +55,7 @@ class BlockingChannel( val host: String,
 channel.socket.setSoTimeout(readTimeoutMs)
 channel.socket.setKeepAlive(true)
 channel.socket.setTcpNoDelay(true)
-channel.connect(new InetSocketAddress(host, port))
+channel.socket.connect(new InetSocketAddress(host, port), 
connectTimeoutMs)

 writeChannel = channel
 readChannel = Channels.newChannel(channel.socket().getInputStream)

Is the next step to create a JIRA with this information? Thanks.

-- 
Jack Foy 

[jira] [Created] (KAFKA-1637) SimpleConsumer.fetchOffset returns wrong error code

[Amir Malekpour (JIRA)]

Amir Malekpour created KAFKA-1637:
-

 Summary: SimpleConsumer.fetchOffset returns wrong error code
 Key: KAFKA-1637
 URL: https://issues.apache.org/jira/browse/KAFKA-1637
 Project: Kafka
  Issue Type: Bug
  Components: consumer, core
Affects Versions: 0.8.1.1, 0.8.1
 Environment: Linux
Reporter: Amir Malekpour
Assignee: Neha Narkhede


This concerns Kafka's Offset  Fetch API:

According to Kafka's current documentation, "if there is no offset associated 
with a topic-partition under that consumer group the broker does not set an 
error code (since it is not really an error), but returns empty metadata and 
sets the offset field to -1."  (Like below)

However, in Kafka 08.1.1 Error code '3' is returned, which effectively makes it 
impossible for the client to decide if there was an error, or if there is no 
offset associated with a topic-partition under that consumer group.


https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-MetadataAPI




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1637) SimpleConsumer.fetchOffset returns wrong error code when no offset exists for topic/partition/consumer group

[Amir Malekpour (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amir Malekpour updated KAFKA-1637:
--
Summary: SimpleConsumer.fetchOffset returns wrong error code when no offset 
exists for topic/partition/consumer group  (was: SimpleConsumer.fetchOffset 
returns wrong error code)

> SimpleConsumer.fetchOffset returns wrong error code when no offset exists for 
> topic/partition/consumer group
> 
>
> Key: KAFKA-1637
> URL: https://issues.apache.org/jira/browse/KAFKA-1637
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, core
>Affects Versions: 0.8.1, 0.8.1.1
> Environment: Linux
>Reporter: Amir Malekpour
>Assignee: Neha Narkhede
>
> This concerns Kafka's Offset  Fetch API:
> According to Kafka's current documentation, "if there is no offset associated 
> with a topic-partition under that consumer group the broker does not set an 
> error code (since it is not really an error), but returns empty metadata and 
> sets the offset field to -1."  (Like below)
> However, in Kafka 08.1.1 Error code '3' is returned, which effectively makes 
> it impossible for the client to decide if there was an error, or if there is 
> no offset associated with a topic-partition under that consumer group.
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-MetadataAPI



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1637) SimpleConsumer.fetchOffset returns wrong error code when no offset exists for topic/partition/consumer group

[Amir Malekpour (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amir Malekpour updated KAFKA-1637:
--
Description: 
This concerns Kafka's Offset  Fetch API:

According to Kafka's current documentation, "if there is no offset associated 
with a topic-partition under that consumer group the broker does not set an 
error code (since it is not really an error), but returns empty metadata and 
sets the offset field to -1."  (Link below)

However, in Kafka 08.1.1 Error code '3' is returned, which effectively makes it 
impossible for the client to decide if there was an error, or if there is no 
offset associated with a topic-partition under that consumer group.


https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-MetadataAPI


  was:
This concerns Kafka's Offset  Fetch API:

According to Kafka's current documentation, "if there is no offset associated 
with a topic-partition under that consumer group the broker does not set an 
error code (since it is not really an error), but returns empty metadata and 
sets the offset field to -1."  (Like below)

However, in Kafka 08.1.1 Error code '3' is returned, which effectively makes it 
impossible for the client to decide if there was an error, or if there is no 
offset associated with a topic-partition under that consumer group.


https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-MetadataAPI



> SimpleConsumer.fetchOffset returns wrong error code when no offset exists for 
> topic/partition/consumer group
> 
>
> Key: KAFKA-1637
> URL: https://issues.apache.org/jira/browse/KAFKA-1637
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, core
>Affects Versions: 0.8.1, 0.8.1.1
> Environment: Linux
>Reporter: Amir Malekpour
>Assignee: Neha Narkhede
>
> This concerns Kafka's Offset  Fetch API:
> According to Kafka's current documentation, "if there is no offset associated 
> with a topic-partition under that consumer group the broker does not set an 
> error code (since it is not really an error), but returns empty metadata and 
> sets the offset field to -1."  (Link below)
> However, in Kafka 08.1.1 Error code '3' is returned, which effectively makes 
> it impossible for the client to decide if there was an error, or if there is 
> no offset associated with a topic-partition under that consumer group.
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-MetadataAPI



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1620) Make kafka api protocol implementation public

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136754#comment-14136754
 ] 

Jun Rao commented on KAFKA-1620:


Thanks for the new patch. Similarly, do we need to make RequestOrResponse 
public since it's just an abstract base class?

> Make kafka api protocol implementation public
> -
>
> Key: KAFKA-1620
> URL: https://issues.apache.org/jira/browse/KAFKA-1620
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Anton Karamanov
>Assignee: Anton Karamanov
> Attachments: 
> 0001-KAFKA-1620-Make-kafka-api-protocol-implementation-pu.patch, 
> 0002-KAFKA-1620-Make-kafka-api-protocol-implementation-pu.patch
>
>
> Some of the classes which implement Kafka api protocol, such as 
> {{RequestOrResponse}} and {{FetchRequest}} are defined as private to 
> {{kafka}} package. Those classes would be extremely usefull for writing 
> custom clients (we're using Scala with Akka and implementing one directly on 
> top of Akka TCP), and don't seem to contain any actuall internal logic of 
> Kafka. Therefore it seems like a nice idea to make them public.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1282) Disconnect idle socket connection in Selector

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136767#comment-14136767
 ] 

Jun Rao commented on KAFKA-1282:


Did you do "./gradlew jar" first?

> Disconnect idle socket connection in Selector
> -
>
> Key: KAFKA-1282
> URL: https://issues.apache.org/jira/browse/KAFKA-1282
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: nicu marasoiu
>  Labels: newbie++
> Fix For: 0.9.0
>
> Attachments: 1282_the_ultimate,_close_max_one_per_select.patch, 
> KAFKA-1282_Disconnect_idle_socket_connection_in_Selector.patch, 
> idleDisconnect.patch
>
>
> To reduce # socket connections, it would be useful for the new producer to 
> close socket connections that are idle. We can introduce a new producer 
> config for the idle time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: kafka 0.8.1: Producer.send() can block forever when a broker is down

[Jun Rao]

Jack,

If the broker is down, channel.connect() should throw an IOException,
instead of blocking forever. In your case, is the broker host down? In that
case, the connect call will likely wait for the default tcp connection
timeout, which is 8+ mins.

Thanks,

Jun

On Tue, Sep 16, 2014 at 5:43 PM, Jack Foy  wrote:

> We observe that when a broker is down, Producer.send() can get into a
> state where it will block forever, even when using the async producer.
>
> When a Producer first sends data, it fetches topic metadata from the
> broker cluster. To do this, it shuffles the list of hosts in the cluster,
> then iterates through the list querying each broker.
>
> For each broker in the shuffled list, the Producer creates a SyncProducer
> and invokes SyncProducer.send().
> SyncProducer.send() creates a BlockingChannel and invokes
> BlockingChannel.connect().
> BlockingChannel.connect() retrieves a java.nio.channels.SocketChannel,
> sets it to blocking mode, and invokes SocketChannel.connect(), passing the
> current broker hostname.
>
> If the first broker in the list is nonresponsive, SocketChannel.connect()
> will wait forever.
>
> I think the correct change is as follows:
>
> diff --git a/core/src/main/scala/kafka/network/BlockingChannel.scala
> b/core/src/main/scala/kafka/network/BlockingChannel.scala
> index eb7bb14..9bb102a 100644
> --- a/core/src/main/scala/kafka/network/BlockingChannel.scala
> +++ b/core/src/main/scala/kafka/network/BlockingChannel.scala
> @@ -55,7 +55,7 @@ class BlockingChannel( val host: String,
>  channel.socket.setSoTimeout(readTimeoutMs)
>  channel.socket.setKeepAlive(true)
>  channel.socket.setTcpNoDelay(true)
> -channel.connect(new InetSocketAddress(host, port))
> +channel.socket.connect(new InetSocketAddress(host, port),
> connectTimeoutMs)
>
>  writeChannel = channel
>  readChannel = Channels.newChannel(channel.socket().getInputStream)
>
> Is the next step to create a JIRA with this information? Thanks.
>
> --
> Jack Foy 
>
>
>
>

[jira] [Resolved] (KAFKA-1635) Java doc of makeLeaders in ReplicaManager is wrong

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-1635.

   Resolution: Fixed
Fix Version/s: 0.8.2
 Assignee: Lantao Jin  (was: Neha Narkhede)

Thanks for the patch. +1 and committed to trunk.

> Java doc of makeLeaders in ReplicaManager is wrong
> --
>
> Key: KAFKA-1635
> URL: https://issues.apache.org/jira/browse/KAFKA-1635
> Project: Kafka
>  Issue Type: Bug
>  Components: core, replication
>Reporter: Lantao Jin
>Assignee: Lantao Jin
>Priority: Trivial
>  Labels: doc, server
> Fix For: 0.8.2
>
> Attachments: kafka-1635-1.patch
>
>
> ReplicaManager have an incorrect java doc. The overview of function  
> makeLeaders() is the same as makeFollowers().
> Also see commit at 
> https://github.com/apache/kafka/commit/6739a8e601331ad07d9856dc351785351755a5d5



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: Kafka-trunk #267

[Apache Jenkins Server]

See 

Changes:

[junrao] trivial change to javadoc for makeLeaders(); patched by Lantao Jin

--
[...truncated 2717 lines...]
at sun.nio.ch.Net.bind(Native Method)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at 
org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:95)
at kafka.zk.EmbeddedZookeeper.(EmbeddedZookeeper.scala:34)
at 
kafka.zk.ZooKeeperTestHarness$class.setUp(ZooKeeperTestHarness.scala:33)
at 
kafka.integration.PrimitiveApiTest.kafka$integration$KafkaServerTestHarness$$super$setUp(PrimitiveApiTest.scala:39)
at 
kafka.integration.KafkaServerTestHarness$class.setUp(KafkaServerTestHarness.scala:36)
at 
kafka.integration.PrimitiveApiTest.kafka$integration$ProducerConsumerTestHarness$$super$setUp(PrimitiveApiTest.scala:39)
at 
kafka.integration.ProducerConsumerTestHarness$class.setUp(ProducerConsumerTestHarness.scala:33)
at kafka.integration.PrimitiveApiTest.setUp(PrimitiveApiTest.scala:39)

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic FAILED
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at 
org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:95)
at kafka.zk.EmbeddedZookeeper.(EmbeddedZookeeper.scala:34)
at 
kafka.zk.ZooKeeperTestHarness$class.setUp(ZooKeeperTestHarness.scala:33)
at 
kafka.integration.PrimitiveApiTest.kafka$integration$KafkaServerTestHarness$$super$setUp(PrimitiveApiTest.scala:39)
at 
kafka.integration.KafkaServerTestHarness$class.setUp(KafkaServerTestHarness.scala:36)
at 
kafka.integration.PrimitiveApiTest.kafka$integration$ProducerConsumerTestHarness$$super$setUp(PrimitiveApiTest.scala:39)
at 
kafka.integration.ProducerConsumerTestHarness$class.setUp(ProducerConsumerTestHarness.scala:33)
at kafka.integration.PrimitiveApiTest.setUp(PrimitiveApiTest.scala:39)

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests FAILED
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at 
org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:95)
at kafka.zk.EmbeddedZookeeper.(EmbeddedZookeeper.scala:34)
at 
kafka.zk.ZooKeeperTestHarness$class.setUp(ZooKeeperTestHarness.scala:33)
at 
kafka.integration.PrimitiveApiTest.kafka$integration$KafkaServerTestHarness$$super$setUp(PrimitiveApiTest.scala:39)
at 
kafka.integration.KafkaServerTestHarness$class.setUp(KafkaServerTestHarness.scala:36)
at 
kafka.integration.PrimitiveApiTest.kafka$integration$ProducerConsumerTestHarness$$super$setUp(PrimitiveApiTest.scala:39)
at 
kafka.integration.ProducerConsumerTestHarness$class.setUp(ProducerConsumerTestHarness.scala:33)
at kafka.integration.PrimitiveApiTest.setUp(PrimitiveApiTest.scala:39)

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooHigh 
FAILED
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at 
org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:95)
at kafka.zk.EmbeddedZookeeper.(EmbeddedZookeeper.scala:34)
at 
kafka.zk.ZooKeeperTestHarness$class.setUp(ZooKeeperTestHarness.scala:33)
at 
kafka.integration.AutoOffsetResetTest.kafka$integration$KafkaServerTestHarness$$super$setUp(AutoOffsetResetTest.scala:32)
at 
kafka.integration.KafkaServerTestHarness$class.setUp(KafkaServerTestHarness.scala:36)
at 
kafka.integration.AutoOffsetResetTest.setUp(AutoOffsetResetTest.scala:46)

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
FAILED
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
at sun.nio.ch.Serv

[jira] [Commented] (KAFKA-1620) Make kafka api protocol implementation public

[Anton Karamanov (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136817#comment-14136817
 ] 

Anton Karamanov commented on KAFKA-1620:


Yes, it is actually one of the primary motives for this issue, since it allows 
to abstract from specific requests/responses on a low level. Without a common 
class any generic collection of messages would have to have a type of 
{{Collection[Any]}} reducing type safety and making it harder to control the 
flow of requests/responses.

> Make kafka api protocol implementation public
> -
>
> Key: KAFKA-1620
> URL: https://issues.apache.org/jira/browse/KAFKA-1620
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Anton Karamanov
>Assignee: Anton Karamanov
> Attachments: 
> 0001-KAFKA-1620-Make-kafka-api-protocol-implementation-pu.patch, 
> 0002-KAFKA-1620-Make-kafka-api-protocol-implementation-pu.patch
>
>
> Some of the classes which implement Kafka api protocol, such as 
> {{RequestOrResponse}} and {{FetchRequest}} are defined as private to 
> {{kafka}} package. Those classes would be extremely usefull for writing 
> custom clients (we're using Scala with Akka and implementing one directly on 
> top of Akka TCP), and don't seem to contain any actuall internal logic of 
> Kafka. Therefore it seems like a nice idea to make them public.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)