Re: blog with some "out of the box" pains

The first issue he runs into is one I also find frustrating -- with cloud providers pushing SSDs, you have to use a pretty large instance type to get a reasonable test setup. I'm not sure if he couldn't launch an older type like m1.large (I think some newer AWS accounts aren't able to) or if he jus

Re: Kafka Command Line & Shell

+1 to http interface and cli. Http layer will make it easier to integrate with gui like Hue. Gwen — Sent from Mailbox On Fri, Oct 17, 2014 at 10:14 PM, Todd Palino wrote: > Absolutely. My suggestion of an HTTP interface was in addition to a CLI. I > think the CLI should can the HTTP interfac

Re: Kafka Command Line & Shell

Absolutely. My suggestion of an HTTP interface was in addition to a CLI. I think the CLI should can the HTTP interface underneath to keep it simple. -Todd > On Oct 17, 2014, at 6:24 PM, Joel Koshy wrote: > > +1 > It would definitely be useful to have a CLI. We had a cursory > discussion on t

Re: Kafka Command Line & Shell

+1 It would definitely be useful to have a CLI. We had a cursory discussion on this in the past [1] but it would be useful to have a full proposal describing everything the CLI should provide. [1] http://grokbase.com/t/kafka/dev/1435tr3pfc/command-line-tools On Fri, Oct 17, 2014 at 05:12:16PM -

[jira] [Commented] (KAFKA-1583) Kafka API Refactoring

[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14175763#comment-14175763 ] Joel Koshy commented on KAFKA-1583: --- Reviewing is taking a bit longer than expected - I'

Re: Kafka Command Line & Shell

+100 -- Colin Clark +1-320-221-9531 > On Oct 17, 2014, at 7:10 PM, Steve Morin wrote: > > Joe I think this is great! > >> On Fri, Oct 17, 2014 at 5:03 PM, Joe Stein wrote: >> >> Hi, I have been thinking about the ease of use for operations with Kafka. >> We have lots of tools doing a lot

Re: Kafka Command Line & Shell

We've been talking about this a little internally as well. What about the idea of presenting all the admin functions through a web API interface (restful or not) complete with authentication? That would make it much easier for creating structure around Kafka without having to layer commands on t

Re: blog with some "out of the box" pains

Basically, the issue (or at least one of very many possible network issues...) is that the server has "localhost" hardcoded as its canonical name in /etc/hosts: [root@Billc-cent70x64 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 Billc-cent70x64 :

Re: Kafka Command Line & Shell

Joe I think this is great! On Fri, Oct 17, 2014 at 5:03 PM, Joe Stein wrote: > Hi, I have been thinking about the ease of use for operations with Kafka. > We have lots of tools doing a lot of different things and they are all kind > of in different places. > > So, what I was thinking is to have

Kafka Command Line & Shell

Hi, I have been thinking about the ease of use for operations with Kafka. We have lots of tools doing a lot of different things and they are all kind of in different places. So, what I was thinking is to have a single interface for our tooling https://issues.apache.org/jira/browse/KAFKA-1694 This

Re: blog with some "out of the box" pains

It looks like we are using canonical hostname: def register() { val advertisedHostName = if(advertisedHost == null || advertisedHost.trim.isEmpty) InetAddress.getLocalHost.getCanonicalHostName else advertisedHost val jmxPort = System.getProperty("com.sun.manage

Re: blog with some "out of the box" pains

Hmm, yes, actually I don't think I actually understand the issue. Basically as I understand it we do InetAddress.getLocalHost.getHostAddress which on AWS picks the wrong hostname/ip and then the producer can't connect. People eventually find this FAQ, but I was hoping there was a more automatic way

[jira] [Closed] (KAFKA-1713) Partition files not created

[ https://issues.apache.org/jira/browse/KAFKA-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede closed KAFKA-1713. > Partition files not created > --- > > Key: KAFKA-1713 >

[jira] [Resolved] (KAFKA-1713) Partition files not created

[ https://issues.apache.org/jira/browse/KAFKA-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede resolved KAFKA-1713. -- Resolution: Won't Fix Please can you redirect such questions to the mailing list? Regarding your

[jira] [Created] (KAFKA-1713) Partition files not created

Pradeep created KAFKA-1713: -- Summary: Partition files not created Key: KAFKA-1713 URL: https://issues.apache.org/jira/browse/KAFKA-1713 Project: Kafka Issue Type: Bug Components: core

Jenkins build is back to normal : Kafka-trunk #307

See

Re: blog with some "out of the box" pains

In #2, do you refer to advertising the "internal" hostname instead of the external one? In this case, will it be enough to use getCanonicalHostName (which uses a name service)? Note that I think the problem the blog reported (wrong name advertised) is somewhat orthogonal to the question of which i

[jira] [Updated] (KAFKA-1108) when controlled shutdown attempt fails, the reason is not always logged

[ https://issues.apache.org/jira/browse/KAFKA-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1108: - Fix Version/s: (was: 0.9.0) 0.8.2 > when controlled shutdown attempt fails,

[jira] [Updated] (KAFKA-1108) when controlled shutdown attempt fails, the reason is not always logged

[ https://issues.apache.org/jira/browse/KAFKA-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1108: - Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the patch. Pushed to

[jira] [Commented] (KAFKA-1710) [New Java Producer Potential Deadlock] Producer Deadlock when all messages is being sent to single partition

[ https://issues.apache.org/jira/browse/KAFKA-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14175586#comment-14175586 ] Jay Kreps commented on KAFKA-1710: -- Well but of course you can't have multiple threads ap

[jira] [Comment Edited] (KAFKA-1710) [New Java Producer Potential Deadlock] Producer Deadlock when all messages is being sent to single partition

[ https://issues.apache.org/jira/browse/KAFKA-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14175574#comment-14175574 ] Bhavesh Mistry edited comment on KAFKA-1710 at 10/17/14 9:32 PM: ---

[jira] [Commented] (KAFKA-1710) [New Java Producer Potential Deadlock] Producer Deadlock when all messages is being sent to single partition

[ https://issues.apache.org/jira/browse/KAFKA-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14175574#comment-14175574 ] Bhavesh Mistry commented on KAFKA-1710: --- [~jkreps], Only request is to make New Pro

blog with some "out of the box" pains

This guy documented a few struggles getting going with Kafka. Not sure if there is anything we can do to make it better? http://ispyker.blogspot.com/2014/10/kafka-part-1.html 1. Would be great to figure out the apache/gradle thing. 2. The problem of having Kafka advertise localhost on AWS is reall

[jira] [Comment Edited] (KAFKA-1710) [New Java Producer Potential Deadlock] Producer Deadlock when all messages is being sent to single partition

[ https://issues.apache.org/jira/browse/KAFKA-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14175548#comment-14175548 ] Bhavesh Mistry edited comment on KAFKA-1710 at 10/17/14 9:19 PM: ---

[jira] [Commented] (KAFKA-1710) [New Java Producer Potential Deadlock] Producer Deadlock when all messages is being sent to single partition

[ https://issues.apache.org/jira/browse/KAFKA-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14175551#comment-14175551 ] Jay Kreps commented on KAFKA-1710: -- [~Bmis13] What is the performance you see? What do yo

[jira] [Commented] (KAFKA-1710) [New Java Producer Potential Deadlock] Producer Deadlock when all messages is being sent to single partition

[ https://issues.apache.org/jira/browse/KAFKA-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14175548#comment-14175548 ] Bhavesh Mistry commented on KAFKA-1710: --- [~ewencp], Thank you for entertaining this

[jira] [Commented] (KAFKA-1476) Get a list of consumer groups

[ https://issues.apache.org/jira/browse/KAFKA-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14175503#comment-14175503 ] BalajiSeshadri commented on KAFKA-1476: --- [~nehanarkhede] Please find the attached co

[jira] [Resolved] (KAFKA-1710) [New Java Producer Potential Deadlock] Producer Deadlock when all messages is being sent to single partition

[ https://issues.apache.org/jira/browse/KAFKA-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava resolved KAFKA-1710. -- Resolution: Invalid Assignee: Ewen Cheslack-Postava > [New Java Producer P

[jira] [Updated] (KAFKA-1476) Get a list of consumer groups

[ https://issues.apache.org/jira/browse/KAFKA-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BalajiSeshadri updated KAFKA-1476: -- Attachment: ConsumerCommand.scala > Get a list of consumer groups >

[jira] [Commented] (KAFKA-1476) Get a list of consumer groups

[ https://issues.apache.org/jira/browse/KAFKA-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14175500#comment-14175500 ] BalajiSeshadri commented on KAFKA-1476: --- [~nehanarkhede] I tried to implement the de

Build failed in Jenkins: Kafka-trunk #306

See Changes: [junrao] kafka-1671; uploaded archives are missing for Scala version 2.11; patched by Ivan Lyutov; reviewed by Jun Rao -- [...truncated 1011 lines...] kafka.producer.ProducerTest > testS

[jira] [Commented] (KAFKA-1710) [New Java Producer Potential Deadlock] Producer Deadlock when all messages is being sent to single partition

[ https://issues.apache.org/jira/browse/KAFKA-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14175430#comment-14175430 ] Ewen Cheslack-Postava commented on KAFKA-1710: -- bq. The dead lock will occur

[jira] [Updated] (KAFKA-1671) uploaded archives are missing for Scala version 2.11

[ https://issues.apache.org/jira/browse/KAFKA-1671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1671: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the patch. +1. Committed to trun

[jira] [Created] (KAFKA-1712) Excessive storage usage on newly added node

Oleg Golovin created KAFKA-1712: --- Summary: Excessive storage usage on newly added node Key: KAFKA-1712 URL: https://issues.apache.org/jira/browse/KAFKA-1712 Project: Kafka Issue Type: Bug

[jira] [Commented] (KAFKA-1485) Upgrade to Zookeeper 3.4.6 and create "shim" for ZKCLI so system tests can run

[ https://issues.apache.org/jira/browse/KAFKA-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14175365#comment-14175365 ] Jun Rao commented on KAFKA-1485: Gwen, Joe, It seems that ZK 3.4.6 added a runtime depend

Review Request 26885: Patch for KAFKA-1642

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26885/ --- Review request for kafka. Bugs: KAFKA-1642 https://issues.apache.org/jira/b

[jira] [Updated] (KAFKA-1642) [Java New Producer Kafka Trunk] CPU Usage Spike to 100% when network connection is lost

[ https://issues.apache.org/jira/browse/KAFKA-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-1642: - Attachment: KAFKA-1642.patch > [Java New Producer Kafka Trunk] CPU Usage Spike to

[jira] [Commented] (KAFKA-1642) [Java New Producer Kafka Trunk] CPU Usage Spike to 100% when network connection is lost

[ https://issues.apache.org/jira/browse/KAFKA-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14175360#comment-14175360 ] Ewen Cheslack-Postava commented on KAFKA-1642: -- Created reviewboard https://r

[jira] [Updated] (KAFKA-1642) [Java New Producer Kafka Trunk] CPU Usage Spike to 100% when network connection is lost

[ https://issues.apache.org/jira/browse/KAFKA-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-1642: - Assignee: Ewen Cheslack-Postava (was: Jun Rao) Status: Patch Available (was

Re: Review Request 24676: Rebase KAFKA-1583

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24676/#review57178 --- Thanks for the patch. +1 after addressing a couple of more minor com

Re: Security JIRAS

Yes, I think we can focus on Broker to Zookeeper communication only. At least for initial stage. Gwen On Fri, Oct 17, 2014 at 2:10 PM, Todd Palino wrote: > For the moment, consumers still need to write under the /consumers tree. > Even if they are committing offsets to Kafka instead of ZK, they

Re: Security JIRAS

For the moment, consumers still need to write under the /consumers tree. Even if they are committing offsets to Kafka instead of ZK, they will need to write owner information there when they are balancing. Eventually, you are correct, this is going away with the new consumer. -Todd On Fri, Oct 17

Jenkins build is back to normal : Kafka-trunk #305

See

[jira] [Updated] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option

[ https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1493: --- Resolution: Fixed Status: Resolved (was: Patch Available) James, Thanks a lot for the patch. +1 and

RE: Security JIRAS

I'm looking at Kafka Brokers authentication with ZooKeeper since this looks independent of other tasks. [AM] 1) Is authentication required only between kafka broker and zookeeper? Can we assume "world" read so that consumers don't have to be authenticated (I believe in any case kafka is plann

[jira] [Commented] (KAFKA-1583) Kafka API Refactoring

[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14175239#comment-14175239 ] Guozhang Wang commented on KAFKA-1583: -- Updated reviewboard https://reviews.apache.or

[jira] [Created] (KAFKA-1711) WARN Property topic is not valid when running console producer

Jun Rao created KAFKA-1711: -- Summary: WARN Property topic is not valid when running console producer Key: KAFKA-1711 URL: https://issues.apache.org/jira/browse/KAFKA-1711 Project: Kafka Issue Type:

[jira] [Updated] (KAFKA-1583) Kafka API Refactoring

[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1583: - Attachment: KAFKA-1583_2014-10-17_09:56:33.patch > Kafka API Refactoring > - >

Re: Review Request 24676: Rebase KAFKA-1583

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24676/ --- (Updated Oct. 17, 2014, 4:56 p.m.) Review request for kafka. Summary (updated

Re: Review Request 24676: Fix KAFKA-1583

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24676/#review57147 --- Thanks for the patch. A few more comments. core/src/main/scala/kaf