Re: Kafka Command Line & Shell

2014-10-19 Thread Harsha
+1 for Web Api On Sat, Oct 18, 2014, at 11:48 PM, Glen Mazza wrote: > Apache Karaf has been doing this for quite a few years, albeit in Java > not Scala. Still, their coding approach to creating a CLI probably > captures many lessons learned over that time. > > Glen > > On 10/17/2014 08:03 PM

Re: Kafka Command Line & Shell

2014-10-19 Thread Joe Stein
Maybe we should add some AdminMessage RQ/RP wire protocol structure(s) and let the controller handle it? We could then build the CLI and Shell in the project both as useful tools and samples for others. Making a http interface should be simple after KAFKA-1494 is done which all client libraries co

Re: [DISCUSSION] Message Metadata

2014-10-19 Thread Jun Rao
Hi, Guozhang, Thanks for the writeup. A few high level comments. 1. Associating (versioned) schemas to a topic can be a good thing overall. Yes, this could add a bit more management overhead in Kafka. However, it makes sure that the data format contract between a producer and a consumer is kept

[jira] [Created] (KAFKA-1716) hang during shutdown of ZookeeperConsumerConnector

2014-10-19 Thread Sean Fay (JIRA)
Sean Fay created KAFKA-1716: --- Summary: hang during shutdown of ZookeeperConsumerConnector Key: KAFKA-1716 URL: https://issues.apache.org/jira/browse/KAFKA-1716 Project: Kafka Issue Type: Bug

Re: [DISCUSSION] Message Metadata

2014-10-19 Thread Todd Palino
A couple notes on this 1 / 2 - Message format should not be important to Kafka or the common infrastructure (audit, mirror maker). As I have noted elsewhere, having to ensure the data contract just to provide a general messaging service is additional overhead that is not always needed. It eith

[jira] [Commented] (KAFKA-1716) hang during shutdown of ZookeeperConsumerConnector

2014-10-19 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14176624#comment-14176624 ] Guozhang Wang commented on KAFKA-1716: -- This may be resolved in KAFKA-1305. Did you p

Re: Kafka Command Line & Shell

2014-10-19 Thread Guozhang Wang
One concern I have is that using RQ/RP wire protocol to the controller instead of the current way (via ZK admin path) may expose concurrency on the admin requests, which may not be supported yet. Some initial discussion about this is on KAFKA-1305. Guozhang On Sun, Oct 19, 2014 at 1:55 PM, Joe S

Re: [DISCUSSION] Message Metadata

2014-10-19 Thread Guozhang Wang
Thanks for the detailed comments Jun! Some replies inlined. On Sun, Oct 19, 2014 at 7:42 PM, Jun Rao wrote: > Hi, Guozhang, > > Thanks for the writeup. > > A few high level comments. > > 1. Associating (versioned) schemas to a topic can be a good thing overall. > Yes, this could add a bit more m