I started writing this up on the wiki https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements
Instead of starting a new thread I figure just continue this one I started. I also added another (important) component for centralized management of configuration as "global level" much like we have topic level. These global configuration would be overridden (perhaps not all) from the server.properties on start (so like in case one broker needs a different port, sure). <<One concern I have is that using RQ/RP wire protocol to the controller instead of the current way (via ZK admin path) may expose concurrency on the admin requests, which may not be supported yet. Guozhang, take a look at the diagram how I am thinking of this it would be a new handle request that will execute the tools pretty much how they are today. My thinking is maybe to-do one at a time (so TopicCommand first I think) and have what the TopicCommand is doing happen on server and send the RQ/RP to the client but execute on the server. If there is something not supported we will of course have to deal with that and implement it for sure. Once we get one working end to end I think adding the rest will be (more or less) concise iterations to get it done. I added your concern to the wiki under the "gotchas" section. /******************************************* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> ********************************************/ On Mon, Oct 20, 2014 at 2:15 AM, Guozhang Wang <wangg...@gmail.com> wrote: > One concern I have is that using RQ/RP wire protocol to the controller > instead of the current way (via ZK admin path) may expose concurrency on > the admin requests, which may not be supported yet. > > Some initial discussion about this is on KAFKA-1305. > > Guozhang > > On Sun, Oct 19, 2014 at 1:55 PM, Joe Stein <joe.st...@stealth.ly> wrote: > > > Maybe we should add some AdminMessage RQ/RP wire protocol structure(s) > and > > let the controller handle it? We could then build the CLI and Shell in > the > > project both as useful tools and samples for others. > > > > Making a http interface should be simple after KAFKA-1494 is done which > all > > client libraries could offer. > > > > I will update the design tonight/tomorrow and should be able to have > > someone starting to work on it this week. > > > > /******************************************* > > Joe Stein > > Founder, Principal Consultant > > Big Data Open Source Security LLC > > http://www.stealth.ly > > Twitter: @allthingshadoop > > ********************************************/ > > On Oct 19, 2014 1:21 PM, "Harsha" <ka...@harsha.io> wrote: > > > > > +1 for Web Api > > > > > > On Sat, Oct 18, 2014, at 11:48 PM, Glen Mazza wrote: > > > > Apache Karaf has been doing this for quite a few years, albeit in > Java > > > > not Scala. Still, their coding approach to creating a CLI probably > > > > captures many lessons learned over that time. > > > > > > > > Glen > > > > > > > > On 10/17/2014 08:03 PM, Joe Stein wrote: > > > > > Hi, I have been thinking about the ease of use for operations with > > > Kafka. > > > > > We have lots of tools doing a lot of different things and they are > > all > > > kind > > > > > of in different places. > > > > > > > > > > So, what I was thinking is to have a single interface for our > tooling > > > > > https://issues.apache.org/jira/browse/KAFKA-1694 > > > > > > > > > > This would manifest itself in two ways 1) a command line interface > 2) > > > a repl > > > > > > > > > > We would have one entry point centrally for all Kafka commands. > > > > > kafka <CMD> <ARGS> > > > > > kafka createTopic --brokerList etc, > > > > > kafka reassignPartition --brokerList etc, > > > > > > > > > > or execute and run the shell > > > > > > > > > > kafka --brokerList localhost > > > > > kafka>use topicName; > > > > > kafka>set acl='label'; > > > > > > > > > > I was thinking that all calls would be initialized through > > > --brokerList and > > > > > the broker can tell the KafkaCommandTool what server to connect to > > for > > > > > MetaData. > > > > > > > > > > Thoughts? Tomatoes? > > > > > > > > > > /******************************************* > > > > > Joe Stein > > > > > Founder, Principal Consultant > > > > > Big Data Open Source Security LLC > > > > > http://www.stealth.ly > > > > > Twitter: @allthingshadoop < > http://www.twitter.com/allthingshadoop> > > > > > ********************************************/ > > > > > > > > > > > > > > > > > > -- > -- Guozhang >
