A centralized admin operation protocol would be very useful. One more general comment here is that controller is originally designed to only talk to other brokers through ControllerChannel, while the broker instance which carries the current controller is agnostic of its existence, and use KafkaApis to handle general Kafka requests. Having all admin requests redirected to the controller instance will force the broker to be aware of its carried controller, and access its internal data for handling these requests. Plus with the number of clients out of Kafka's control, this may easily cause the controller to be a hot spot in terms of request load.
On Thu, Jan 22, 2015 at 10:09 PM, Joe Stein <joe.st...@stealth.ly> wrote: > inline > > On Thu, Jan 22, 2015 at 11:59 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > > > Hey Joe, > > > > This is great. A few comments on KIP-4 > > > > 1. This is much needed functionality, but there are a lot of the so let's > > really think these protocols through. We really want to end up with a set > > of well thought-out, orthoganol apis. For this reason I think it is > really > > important to think through the end state even if that includes APIs we > > won't implement in the first phase. > > > > ok > > > > > > 2. Let's please please please wait until we have switched the server over > > to the new java protocol definitions. If we add upteen more ad hoc scala > > objects that is just generating more work for the conversion we know we > > have to do. > > > > ok :) > > > > > > 3. This proposal introduces a new type of optional parameter. This is > > inconsistent with everything else in the protocol where we use -1 or some > > other marker value. You could argue either way but let's stick with that > > for consistency. For clients that implemented the protocol in a better > way > > than our scala code these basic primitives are hard to change. > > > > yes, less confusing, ok. > > > > > > 4. ClusterMetadata: This seems to duplicate TopicMetadataRequest which > has > > brokers, topics, and partitions. I think we should rename that request > > ClusterMetadataRequest (or just MetadataRequest) and include the id of > the > > controller. Or are there other things we could add here? > > > > We could add broker version to it. > > > > > > 5. We have a tendency to try to make a lot of requests that can only go > to > > particular nodes. This adds a lot of burden for client implementations > (it > > sounds easy but each discovery can fail in many parts so it ends up > being a > > full state machine to do right). I think we should consider making admin > > commands and ideally as many of the other apis as possible available on > all > > brokers and just redirect to the controller on the broker side. Perhaps > > there would be a general way to encapsulate this re-routing behavior. > > > > If we do that then we should also preserve what we have and do both. The > client can then decide "do I want to go to any broker and proxy" or just > "go to controller and run admin task". Lots of folks have seen controllers > come under distress because of their producers/consumers. There is ticket > too for controller elect and re-elect > https://issues.apache.org/jira/browse/KAFKA-1778 so you can force it to a > broker that has 0 load. > > > > > > 6. We should probably normalize the key value pairs used for configs > rather > > than embedding a new formatting. So two strings rather than one with an > > internal equals sign. > > > > ok > > > > > > 7. Is the postcondition of these APIs that the command has begun or that > > the command has been completed? It is a lot more usable if the command > has > > been completed so you know that if you create a topic and then publish to > > it you won't get an exception about there being no such topic. > > > > We should define that more. There needs to be some more state there, yes. > > We should try to cover https://issues.apache.org/jira/browse/KAFKA-1125 > within what we come up with. > > > > > > 8. Describe topic and list topics duplicate a lot of stuff in the > metadata > > request. Is there a reason to give back topics marked for deletion? I > feel > > like if we just make the post-condition of the delete command be that the > > topic is deleted that will get rid of the need for this right? And it > will > > be much more intuitive. > > > > I will go back and look through it. > > > > > > 9. Should we consider batching these requests? We have generally tried to > > allow multiple operations to be batched. My suspicion is that without > this > > we will get a lot of code that does something like > > for(topic: adminClient.listTopics()) > > adminClient.describeTopic(topic) > > this code will work great when you test on 5 topics but not do as well if > > you have 50k. > > > > So => Input is a list of topics (or none for all) and a batch response from > the controller (which could be routed through another broker) of the entire > response? We could introduce a Batch keyword to explicitly show the usage > of it. > > > > 10. I think we should also discuss how we want to expose a programmatic > JVM > > client api for these operations. Currently people rely on AdminUtils > which > > is totally sketchy. I think we probably need another client under > clients/ > > that exposes administrative functionality. We will need this just to > > properly test the new apis, I suspect. We should figure out that API. > > > > We were talking about that here > https://issues.apache.org/jira/browse/KAFKA-1774 and wrote it in java > https://reviews.apache.org/r/29301/diff/7/?page=4#75 so we could do > something like that, sure. > > > > > > 11. The other information that would be really useful to get would be > > information about partitions--how much data is in the partition, what are > > the segment offsets, what is the log-end offset (i.e. last offset), what > is > > the compaction point, etc. I think that done right this would be the > > successor to the very awkward OffsetRequest we have today. > > > > yes! > > > > > > -Jay > > > > On Wed, Jan 21, 2015 at 10:27 PM, Joe Stein <joe.st...@stealth.ly> > wrote: > > > > > Hi, created a KIP > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > > > > > > JIRA https://issues.apache.org/jira/browse/KAFKA-1694 > > > > > > /******************************************* > > > Joe Stein > > > Founder, Principal Consultant > > > Big Data Open Source Security LLC > > > http://www.stealth.ly > > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > > > ********************************************/ > > > > > > -- -- Guozhang
