It would also be good to think through how we can use those new admin requests in the producer/consumer client as well. Currently, both the producer and the consumer use TopicMetadataRequest to obtain the metadata, which will trigger a topic creation if auto topic creation is enabled. This is a bit weird for the consumer since the reader shouldn't be creating new topics. With the new admin requests, we can potentially decouple topic creation from obtaining the metadata. The consumer can just be issuing the metadata requests without triggering the topic creation. The producer can fetch the metadata first and then issue a create topic request if the topic doesn't exist. We will have to think through how this works with the auto topic creation logic though.
Thanks, Jun On Mon, Mar 2, 2015 at 9:16 AM, Gwen Shapira <gshap...@cloudera.com> wrote: > Thanks for sending this out Joe. Looking forward to chatting with everyone > :) > > On Mon, Mar 2, 2015 at 6:46 AM, Joe Stein <joe.st...@stealth.ly> wrote: > > Hey, I just sent out a google hangout invite to all pmc, committers and > > everyone I found working on a KIP. If I missed anyone in the invite > please > > let me know and can update it, np. > > > > We should do this every Tuesday @ 2pm Eastern Time. Maybe we can get > INFRA > > help to make a google account so we can manage better? > > > > To discuss > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > in progress and related JIRA that are interdependent and common work. > > > > ~ Joe Stein > > > > On Tue, Feb 24, 2015 at 2:59 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > > > >> Let's stay on Google hangouts that will also record and make the > sessions > >> available on youtube. > >> > >> -Jay > >> > >> On Tue, Feb 24, 2015 at 11:49 AM, Jeff Holoman <jholo...@cloudera.com> > >> wrote: > >> > >> > Jay / Joe > >> > > >> > We're happy to send out a Webex for this purpose. We could record the > >> > sessions if there is interest and publish them out. > >> > > >> > Thanks > >> > > >> > Jeff > >> > > >> > On Tue, Feb 24, 2015 at 11:28 AM, Jay Kreps <jay.kr...@gmail.com> > wrote: > >> > > >> > > Let's try to get the technical hang-ups sorted out, though. I really > >> > think > >> > > there is some benefit to live discussion vs writing. I am hopeful > that > >> if > >> > > we post instructions and give ourselves a few attempts we can get it > >> > > working. > >> > > > >> > > Tuesday at that time would work for me...any objections? > >> > > > >> > > -Jay > >> > > > >> > > On Tue, Feb 24, 2015 at 8:18 AM, Joe Stein <joe.st...@stealth.ly> > >> wrote: > >> > > > >> > > > Weekly would be great maybe like every Tuesday ~ 1pm ET / 10am PT > >> ???? > >> > > > > >> > > > I don't mind google hangout but there is always some issue or > >> whatever > >> > so > >> > > > we know the apache irc channel works. We can start there and see > how > >> it > >> > > > goes? We can pull transcripts too and associate to tickets if > need be > >> > > makes > >> > > > it helpful for things. > >> > > > > >> > > > ~ Joestein > >> > > > > >> > > > On Tue, Feb 24, 2015 at 11:10 AM, Jay Kreps <jay.kr...@gmail.com> > >> > wrote: > >> > > > > >> > > > > We'd talked about doing a Google Hangout to chat about this. > What > >> > about > >> > > > > generalizing that a little further...I actually think it would > be > >> > good > >> > > > for > >> > > > > everyone spending a reasonable chunk of their week on Kafka > stuff > >> to > >> > > > maybe > >> > > > > sync up once a week. I think we could use time to talk through > >> design > >> > > > > stuff, make sure we are on top of code reviews, talk through any > >> > tricky > >> > > > > issues, etc. > >> > > > > > >> > > > > We can make it publicly available so that any one can follow > along > >> > who > >> > > > > likes. > >> > > > > > >> > > > > Any interest in doing this? If so I'll try to set it up starting > >> next > >> > > > week. > >> > > > > > >> > > > > -Jay > >> > > > > > >> > > > > On Tue, Feb 24, 2015 at 3:57 AM, Andrii Biletskyi < > >> > > > > andrii.bilets...@stealth.ly> wrote: > >> > > > > > >> > > > > > Hi all, > >> > > > > > > >> > > > > > I've updated KIP page, fixed / aligned document structure. > Also I > >> > > added > >> > > > > > some > >> > > > > > very initial proposal for AdminClient so we have something to > >> start > >> > > > from > >> > > > > > while > >> > > > > > discussing the KIP. > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > >> > > > > > > >> > > > > > Thanks, > >> > > > > > Andrii Biletskyi > >> > > > > > > >> > > > > > On Wed, Feb 18, 2015 at 9:01 PM, Andrii Biletskyi < > >> > > > > > andrii.bilets...@stealth.ly> wrote: > >> > > > > > > >> > > > > > > Jay, > >> > > > > > > > >> > > > > > > Re error messages: you are right, in most cases client will > >> have > >> > > > enough > >> > > > > > > context to show descriptive error message. My concern is > that > >> we > >> > > will > >> > > > > > have > >> > > > > > > to > >> > > > > > > add lots of new error codes for each possible error. Of > course, > >> > we > >> > > > > could > >> > > > > > > reuse > >> > > > > > > some of existing like UknownTopicOrPartitionCode, but we > will > >> > also > >> > > > need > >> > > > > > to > >> > > > > > > add smth like: TopicAlreadyExistsCode, TopicConfigInvalid > (both > >> > for > >> > > > > topic > >> > > > > > > name and config, and probably user would like to know what > >> > exactly > >> > > > > > > is wrong in his config), InvalidReplicaAssignment, > >> InternalError > >> > > > (e.g. > >> > > > > > > zookeeper failure) etc. > >> > > > > > > And this is only for TopicCommand, we will also need to add > >> > similar > >> > > > > stuff > >> > > > > > > for > >> > > > > > > ReassignPartitions, PreferredReplica. So we'll end up with a > >> > large > >> > > > list > >> > > > > > of > >> > > > > > > error codes, used only in Admin protocol. > >> > > > > > > Having said that, I agree my proposal is not consistent with > >> > other > >> > > > > cases. > >> > > > > > > Maybe we can find better solution or something in-between. > >> > > > > > > > >> > > > > > > Re Hangout chat: I think it is a great idea. This way we can > >> move > >> > > on > >> > > > > > > faster. > >> > > > > > > Let's agree somehow on date/time so people can join. Will > work > >> > for > >> > > me > >> > > > > > this > >> > > > > > > and > >> > > > > > > next week almost anytime if agreed in advance. > >> > > > > > > > >> > > > > > > Thanks, > >> > > > > > > Andrii > >> > > > > > > > >> > > > > > > On Wed, Feb 18, 2015 at 7:09 PM, Jay Kreps < > >> jay.kr...@gmail.com> > >> > > > > wrote: > >> > > > > > > > >> > > > > > >> Hey Andrii, > >> > > > > > >> > >> > > > > > >> Generally we can do good error handling without needing > custom > >> > > > > > server-side > >> > > > > > >> messages. I.e. generally the client has the context to know > >> that > >> > > if > >> > > > it > >> > > > > > got > >> > > > > > >> an error that the topic doesn't exist to say "Topic X > doesn't > >> > > exist" > >> > > > > > >> rather > >> > > > > > >> than "error code 14" (or whatever). Maybe there are > specific > >> > cases > >> > > > > where > >> > > > > > >> this is hard? If we want to add server-side error messages > we > >> > > really > >> > > > > do > >> > > > > > >> need to do this in a consistent way across the protocol. > >> > > > > > >> > >> > > > > > >> I still have a bunch of open questions here from my > previous > >> > > list. I > >> > > > > > will > >> > > > > > >> be out for the next few days for Strata though. Maybe we > could > >> > do > >> > > a > >> > > > > > Google > >> > > > > > >> Hangout chat on any open issues some time towards the end > of > >> > next > >> > > > week > >> > > > > > for > >> > > > > > >> anyone interested in this ticket? I have a feeling that > might > >> > > > progress > >> > > > > > >> things a little faster than email--I think we could talk > >> through > >> > > > those > >> > > > > > >> issues I brought up fairly quickly... > >> > > > > > >> > >> > > > > > >> -Jay > >> > > > > > >> > >> > > > > > >> On Wed, Feb 18, 2015 at 7:27 AM, Andrii Biletskyi < > >> > > > > > >> andrii.bilets...@stealth.ly> wrote: > >> > > > > > >> > >> > > > > > >> > Hi all, > >> > > > > > >> > > >> > > > > > >> > I'm trying to address some of the issues which were > >> mentioned > >> > > > > earlier > >> > > > > > >> about > >> > > > > > >> > Admin RQ/RP format. One of those was about batching > >> > operations. > >> > > > What > >> > > > > > if > >> > > > > > >> we > >> > > > > > >> > follow TopicCommand approach and let people specify > >> topic-name > >> > > by > >> > > > > > >> regexp - > >> > > > > > >> > would that cover most of the use cases? > >> > > > > > >> > > >> > > > > > >> > Secondly, is what information should we generally > provide in > >> > > Admin > >> > > > > > >> > responses. > >> > > > > > >> > I realize that Admin commands don't imply they will be > used > >> > only > >> > > > in > >> > > > > > CLI > >> > > > > > >> > but, > >> > > > > > >> > it seems to me, CLI is a very important client of this > >> > feature. > >> > > In > >> > > > > > this > >> > > > > > >> > case, > >> > > > > > >> > seems logical, we would like to provide users with rich > >> > > experience > >> > > > > in > >> > > > > > >> terms > >> > > > > > >> > of > >> > > > > > >> > getting results / errors of the executed commands. > Usually > >> we > >> > > > supply > >> > > > > > >> with > >> > > > > > >> > responses only errorCode, which looks very limiting, in > case > >> > of > >> > > > CLI > >> > > > > we > >> > > > > > >> may > >> > > > > > >> > want to print human readable error description. > >> > > > > > >> > > >> > > > > > >> > So, taking into account previous item about batching, > what > >> do > >> > > you > >> > > > > > think > >> > > > > > >> > about > >> > > > > > >> > having smth like: > >> > > > > > >> > > >> > > > > > >> > ('create' doesn't support regexp) > >> > > > > > >> > CreateTopicRequest => TopicName Partitions Replicas > >> > > > > ReplicaAssignment > >> > > > > > >> > [Config] > >> > > > > > >> > CreateTopicResponse => ErrorCode ErrorDescription > >> > > > > > >> > ErrorCode => int16 > >> > > > > > >> > ErrorDescription => string (empty if successful) > >> > > > > > >> > > >> > > > > > >> > AlterTopicRequest -> TopicNameRegexp Partitions > >> > > ReplicaAssignment > >> > > > > > >> > [AddedConfig] [DeletedConfig] > >> > > > > > >> > AlterTopicResponse -> [TopicName ErrorCode > ErrorDescription] > >> > > > > > >> > CommandErrorCode CommandErrorDescription > >> > > > > > >> > CommandErrorCode => int16 > >> > > > > > >> > CommandErrorDescription => string (nonempty in case of > >> fatal > >> > > > > error, > >> > > > > > >> e.g. > >> > > > > > >> > we couldn't get topics by regexp) > >> > > > > > >> > > >> > > > > > >> > DescribeTopicRequest -> TopicNameRegexp > >> > > > > > >> > DescribeTopicResponse -> [TopicName TopicDescription > >> ErrorCode > >> > > > > > >> > ErrorDescription] CommandErrorCode > CommandErrorDescription > >> > > > > > >> > > >> > > > > > >> > Also, any thoughts about our discussion regarding > re-routing > >> > > > > facility? > >> > > > > > >> In > >> > > > > > >> > my > >> > > > > > >> > understanding, it is like between augmenting > >> > > TopicMetadataRequest > >> > > > > > >> > (to include at least controllerId) and implementing new > >> > generic > >> > > > > > >> re-routing > >> > > > > > >> > facility so sending messages to controller will be > handled > >> by > >> > > it. > >> > > > > > >> > > >> > > > > > >> > Thanks, > >> > > > > > >> > Andrii Biletskyi > >> > > > > > >> > > >> > > > > > >> > On Mon, Feb 16, 2015 at 5:26 PM, Andrii Biletskyi < > >> > > > > > >> > andrii.bilets...@stealth.ly> wrote: > >> > > > > > >> > > >> > > > > > >> > > @Guozhang: > >> > > > > > >> > > Thanks for your comments, I've answered some of those. > The > >> > > main > >> > > > > > thing > >> > > > > > >> is > >> > > > > > >> > > having merged request for create-alter-delete-describe > - I > >> > > have > >> > > > > some > >> > > > > > >> > > concerns about this approach. > >> > > > > > >> > > > >> > > > > > >> > > @*Jay*: > >> > > > > > >> > > I see that introduced ClusterMetadaRequest is also one > of > >> > the > >> > > > > > >> concerns. > >> > > > > > >> > We > >> > > > > > >> > > can solve it if we implement re-routing facility. But I > >> > agree > >> > > > with > >> > > > > > >> > > Guozhang - it will make clients' internals a little bit > >> > easier > >> > > > but > >> > > > > > >> this > >> > > > > > >> > > seems to be a complex logic to implement and support > then. > >> > > > > > Especially > >> > > > > > >> for > >> > > > > > >> > > Fetch and Produce (even if we add re-routing later for > >> these > >> > > > > > >> requests). > >> > > > > > >> > > Also people will tend to avoid this re-routing facility > >> and > >> > > hold > >> > > > > > local > >> > > > > > >> > > cluster cache to ensure their high-priority requests > >> (which > >> > > some > >> > > > > of > >> > > > > > >> the > >> > > > > > >> > > admin requests are) not sent to some busy broker where > >> they > >> > > wait > >> > > > > to > >> > > > > > be > >> > > > > > >> > > routed to the correct one. > >> > > > > > >> > > As pointed out by Jun here ( > >> > > > > > >> > > > >> > > > > > >> > > >> > > > > > >> > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://issues.apache.org/jira/browse/KAFKA-1772?focusedCommentId=14234530&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14234530 > >> > > > > > >> > ) > >> > > > > > >> > > to solve the issue we might introduce a message type to > >> get > >> > > > > cluster > >> > > > > > >> > state. > >> > > > > > >> > > But I agree we can just update TopicMetadataResponse to > >> > > include > >> > > > > > >> > > controllerId (and probably smth else). > >> > > > > > >> > > What are you thougths? > >> > > > > > >> > > > >> > > > > > >> > > Thanks, > >> > > > > > >> > > Andrii > >> > > > > > >> > > > >> > > > > > >> > > On Thu, Feb 12, 2015 at 8:31 AM, Guozhang Wang < > >> > > > > wangg...@gmail.com> > >> > > > > > >> > wrote: > >> > > > > > >> > > > >> > > > > > >> > >> I think for the topics commands we can actually merge > >> > > > > > >> > >> create/alter/delete/describe as one request type since > >> > their > >> > > > > > formats > >> > > > > > >> are > >> > > > > > >> > >> very much similar, and keep list-topics and others > like > >> > > > > > >> > >> partition-reassignment / preferred-leader-election as > >> > > separate > >> > > > > > >> request > >> > > > > > >> > >> types, I also left some other comments on the RB ( > >> > > > > > >> > >> https://reviews.apache.org/r/29301/). > >> > > > > > >> > >> > >> > > > > > >> > >> On Wed, Feb 11, 2015 at 2:04 PM, Jay Kreps < > >> > > > jay.kr...@gmail.com> > >> > > > > > >> wrote: > >> > > > > > >> > >> > >> > > > > > >> > >> > Yeah I totally agree that we don't want to just have > >> one > >> > > "do > >> > > > > > admin > >> > > > > > >> > >> stuff" > >> > > > > > >> > >> > command that has the union of all parameters. > >> > > > > > >> > >> > > >> > > > > > >> > >> > What I am saying is that command line tools are one > >> > client > >> > > of > >> > > > > the > >> > > > > > >> > >> > administrative apis, but these will be used in a > number > >> > of > >> > > > > > >> scenarios > >> > > > > > >> > so > >> > > > > > >> > >> > they should make logical sense even in the absence > of > >> the > >> > > > > command > >> > > > > > >> line > >> > > > > > >> > >> > tool. Hence comments like trying to clarify the > >> > > relationship > >> > > > > > >> between > >> > > > > > >> > >> > ClusterMetadata and TopicMetadata...these kinds of > >> things > >> > > > > really > >> > > > > > >> need > >> > > > > > >> > >> to be > >> > > > > > >> > >> > thought through. > >> > > > > > >> > >> > > >> > > > > > >> > >> > Hope that makes sense. > >> > > > > > >> > >> > > >> > > > > > >> > >> > -Jay > >> > > > > > >> > >> > > >> > > > > > >> > >> > On Wed, Feb 11, 2015 at 1:41 PM, Andrii Biletskyi < > >> > > > > > >> > >> > andrii.bilets...@stealth.ly> wrote: > >> > > > > > >> > >> > > >> > > > > > >> > >> > > Jay, > >> > > > > > >> > >> > > > >> > > > > > >> > >> > > Thanks for answering. You understood correctly, > most > >> of > >> > > my > >> > > > > > >> comments > >> > > > > > >> > >> were > >> > > > > > >> > >> > > related to your point 1) - about "well > thought-out" > >> > apis. > >> > > > > Also, > >> > > > > > >> yes, > >> > > > > > >> > >> as I > >> > > > > > >> > >> > > understood we would like to introduce a single > >> unified > >> > > CLI > >> > > > > tool > >> > > > > > >> with > >> > > > > > >> > >> > > centralized server-side request handling for lots > of > >> > > > existing > >> > > > > > >> ones > >> > > > > > >> > >> (incl. > >> > > > > > >> > >> > > TopicCommand, CommitOffsetChecker, > >> ReassignPartitions, > >> > > smth > >> > > > > > else > >> > > > > > >> if > >> > > > > > >> > >> added > >> > > > > > >> > >> > > in future). In our previous discussion ( > >> > > > > > >> > >> > > https://issues.apache.org/jira/browse/KAFKA-1694) > >> > people > >> > > > > said > >> > > > > > >> > they'd > >> > > > > > >> > >> > > rather > >> > > > > > >> > >> > > have a separate message for each command, so, yes, > >> this > >> > > > way I > >> > > > > > >> came > >> > > > > > >> > to > >> > > > > > >> > >> 1-1 > >> > > > > > >> > >> > > mapping between commands in the tool and protocol > >> > > > additions. > >> > > > > > But > >> > > > > > >> I > >> > > > > > >> > >> might > >> > > > > > >> > >> > be > >> > > > > > >> > >> > > wrong. > >> > > > > > >> > >> > > At the end I just try to start discussion how at > >> least > >> > > > > > generally > >> > > > > > >> > this > >> > > > > > >> > >> > > protocol should look like. > >> > > > > > >> > >> > > > >> > > > > > >> > >> > > Thanks, > >> > > > > > >> > >> > > Andrii > >> > > > > > >> > >> > > > >> > > > > > >> > >> > > On Wed, Feb 11, 2015 at 11:10 PM, Jay Kreps < > >> > > > > > jay.kr...@gmail.com > >> > > > > > >> > > >> > > > > > >> > >> wrote: > >> > > > > > >> > >> > > > >> > > > > > >> > >> > > > Hey Andrii, > >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > To answer your earlier question we just really > >> can't > >> > be > >> > > > > > adding > >> > > > > > >> any > >> > > > > > >> > >> more > >> > > > > > >> > >> > > > scala protocol objects. These things are super > hard > >> > to > >> > > > > > maintain > >> > > > > > >> > >> because > >> > > > > > >> > >> > > > they hand code the byte parsing and don't have > good > >> > > > > > versioning > >> > > > > > >> > >> support. > >> > > > > > >> > >> > > > Since we are already planning on converting we > >> > > definitely > >> > > > > > don't > >> > > > > > >> > >> want to > >> > > > > > >> > >> > > add > >> > > > > > >> > >> > > > a ton more of these--they are total tech debt. > >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > What does it mean that the changes are isolated > >> from > >> > > the > >> > > > > > >> current > >> > > > > > >> > >> code > >> > > > > > >> > >> > > base? > >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > I actually didn't understand the remaining > >> comments, > >> > > > which > >> > > > > of > >> > > > > > >> the > >> > > > > > >> > >> > points > >> > > > > > >> > >> > > > are you responding to? > >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > Maybe one sticking point here is that it seems > like > >> > you > >> > > > > want > >> > > > > > to > >> > > > > > >> > make > >> > > > > > >> > >> > some > >> > > > > > >> > >> > > > kind of tool, and you have made a 1-1 mapping > >> between > >> > > > > > commands > >> > > > > > >> you > >> > > > > > >> > >> > > imagine > >> > > > > > >> > >> > > > in the tool and protocol additions. I want to > make > >> > sure > >> > > > we > >> > > > > > >> don't > >> > > > > > >> > do > >> > > > > > >> > >> > that. > >> > > > > > >> > >> > > > The protocol needs to be really really well > thought > >> > out > >> > > > > > against > >> > > > > > >> > many > >> > > > > > >> > >> > use > >> > > > > > >> > >> > > > cases so it should make perfect logical sense in > >> the > >> > > > > absence > >> > > > > > of > >> > > > > > >> > >> knowing > >> > > > > > >> > >> > > the > >> > > > > > >> > >> > > > command line tool, right? > >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > -Jay > >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > On Wed, Feb 11, 2015 at 11:57 AM, Andrii > Biletskyi > >> < > >> > > > > > >> > >> > > > andrii.bilets...@stealth.ly> wrote: > >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > > Hey Jay, > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > I would like to continue this discussion as it > >> seem > >> > > > there > >> > > > > > is > >> > > > > > >> no > >> > > > > > >> > >> > > progress > >> > > > > > >> > >> > > > > here. > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > First of all, could you please explain what > did > >> you > >> > > > mean > >> > > > > in > >> > > > > > >> 2? > >> > > > > > >> > How > >> > > > > > >> > >> > > > exactly > >> > > > > > >> > >> > > > > are we going to migrate to the new java > protocol > >> > > > > > definitions. > >> > > > > > >> > And > >> > > > > > >> > >> why > >> > > > > > >> > >> > > > it's > >> > > > > > >> > >> > > > > a blocker for centralized CLI? > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > I agree with you, this feature includes lots > of > >> > > stuff, > >> > > > > but > >> > > > > > >> > >> thankfully > >> > > > > > >> > >> > > > > almost all changes are isolated from the > current > >> > code > >> > > > > base, > >> > > > > > >> > >> > > > > so the main thing, I think, we need to agree > is > >> > RQ/RP > >> > > > > > format. > >> > > > > > >> > >> > > > > So how can we start discussion about the > concrete > >> > > > > messages > >> > > > > > >> > format? > >> > > > > > >> > >> > > > > Can we take ( > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > >> > > > > > >> > >> > > >> > > > > > >> > >> > >> > > > > > >> > > >> > > > > > >> > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-ProposedRQ/RPFormat > >> > > > > > >> > >> > > > > ) > >> > > > > > >> > >> > > > > as starting point? > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > We had some doubts earlier whether it worth > >> > > introducing > >> > > > > one > >> > > > > > >> > >> generic > >> > > > > > >> > >> > > Admin > >> > > > > > >> > >> > > > > Request for all commands ( > >> > > > > > >> > >> > > > > https://issues.apache.org/jira/browse/KAFKA-1694 > >> > > > > > >> > >> > > > > ) > >> > > > > > >> > >> > > > > but then everybody agreed it would be better > to > >> > have > >> > > > > > separate > >> > > > > > >> > >> message > >> > > > > > >> > >> > > for > >> > > > > > >> > >> > > > > each admin command. The Request part is really > >> > > dictated > >> > > > > > from > >> > > > > > >> the > >> > > > > > >> > >> > > command > >> > > > > > >> > >> > > > > (e.g. TopicCommand) arguments itself, so the > >> > proposed > >> > > > > > version > >> > > > > > >> > >> should > >> > > > > > >> > >> > be > >> > > > > > >> > >> > > > > fine (let's put aside for now remarks about > >> > Optional > >> > > > > type, > >> > > > > > >> > >> batching, > >> > > > > > >> > >> > > > > configs normalization - I agree with all of > >> them). > >> > > > > > >> > >> > > > > So the second part is Response. I see there > are > >> two > >> > > > cases > >> > > > > > >> here. > >> > > > > > >> > >> > > > > a) "Mutate" requests - Create/Alter/... ; b) > >> "Get" > >> > > > > > requests - > >> > > > > > >> > >> > > > > List/Describe... > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > a) should only hold request result (regardless > >> what > >> > > we > >> > > > > > decide > >> > > > > > >> > >> about > >> > > > > > >> > >> > > > > blocking/non-blocking commands execution). > >> > > > > > >> > >> > > > > Usually we provide error code in response but > >> since > >> > > we > >> > > > > will > >> > > > > > >> use > >> > > > > > >> > >> this > >> > > > > > >> > >> > in > >> > > > > > >> > >> > > > > interactive shell we need some human readable > >> error > >> > > > > > >> description > >> > > > > > >> > - > >> > > > > > >> > >> so > >> > > > > > >> > >> > I > >> > > > > > >> > >> > > > > added errorDesription field where you can at > >> least > >> > > > leave > >> > > > > > >> > >> > > > > exception.getMessage. > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > b) in addition to previous item message should > >> hold > >> > > > > command > >> > > > > > >> > >> specific > >> > > > > > >> > >> > > > > response data. We can discuss in detail each > of > >> > them > >> > > > but > >> > > > > > >> let's > >> > > > > > >> > for > >> > > > > > >> > >> > now > >> > > > > > >> > >> > > > > agree about the overall pattern. > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > Thanks, > >> > > > > > >> > >> > > > > Andrii Biletskyi > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > On Fri, Jan 23, 2015 at 6:59 AM, Jay Kreps < > >> > > > > > >> jay.kr...@gmail.com > >> > > > > > >> > > > >> > > > > > >> > >> > > wrote: > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > > Hey Joe, > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > This is great. A few comments on KIP-4 > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > 1. This is much needed functionality, but > there > >> > > are a > >> > > > > lot > >> > > > > > >> of > >> > > > > > >> > >> the so > >> > > > > > >> > >> > > > let's > >> > > > > > >> > >> > > > > > really think these protocols through. We > really > >> > > want > >> > > > to > >> > > > > > >> end up > >> > > > > > >> > >> > with a > >> > > > > > >> > >> > > > set > >> > > > > > >> > >> > > > > > of well thought-out, orthoganol apis. For > this > >> > > > reason I > >> > > > > > >> think > >> > > > > > >> > >> it is > >> > > > > > >> > >> > > > > really > >> > > > > > >> > >> > > > > > important to think through the end state > even > >> if > >> > > that > >> > > > > > >> includes > >> > > > > > >> > >> APIs > >> > > > > > >> > >> > > we > >> > > > > > >> > >> > > > > > won't implement in the first phase. > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > 2. Let's please please please wait until we > >> have > >> > > > > switched > >> > > > > > >> the > >> > > > > > >> > >> > server > >> > > > > > >> > >> > > > over > >> > > > > > >> > >> > > > > > to the new java protocol definitions. If we > add > >> > > > upteen > >> > > > > > >> more ad > >> > > > > > >> > >> hoc > >> > > > > > >> > >> > > > scala > >> > > > > > >> > >> > > > > > objects that is just generating more work > for > >> the > >> > > > > > >> conversion > >> > > > > > >> > we > >> > > > > > >> > >> > know > >> > > > > > >> > >> > > we > >> > > > > > >> > >> > > > > > have to do. > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > 3. This proposal introduces a new type of > >> > optional > >> > > > > > >> parameter. > >> > > > > > >> > >> This > >> > > > > > >> > >> > is > >> > > > > > >> > >> > > > > > inconsistent with everything else in the > >> protocol > >> > > > where > >> > > > > > we > >> > > > > > >> use > >> > > > > > >> > >> -1 > >> > > > > > >> > >> > or > >> > > > > > >> > >> > > > some > >> > > > > > >> > >> > > > > > other marker value. You could argue either > way > >> > but > >> > > > > let's > >> > > > > > >> stick > >> > > > > > >> > >> with > >> > > > > > >> > >> > > > that > >> > > > > > >> > >> > > > > > for consistency. For clients that > implemented > >> the > >> > > > > > protocol > >> > > > > > >> in > >> > > > > > >> > a > >> > > > > > >> > >> > > better > >> > > > > > >> > >> > > > > way > >> > > > > > >> > >> > > > > > than our scala code these basic primitives > are > >> > hard > >> > > > to > >> > > > > > >> change. > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > 4. ClusterMetadata: This seems to duplicate > >> > > > > > >> > TopicMetadataRequest > >> > > > > > >> > >> > > which > >> > > > > > >> > >> > > > > has > >> > > > > > >> > >> > > > > > brokers, topics, and partitions. I think we > >> > should > >> > > > > rename > >> > > > > > >> that > >> > > > > > >> > >> > > request > >> > > > > > >> > >> > > > > > ClusterMetadataRequest (or just > >> MetadataRequest) > >> > > and > >> > > > > > >> include > >> > > > > > >> > >> the id > >> > > > > > >> > >> > > of > >> > > > > > >> > >> > > > > the > >> > > > > > >> > >> > > > > > controller. Or are there other things we > could > >> > add > >> > > > > here? > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > 5. We have a tendency to try to make a lot > of > >> > > > requests > >> > > > > > that > >> > > > > > >> > can > >> > > > > > >> > >> > only > >> > > > > > >> > >> > > go > >> > > > > > >> > >> > > > > to > >> > > > > > >> > >> > > > > > particular nodes. This adds a lot of burden > for > >> > > > client > >> > > > > > >> > >> > > implementations > >> > > > > > >> > >> > > > > (it > >> > > > > > >> > >> > > > > > sounds easy but each discovery can fail in > many > >> > > parts > >> > > > > so > >> > > > > > it > >> > > > > > >> > >> ends up > >> > > > > > >> > >> > > > > being a > >> > > > > > >> > >> > > > > > full state machine to do right). I think we > >> > should > >> > > > > > consider > >> > > > > > >> > >> making > >> > > > > > >> > >> > > > admin > >> > > > > > >> > >> > > > > > commands and ideally as many of the other > apis > >> as > >> > > > > > possible > >> > > > > > >> > >> > available > >> > > > > > >> > >> > > on > >> > > > > > >> > >> > > > > all > >> > > > > > >> > >> > > > > > brokers and just redirect to the controller > on > >> > the > >> > > > > broker > >> > > > > > >> > side. > >> > > > > > >> > >> > > Perhaps > >> > > > > > >> > >> > > > > > there would be a general way to encapsulate > >> this > >> > > > > > re-routing > >> > > > > > >> > >> > behavior. > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > 6. We should probably normalize the key > value > >> > pairs > >> > > > > used > >> > > > > > >> for > >> > > > > > >> > >> > configs > >> > > > > > >> > >> > > > > rather > >> > > > > > >> > >> > > > > > than embedding a new formatting. So two > strings > >> > > > rather > >> > > > > > than > >> > > > > > >> > one > >> > > > > > >> > >> > with > >> > > > > > >> > >> > > an > >> > > > > > >> > >> > > > > > internal equals sign. > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > 7. Is the postcondition of these APIs that > the > >> > > > command > >> > > > > > has > >> > > > > > >> > >> begun or > >> > > > > > >> > >> > > > that > >> > > > > > >> > >> > > > > > the command has been completed? It is a lot > >> more > >> > > > usable > >> > > > > > if > >> > > > > > >> the > >> > > > > > >> > >> > > command > >> > > > > > >> > >> > > > > has > >> > > > > > >> > >> > > > > > been completed so you know that if you > create a > >> > > topic > >> > > > > and > >> > > > > > >> then > >> > > > > > >> > >> > > publish > >> > > > > > >> > >> > > > to > >> > > > > > >> > >> > > > > > it you won't get an exception about there > being > >> > no > >> > > > such > >> > > > > > >> topic. > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > 8. Describe topic and list topics duplicate > a > >> lot > >> > > of > >> > > > > > stuff > >> > > > > > >> in > >> > > > > > >> > >> the > >> > > > > > >> > >> > > > > metadata > >> > > > > > >> > >> > > > > > request. Is there a reason to give back > topics > >> > > marked > >> > > > > for > >> > > > > > >> > >> > deletion? I > >> > > > > > >> > >> > > > > feel > >> > > > > > >> > >> > > > > > like if we just make the post-condition of > the > >> > > delete > >> > > > > > >> command > >> > > > > > >> > be > >> > > > > > >> > >> > that > >> > > > > > >> > >> > > > the > >> > > > > > >> > >> > > > > > topic is deleted that will get rid of the > need > >> > for > >> > > > this > >> > > > > > >> right? > >> > > > > > >> > >> And > >> > > > > > >> > >> > it > >> > > > > > >> > >> > > > > will > >> > > > > > >> > >> > > > > > be much more intuitive. > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > 9. Should we consider batching these > requests? > >> We > >> > > > have > >> > > > > > >> > generally > >> > > > > > >> > >> > > tried > >> > > > > > >> > >> > > > to > >> > > > > > >> > >> > > > > > allow multiple operations to be batched. My > >> > > suspicion > >> > > > > is > >> > > > > > >> that > >> > > > > > >> > >> > without > >> > > > > > >> > >> > > > > this > >> > > > > > >> > >> > > > > > we will get a lot of code that does > something > >> > like > >> > > > > > >> > >> > > > > > for(topic: adminClient.listTopics()) > >> > > > > > >> > >> > > > > > adminClient.describeTopic(topic) > >> > > > > > >> > >> > > > > > this code will work great when you test on 5 > >> > topics > >> > > > but > >> > > > > > >> not do > >> > > > > > >> > >> as > >> > > > > > >> > >> > > well > >> > > > > > >> > >> > > > if > >> > > > > > >> > >> > > > > > you have 50k. > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > 10. I think we should also discuss how we > want > >> to > >> > > > > expose > >> > > > > > a > >> > > > > > >> > >> > > programmatic > >> > > > > > >> > >> > > > > JVM > >> > > > > > >> > >> > > > > > client api for these operations. Currently > >> people > >> > > > rely > >> > > > > on > >> > > > > > >> > >> > AdminUtils > >> > > > > > >> > >> > > > > which > >> > > > > > >> > >> > > > > > is totally sketchy. I think we probably need > >> > > another > >> > > > > > client > >> > > > > > >> > >> under > >> > > > > > >> > >> > > > > clients/ > >> > > > > > >> > >> > > > > > that exposes administrative functionality. > We > >> > will > >> > > > need > >> > > > > > >> this > >> > > > > > >> > >> just > >> > > > > > >> > >> > to > >> > > > > > >> > >> > > > > > properly test the new apis, I suspect. We > >> should > >> > > > figure > >> > > > > > out > >> > > > > > >> > that > >> > > > > > >> > >> > API. > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > 11. The other information that would be > really > >> > > useful > >> > > > > to > >> > > > > > >> get > >> > > > > > >> > >> would > >> > > > > > >> > >> > be > >> > > > > > >> > >> > > > > > information about partitions--how much data > is > >> in > >> > > the > >> > > > > > >> > partition, > >> > > > > > >> > >> > what > >> > > > > > >> > >> > > > are > >> > > > > > >> > >> > > > > > the segment offsets, what is the log-end > offset > >> > > (i.e. > >> > > > > > last > >> > > > > > >> > >> offset), > >> > > > > > >> > >> > > > what > >> > > > > > >> > >> > > > > is > >> > > > > > >> > >> > > > > > the compaction point, etc. I think that done > >> > right > >> > > > this > >> > > > > > >> would > >> > > > > > >> > be > >> > > > > > >> > >> > the > >> > > > > > >> > >> > > > > > successor to the very awkward OffsetRequest > we > >> > have > >> > > > > > today. > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > -Jay > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > On Wed, Jan 21, 2015 at 10:27 PM, Joe Stein > < > >> > > > > > >> > >> joe.st...@stealth.ly> > >> > > > > > >> > >> > > > > wrote: > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > > Hi, created a KIP > >> > > > > > >> > >> > > > > > > > >> > > > > > >> > >> > > > > > > > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > >> > > > > > >> > >> > > >> > > > > > >> > >> > >> > > > > > >> > > >> > > > > > >> > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > >> > > > > > >> > >> > > > > > > > >> > > > > > >> > >> > > > > > > JIRA > >> > > > > https://issues.apache.org/jira/browse/KAFKA-1694 > >> > > > > > >> > >> > > > > > > > >> > > > > > >> > >> > > > > > > > /******************************************* > >> > > > > > >> > >> > > > > > > Joe Stein > >> > > > > > >> > >> > > > > > > Founder, Principal Consultant > >> > > > > > >> > >> > > > > > > Big Data Open Source Security LLC > >> > > > > > >> > >> > > > > > > http://www.stealth.ly > >> > > > > > >> > >> > > > > > > Twitter: @allthingshadoop < > >> > > > > > >> > >> > http://www.twitter.com/allthingshadoop > >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > > > > > ********************************************/ > >> > > > > > >> > >> > > > > > > > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > >> > > > > > >> > >> > > >> > > > > > >> > >> > >> > > > > > >> > >> > >> > > > > > >> > >> > >> > > > > > >> > >> -- > >> > > > > > >> > >> -- Guozhang > >> > > > > > >> > >> > >> > > > > > >> > > > >> > > > > > >> > > > >> > > > > > >> > > >> > > > > > >> > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > >> > > >> > -- > >> > Jeff Holoman > >> > Systems Engineer > >> > > >> >
