Hi all, Today I uploaded the patch that covers some of the discussed and agreed items: - removed MaybeOf optional type - switched to java protocol definitions - simplified messages (normalized configs, removed topic marked for deletion)
I also updated the KIP-4 with respective changes and wrote down my proposal for pending items: - Batch Admin Operations -> updated Wire Protocol schema proposal - Remove ClusterMetadata -> changed to extend TopicMetadataRequest - Admin Client -> updated my initial proposal to reflect batching - Error codes -> proposed fine-grained error code instead of AdminRequestFailed I will also send a separate email to cover all comments from this thread. Thanks, Andrii Biletskyi On Thu, Mar 12, 2015 at 9:26 PM, Gwen Shapira <gshap...@cloudera.com> wrote: > Found KIP-11 ( > https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface > ) > It actually specifies changes to the Metadata protocol, so making sure > both KIPs are consistent in this regard will be good. > > On Thu, Mar 12, 2015 at 12:21 PM, Gwen Shapira <gshap...@cloudera.com> > wrote: > > Specifically for ownership, I think the plan is to add ACL (it sounds > > like you are describing ACL) via an external system (Argus, Sentry). > > I remember KIP-11 described this, but I can't find the KIP any longer. > > > > Regardless, I think KIP-4 focuses on getting information that already > > exists from Kafka brokers, not on adding information that perhaps > > should exist but doesn't yet? > > > > Gwen > > > > > > > > > > > > On Thu, Mar 12, 2015 at 6:37 AM, Guozhang Wang <wangg...@gmail.com> > wrote: > >> Folks, > >> > >> Just want to elaborate a bit more on the create-topic metadata and > batching > >> describe-topic based on config / metadata in my previous email as we > work > >> on KAFKA-1694. The main motivation is to have some sort of topic > management > >> mechanisms, which I think is quite important in a multi-tenant / cloud > >> architecture: today anyone can create topics in a shared Kafka cluster, > but > >> there is no concept or "ownership" of topics that are created by > different > >> users. For example, at LinkedIn we basically distinguish topic owners > via > >> some casual topic name prefix, which is a bit awkward and does not fly > as > >> we scale our customers. It would be great to use describe-topics such > as: > >> > >> Describe all topics that is created by me. > >> > >> Describe all topics whose retention time is overriden to X. > >> > >> Describe all topics whose writable group include user Y (this is > related to > >> authorization), etc.. > >> > >> One possible way to achieve this is to add a metadata file in the > >> create-topic request, whose value will also be written ZK as we create > the > >> topic; then describe-topics can choose to batch topics based on 1) name > >> regex, 2) config K-V matching, 3) metadata regex, etc. > >> > >> Thoughts? > >> > >> Guozhang > >> > >> On Thu, Mar 5, 2015 at 4:37 PM, Guozhang Wang <wangg...@gmail.com> > wrote: > >> > >>> Thanks for the updated wiki. A few comments below: > >>> > >>> 1. Error description in response: I think if some errorCode could > indicate > >>> several different error cases then we should really change it to > multiple > >>> codes. In general the errorCode itself would be precise and sufficient > for > >>> describing the server side errors. > >>> > >>> 2. Describe topic request: it would be great to go beyond just > batching on > >>> topic name regex for this request. For example, a very common use case > of > >>> the topic command is to list all topics whose config A's value is B. > With > >>> topic name regex then we have to first retrieve __all__ topics's > >>> description info and then filter at the client end, which will be a > huge > >>> burden on ZK. > >>> > >>> 3. Config K-Vs in create topic: this is related to the previous point; > >>> maybe we can add another metadata K-V or just a metadata string along > side > >>> with config K-V in create topic like we did for offset commit request. > This > >>> field can be quite useful in storing information like "owner" of the > topic > >>> who issue the create command, etc, which is quite important for a > >>> multi-tenant setting. Then in the describe topic request we can also > batch > >>> on regex of the metadata field. > >>> > >>> 4. Today all the admin operations are async in the sense that command > will > >>> return once it is written in ZK, and that is why we need extra > verification > >>> like testUtil.waitForTopicCreated() / verify partition reassignment > >>> request, etc. With admin requests we could add a flag to enable / > disable > >>> synchronous requests; when it is turned on, the response will not > return > >>> until the request has been completed. And for async requests we can > add a > >>> "token" field in the response, and then only need a general "admin > >>> verification request" with the given token to check if the async > request > >>> has been completed. > >>> > >>> 5. +1 for extending Metadata request to include controller / > coordinator > >>> information, and then we can remove the ConsumerMetadata / > ClusterMetadata > >>> requests. > >>> > >>> Guozhang > >>> > >>> On Tue, Mar 3, 2015 at 10:23 AM, Joel Koshy <jjkosh...@gmail.com> > wrote: > >>> > >>>> Thanks for sending that out Joe - I don't think I will be able to make > >>>> it today, so if notes can be sent out afterward that would be great. > >>>> > >>>> On Mon, Mar 02, 2015 at 09:16:13AM -0800, Gwen Shapira wrote: > >>>> > Thanks for sending this out Joe. Looking forward to chatting with > >>>> everyone :) > >>>> > > >>>> > On Mon, Mar 2, 2015 at 6:46 AM, Joe Stein <joe.st...@stealth.ly> > wrote: > >>>> > > Hey, I just sent out a google hangout invite to all pmc, > committers > >>>> and > >>>> > > everyone I found working on a KIP. If I missed anyone in the > invite > >>>> please > >>>> > > let me know and can update it, np. > >>>> > > > >>>> > > We should do this every Tuesday @ 2pm Eastern Time. Maybe we can > get > >>>> INFRA > >>>> > > help to make a google account so we can manage better? > >>>> > > > >>>> > > To discuss > >>>> > > > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > >>>> > > in progress and related JIRA that are interdependent and common > work. > >>>> > > > >>>> > > ~ Joe Stein > >>>> > > > >>>> > > On Tue, Feb 24, 2015 at 2:59 PM, Jay Kreps <jay.kr...@gmail.com> > >>>> wrote: > >>>> > > > >>>> > >> Let's stay on Google hangouts that will also record and make the > >>>> sessions > >>>> > >> available on youtube. > >>>> > >> > >>>> > >> -Jay > >>>> > >> > >>>> > >> On Tue, Feb 24, 2015 at 11:49 AM, Jeff Holoman < > >>>> jholo...@cloudera.com> > >>>> > >> wrote: > >>>> > >> > >>>> > >> > Jay / Joe > >>>> > >> > > >>>> > >> > We're happy to send out a Webex for this purpose. We could > record > >>>> the > >>>> > >> > sessions if there is interest and publish them out. > >>>> > >> > > >>>> > >> > Thanks > >>>> > >> > > >>>> > >> > Jeff > >>>> > >> > > >>>> > >> > On Tue, Feb 24, 2015 at 11:28 AM, Jay Kreps < > jay.kr...@gmail.com> > >>>> wrote: > >>>> > >> > > >>>> > >> > > Let's try to get the technical hang-ups sorted out, though. I > >>>> really > >>>> > >> > think > >>>> > >> > > there is some benefit to live discussion vs writing. I am > >>>> hopeful that > >>>> > >> if > >>>> > >> > > we post instructions and give ourselves a few attempts we can > >>>> get it > >>>> > >> > > working. > >>>> > >> > > > >>>> > >> > > Tuesday at that time would work for me...any objections? > >>>> > >> > > > >>>> > >> > > -Jay > >>>> > >> > > > >>>> > >> > > On Tue, Feb 24, 2015 at 8:18 AM, Joe Stein < > joe.st...@stealth.ly > >>>> > > >>>> > >> wrote: > >>>> > >> > > > >>>> > >> > > > Weekly would be great maybe like every Tuesday ~ 1pm ET / > 10am > >>>> PT > >>>> > >> ???? > >>>> > >> > > > > >>>> > >> > > > I don't mind google hangout but there is always some issue > or > >>>> > >> whatever > >>>> > >> > so > >>>> > >> > > > we know the apache irc channel works. We can start there > and > >>>> see how > >>>> > >> it > >>>> > >> > > > goes? We can pull transcripts too and associate to tickets > if > >>>> need be > >>>> > >> > > makes > >>>> > >> > > > it helpful for things. > >>>> > >> > > > > >>>> > >> > > > ~ Joestein > >>>> > >> > > > > >>>> > >> > > > On Tue, Feb 24, 2015 at 11:10 AM, Jay Kreps < > >>>> jay.kr...@gmail.com> > >>>> > >> > wrote: > >>>> > >> > > > > >>>> > >> > > > > We'd talked about doing a Google Hangout to chat about > this. > >>>> What > >>>> > >> > about > >>>> > >> > > > > generalizing that a little further...I actually think it > >>>> would be > >>>> > >> > good > >>>> > >> > > > for > >>>> > >> > > > > everyone spending a reasonable chunk of their week on > Kafka > >>>> stuff > >>>> > >> to > >>>> > >> > > > maybe > >>>> > >> > > > > sync up once a week. I think we could use time to talk > >>>> through > >>>> > >> design > >>>> > >> > > > > stuff, make sure we are on top of code reviews, talk > through > >>>> any > >>>> > >> > tricky > >>>> > >> > > > > issues, etc. > >>>> > >> > > > > > >>>> > >> > > > > We can make it publicly available so that any one can > follow > >>>> along > >>>> > >> > who > >>>> > >> > > > > likes. > >>>> > >> > > > > > >>>> > >> > > > > Any interest in doing this? If so I'll try to set it up > >>>> starting > >>>> > >> next > >>>> > >> > > > week. > >>>> > >> > > > > > >>>> > >> > > > > -Jay > >>>> > >> > > > > > >>>> > >> > > > > On Tue, Feb 24, 2015 at 3:57 AM, Andrii Biletskyi < > >>>> > >> > > > > andrii.bilets...@stealth.ly> wrote: > >>>> > >> > > > > > >>>> > >> > > > > > Hi all, > >>>> > >> > > > > > > >>>> > >> > > > > > I've updated KIP page, fixed / aligned document > structure. > >>>> Also I > >>>> > >> > > added > >>>> > >> > > > > > some > >>>> > >> > > > > > very initial proposal for AdminClient so we have > something > >>>> to > >>>> > >> start > >>>> > >> > > > from > >>>> > >> > > > > > while > >>>> > >> > > > > > discussing the KIP. > >>>> > >> > > > > > > >>>> > >> > > > > > > >>>> > >> > > > > > > >>>> > >> > > > > > >>>> > >> > > > > >>>> > >> > > > >>>> > >> > > >>>> > >> > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > >>>> > >> > > > > > > >>>> > >> > > > > > Thanks, > >>>> > >> > > > > > Andrii Biletskyi > >>>> > >> > > > > > > >>>> > >> > > > > > On Wed, Feb 18, 2015 at 9:01 PM, Andrii Biletskyi < > >>>> > >> > > > > > andrii.bilets...@stealth.ly> wrote: > >>>> > >> > > > > > > >>>> > >> > > > > > > Jay, > >>>> > >> > > > > > > > >>>> > >> > > > > > > Re error messages: you are right, in most cases > client > >>>> will > >>>> > >> have > >>>> > >> > > > enough > >>>> > >> > > > > > > context to show descriptive error message. My > concern is > >>>> that > >>>> > >> we > >>>> > >> > > will > >>>> > >> > > > > > have > >>>> > >> > > > > > > to > >>>> > >> > > > > > > add lots of new error codes for each possible error. > Of > >>>> course, > >>>> > >> > we > >>>> > >> > > > > could > >>>> > >> > > > > > > reuse > >>>> > >> > > > > > > some of existing like UknownTopicOrPartitionCode, > but we > >>>> will > >>>> > >> > also > >>>> > >> > > > need > >>>> > >> > > > > > to > >>>> > >> > > > > > > add smth like: TopicAlreadyExistsCode, > >>>> TopicConfigInvalid (both > >>>> > >> > for > >>>> > >> > > > > topic > >>>> > >> > > > > > > name and config, and probably user would like to know > >>>> what > >>>> > >> > exactly > >>>> > >> > > > > > > is wrong in his config), InvalidReplicaAssignment, > >>>> > >> InternalError > >>>> > >> > > > (e.g. > >>>> > >> > > > > > > zookeeper failure) etc. > >>>> > >> > > > > > > And this is only for TopicCommand, we will also need > to > >>>> add > >>>> > >> > similar > >>>> > >> > > > > stuff > >>>> > >> > > > > > > for > >>>> > >> > > > > > > ReassignPartitions, PreferredReplica. So we'll end up > >>>> with a > >>>> > >> > large > >>>> > >> > > > list > >>>> > >> > > > > > of > >>>> > >> > > > > > > error codes, used only in Admin protocol. > >>>> > >> > > > > > > Having said that, I agree my proposal is not > consistent > >>>> with > >>>> > >> > other > >>>> > >> > > > > cases. > >>>> > >> > > > > > > Maybe we can find better solution or something > >>>> in-between. > >>>> > >> > > > > > > > >>>> > >> > > > > > > Re Hangout chat: I think it is a great idea. This > way we > >>>> can > >>>> > >> move > >>>> > >> > > on > >>>> > >> > > > > > > faster. > >>>> > >> > > > > > > Let's agree somehow on date/time so people can join. > >>>> Will work > >>>> > >> > for > >>>> > >> > > me > >>>> > >> > > > > > this > >>>> > >> > > > > > > and > >>>> > >> > > > > > > next week almost anytime if agreed in advance. > >>>> > >> > > > > > > > >>>> > >> > > > > > > Thanks, > >>>> > >> > > > > > > Andrii > >>>> > >> > > > > > > > >>>> > >> > > > > > > On Wed, Feb 18, 2015 at 7:09 PM, Jay Kreps < > >>>> > >> jay.kr...@gmail.com> > >>>> > >> > > > > wrote: > >>>> > >> > > > > > > > >>>> > >> > > > > > >> Hey Andrii, > >>>> > >> > > > > > >> > >>>> > >> > > > > > >> Generally we can do good error handling without > needing > >>>> custom > >>>> > >> > > > > > server-side > >>>> > >> > > > > > >> messages. I.e. generally the client has the context > to > >>>> know > >>>> > >> that > >>>> > >> > > if > >>>> > >> > > > it > >>>> > >> > > > > > got > >>>> > >> > > > > > >> an error that the topic doesn't exist to say "Topic > X > >>>> doesn't > >>>> > >> > > exist" > >>>> > >> > > > > > >> rather > >>>> > >> > > > > > >> than "error code 14" (or whatever). Maybe there are > >>>> specific > >>>> > >> > cases > >>>> > >> > > > > where > >>>> > >> > > > > > >> this is hard? If we want to add server-side error > >>>> messages we > >>>> > >> > > really > >>>> > >> > > > > do > >>>> > >> > > > > > >> need to do this in a consistent way across the > protocol. > >>>> > >> > > > > > >> > >>>> > >> > > > > > >> I still have a bunch of open questions here from my > >>>> previous > >>>> > >> > > list. I > >>>> > >> > > > > > will > >>>> > >> > > > > > >> be out for the next few days for Strata though. > Maybe > >>>> we could > >>>> > >> > do > >>>> > >> > > a > >>>> > >> > > > > > Google > >>>> > >> > > > > > >> Hangout chat on any open issues some time towards > the > >>>> end of > >>>> > >> > next > >>>> > >> > > > week > >>>> > >> > > > > > for > >>>> > >> > > > > > >> anyone interested in this ticket? I have a feeling > that > >>>> might > >>>> > >> > > > progress > >>>> > >> > > > > > >> things a little faster than email--I think we could > talk > >>>> > >> through > >>>> > >> > > > those > >>>> > >> > > > > > >> issues I brought up fairly quickly... > >>>> > >> > > > > > >> > >>>> > >> > > > > > >> -Jay > >>>> > >> > > > > > >> > >>>> > >> > > > > > >> On Wed, Feb 18, 2015 at 7:27 AM, Andrii Biletskyi < > >>>> > >> > > > > > >> andrii.bilets...@stealth.ly> wrote: > >>>> > >> > > > > > >> > >>>> > >> > > > > > >> > Hi all, > >>>> > >> > > > > > >> > > >>>> > >> > > > > > >> > I'm trying to address some of the issues which > were > >>>> > >> mentioned > >>>> > >> > > > > earlier > >>>> > >> > > > > > >> about > >>>> > >> > > > > > >> > Admin RQ/RP format. One of those was about > batching > >>>> > >> > operations. > >>>> > >> > > > What > >>>> > >> > > > > > if > >>>> > >> > > > > > >> we > >>>> > >> > > > > > >> > follow TopicCommand approach and let people > specify > >>>> > >> topic-name > >>>> > >> > > by > >>>> > >> > > > > > >> regexp - > >>>> > >> > > > > > >> > would that cover most of the use cases? > >>>> > >> > > > > > >> > > >>>> > >> > > > > > >> > Secondly, is what information should we generally > >>>> provide in > >>>> > >> > > Admin > >>>> > >> > > > > > >> > responses. > >>>> > >> > > > > > >> > I realize that Admin commands don't imply they > will > >>>> be used > >>>> > >> > only > >>>> > >> > > > in > >>>> > >> > > > > > CLI > >>>> > >> > > > > > >> > but, > >>>> > >> > > > > > >> > it seems to me, CLI is a very important client of > this > >>>> > >> > feature. > >>>> > >> > > In > >>>> > >> > > > > > this > >>>> > >> > > > > > >> > case, > >>>> > >> > > > > > >> > seems logical, we would like to provide users with > >>>> rich > >>>> > >> > > experience > >>>> > >> > > > > in > >>>> > >> > > > > > >> terms > >>>> > >> > > > > > >> > of > >>>> > >> > > > > > >> > getting results / errors of the executed commands. > >>>> Usually > >>>> > >> we > >>>> > >> > > > supply > >>>> > >> > > > > > >> with > >>>> > >> > > > > > >> > responses only errorCode, which looks very > limiting, > >>>> in case > >>>> > >> > of > >>>> > >> > > > CLI > >>>> > >> > > > > we > >>>> > >> > > > > > >> may > >>>> > >> > > > > > >> > want to print human readable error description. > >>>> > >> > > > > > >> > > >>>> > >> > > > > > >> > So, taking into account previous item about > batching, > >>>> what > >>>> > >> do > >>>> > >> > > you > >>>> > >> > > > > > think > >>>> > >> > > > > > >> > about > >>>> > >> > > > > > >> > having smth like: > >>>> > >> > > > > > >> > > >>>> > >> > > > > > >> > ('create' doesn't support regexp) > >>>> > >> > > > > > >> > CreateTopicRequest => TopicName Partitions > Replicas > >>>> > >> > > > > ReplicaAssignment > >>>> > >> > > > > > >> > [Config] > >>>> > >> > > > > > >> > CreateTopicResponse => ErrorCode ErrorDescription > >>>> > >> > > > > > >> > ErrorCode => int16 > >>>> > >> > > > > > >> > ErrorDescription => string (empty if successful) > >>>> > >> > > > > > >> > > >>>> > >> > > > > > >> > AlterTopicRequest -> TopicNameRegexp Partitions > >>>> > >> > > ReplicaAssignment > >>>> > >> > > > > > >> > [AddedConfig] [DeletedConfig] > >>>> > >> > > > > > >> > AlterTopicResponse -> [TopicName ErrorCode > >>>> ErrorDescription] > >>>> > >> > > > > > >> > CommandErrorCode CommandErrorDescription > >>>> > >> > > > > > >> > CommandErrorCode => int16 > >>>> > >> > > > > > >> > CommandErrorDescription => string (nonempty in > case > >>>> of > >>>> > >> fatal > >>>> > >> > > > > error, > >>>> > >> > > > > > >> e.g. > >>>> > >> > > > > > >> > we couldn't get topics by regexp) > >>>> > >> > > > > > >> > > >>>> > >> > > > > > >> > DescribeTopicRequest -> TopicNameRegexp > >>>> > >> > > > > > >> > DescribeTopicResponse -> [TopicName > TopicDescription > >>>> > >> ErrorCode > >>>> > >> > > > > > >> > ErrorDescription] CommandErrorCode > >>>> CommandErrorDescription > >>>> > >> > > > > > >> > > >>>> > >> > > > > > >> > Also, any thoughts about our discussion regarding > >>>> re-routing > >>>> > >> > > > > facility? > >>>> > >> > > > > > >> In > >>>> > >> > > > > > >> > my > >>>> > >> > > > > > >> > understanding, it is like between augmenting > >>>> > >> > > TopicMetadataRequest > >>>> > >> > > > > > >> > (to include at least controllerId) and > implementing > >>>> new > >>>> > >> > generic > >>>> > >> > > > > > >> re-routing > >>>> > >> > > > > > >> > facility so sending messages to controller will be > >>>> handled > >>>> > >> by > >>>> > >> > > it. > >>>> > >> > > > > > >> > > >>>> > >> > > > > > >> > Thanks, > >>>> > >> > > > > > >> > Andrii Biletskyi > >>>> > >> > > > > > >> > > >>>> > >> > > > > > >> > On Mon, Feb 16, 2015 at 5:26 PM, Andrii Biletskyi > < > >>>> > >> > > > > > >> > andrii.bilets...@stealth.ly> wrote: > >>>> > >> > > > > > >> > > >>>> > >> > > > > > >> > > @Guozhang: > >>>> > >> > > > > > >> > > Thanks for your comments, I've answered some of > >>>> those. The > >>>> > >> > > main > >>>> > >> > > > > > thing > >>>> > >> > > > > > >> is > >>>> > >> > > > > > >> > > having merged request for > >>>> create-alter-delete-describe - I > >>>> > >> > > have > >>>> > >> > > > > some > >>>> > >> > > > > > >> > > concerns about this approach. > >>>> > >> > > > > > >> > > > >>>> > >> > > > > > >> > > @*Jay*: > >>>> > >> > > > > > >> > > I see that introduced ClusterMetadaRequest is > also > >>>> one of > >>>> > >> > the > >>>> > >> > > > > > >> concerns. > >>>> > >> > > > > > >> > We > >>>> > >> > > > > > >> > > can solve it if we implement re-routing > facility. > >>>> But I > >>>> > >> > agree > >>>> > >> > > > with > >>>> > >> > > > > > >> > > Guozhang - it will make clients' internals a > little > >>>> bit > >>>> > >> > easier > >>>> > >> > > > but > >>>> > >> > > > > > >> this > >>>> > >> > > > > > >> > > seems to be a complex logic to implement and > >>>> support then. > >>>> > >> > > > > > Especially > >>>> > >> > > > > > >> for > >>>> > >> > > > > > >> > > Fetch and Produce (even if we add re-routing > later > >>>> for > >>>> > >> these > >>>> > >> > > > > > >> requests). > >>>> > >> > > > > > >> > > Also people will tend to avoid this re-routing > >>>> facility > >>>> > >> and > >>>> > >> > > hold > >>>> > >> > > > > > local > >>>> > >> > > > > > >> > > cluster cache to ensure their high-priority > requests > >>>> > >> (which > >>>> > >> > > some > >>>> > >> > > > > of > >>>> > >> > > > > > >> the > >>>> > >> > > > > > >> > > admin requests are) not sent to some busy broker > >>>> where > >>>> > >> they > >>>> > >> > > wait > >>>> > >> > > > > to > >>>> > >> > > > > > be > >>>> > >> > > > > > >> > > routed to the correct one. > >>>> > >> > > > > > >> > > As pointed out by Jun here ( > >>>> > >> > > > > > >> > > > >>>> > >> > > > > > >> > > >>>> > >> > > > > > >> > >>>> > >> > > > > > > >>>> > >> > > > > > >>>> > >> > > > > >>>> > >> > > > >>>> > >> > > >>>> > >> > >>>> > https://issues.apache.org/jira/browse/KAFKA-1772?focusedCommentId=14234530&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14234530 > >>>> > >> > > > > > >> > ) > >>>> > >> > > > > > >> > > to solve the issue we might introduce a message > >>>> type to > >>>> > >> get > >>>> > >> > > > > cluster > >>>> > >> > > > > > >> > state. > >>>> > >> > > > > > >> > > But I agree we can just update > >>>> TopicMetadataResponse to > >>>> > >> > > include > >>>> > >> > > > > > >> > > controllerId (and probably smth else). > >>>> > >> > > > > > >> > > What are you thougths? > >>>> > >> > > > > > >> > > > >>>> > >> > > > > > >> > > Thanks, > >>>> > >> > > > > > >> > > Andrii > >>>> > >> > > > > > >> > > > >>>> > >> > > > > > >> > > On Thu, Feb 12, 2015 at 8:31 AM, Guozhang Wang < > >>>> > >> > > > > wangg...@gmail.com> > >>>> > >> > > > > > >> > wrote: > >>>> > >> > > > > > >> > > > >>>> > >> > > > > > >> > >> I think for the topics commands we can actually > >>>> merge > >>>> > >> > > > > > >> > >> create/alter/delete/describe as one request > type > >>>> since > >>>> > >> > their > >>>> > >> > > > > > formats > >>>> > >> > > > > > >> are > >>>> > >> > > > > > >> > >> very much similar, and keep list-topics and > others > >>>> like > >>>> > >> > > > > > >> > >> partition-reassignment / > preferred-leader-election > >>>> as > >>>> > >> > > separate > >>>> > >> > > > > > >> request > >>>> > >> > > > > > >> > >> types, I also left some other comments on the > RB ( > >>>> > >> > > > > > >> > >> https://reviews.apache.org/r/29301/). > >>>> > >> > > > > > >> > >> > >>>> > >> > > > > > >> > >> On Wed, Feb 11, 2015 at 2:04 PM, Jay Kreps < > >>>> > >> > > > jay.kr...@gmail.com> > >>>> > >> > > > > > >> wrote: > >>>> > >> > > > > > >> > >> > >>>> > >> > > > > > >> > >> > Yeah I totally agree that we don't want to > just > >>>> have > >>>> > >> one > >>>> > >> > > "do > >>>> > >> > > > > > admin > >>>> > >> > > > > > >> > >> stuff" > >>>> > >> > > > > > >> > >> > command that has the union of all parameters. > >>>> > >> > > > > > >> > >> > > >>>> > >> > > > > > >> > >> > What I am saying is that command line tools > are > >>>> one > >>>> > >> > client > >>>> > >> > > of > >>>> > >> > > > > the > >>>> > >> > > > > > >> > >> > administrative apis, but these will be used > in a > >>>> number > >>>> > >> > of > >>>> > >> > > > > > >> scenarios > >>>> > >> > > > > > >> > so > >>>> > >> > > > > > >> > >> > they should make logical sense even in the > >>>> absence of > >>>> > >> the > >>>> > >> > > > > command > >>>> > >> > > > > > >> line > >>>> > >> > > > > > >> > >> > tool. Hence comments like trying to clarify > the > >>>> > >> > > relationship > >>>> > >> > > > > > >> between > >>>> > >> > > > > > >> > >> > ClusterMetadata and TopicMetadata...these > kinds > >>>> of > >>>> > >> things > >>>> > >> > > > > really > >>>> > >> > > > > > >> need > >>>> > >> > > > > > >> > >> to be > >>>> > >> > > > > > >> > >> > thought through. > >>>> > >> > > > > > >> > >> > > >>>> > >> > > > > > >> > >> > Hope that makes sense. > >>>> > >> > > > > > >> > >> > > >>>> > >> > > > > > >> > >> > -Jay > >>>> > >> > > > > > >> > >> > > >>>> > >> > > > > > >> > >> > On Wed, Feb 11, 2015 at 1:41 PM, Andrii > >>>> Biletskyi < > >>>> > >> > > > > > >> > >> > andrii.bilets...@stealth.ly> wrote: > >>>> > >> > > > > > >> > >> > > >>>> > >> > > > > > >> > >> > > Jay, > >>>> > >> > > > > > >> > >> > > > >>>> > >> > > > > > >> > >> > > Thanks for answering. You understood > >>>> correctly, most > >>>> > >> of > >>>> > >> > > my > >>>> > >> > > > > > >> comments > >>>> > >> > > > > > >> > >> were > >>>> > >> > > > > > >> > >> > > related to your point 1) - about "well > >>>> thought-out" > >>>> > >> > apis. > >>>> > >> > > > > Also, > >>>> > >> > > > > > >> yes, > >>>> > >> > > > > > >> > >> as I > >>>> > >> > > > > > >> > >> > > understood we would like to introduce a > single > >>>> > >> unified > >>>> > >> > > CLI > >>>> > >> > > > > tool > >>>> > >> > > > > > >> with > >>>> > >> > > > > > >> > >> > > centralized server-side request handling > for > >>>> lots of > >>>> > >> > > > existing > >>>> > >> > > > > > >> ones > >>>> > >> > > > > > >> > >> (incl. > >>>> > >> > > > > > >> > >> > > TopicCommand, CommitOffsetChecker, > >>>> > >> ReassignPartitions, > >>>> > >> > > smth > >>>> > >> > > > > > else > >>>> > >> > > > > > >> if > >>>> > >> > > > > > >> > >> added > >>>> > >> > > > > > >> > >> > > in future). In our previous discussion ( > >>>> > >> > > > > > >> > >> > > > >>>> https://issues.apache.org/jira/browse/KAFKA-1694) > >>>> > >> > people > >>>> > >> > > > > said > >>>> > >> > > > > > >> > they'd > >>>> > >> > > > > > >> > >> > > rather > >>>> > >> > > > > > >> > >> > > have a separate message for each command, > so, > >>>> yes, > >>>> > >> this > >>>> > >> > > > way I > >>>> > >> > > > > > >> came > >>>> > >> > > > > > >> > to > >>>> > >> > > > > > >> > >> 1-1 > >>>> > >> > > > > > >> > >> > > mapping between commands in the tool and > >>>> protocol > >>>> > >> > > > additions. > >>>> > >> > > > > > But > >>>> > >> > > > > > >> I > >>>> > >> > > > > > >> > >> might > >>>> > >> > > > > > >> > >> > be > >>>> > >> > > > > > >> > >> > > wrong. > >>>> > >> > > > > > >> > >> > > At the end I just try to start discussion > how > >>>> at > >>>> > >> least > >>>> > >> > > > > > generally > >>>> > >> > > > > > >> > this > >>>> > >> > > > > > >> > >> > > protocol should look like. > >>>> > >> > > > > > >> > >> > > > >>>> > >> > > > > > >> > >> > > Thanks, > >>>> > >> > > > > > >> > >> > > Andrii > >>>> > >> > > > > > >> > >> > > > >>>> > >> > > > > > >> > >> > > On Wed, Feb 11, 2015 at 11:10 PM, Jay > Kreps < > >>>> > >> > > > > > jay.kr...@gmail.com > >>>> > >> > > > > > >> > > >>>> > >> > > > > > >> > >> wrote: > >>>> > >> > > > > > >> > >> > > > >>>> > >> > > > > > >> > >> > > > Hey Andrii, > >>>> > >> > > > > > >> > >> > > > > >>>> > >> > > > > > >> > >> > > > To answer your earlier question we just > >>>> really > >>>> > >> can't > >>>> > >> > be > >>>> > >> > > > > > adding > >>>> > >> > > > > > >> any > >>>> > >> > > > > > >> > >> more > >>>> > >> > > > > > >> > >> > > > scala protocol objects. These things are > >>>> super hard > >>>> > >> > to > >>>> > >> > > > > > maintain > >>>> > >> > > > > > >> > >> because > >>>> > >> > > > > > >> > >> > > > they hand code the byte parsing and don't > >>>> have good > >>>> > >> > > > > > versioning > >>>> > >> > > > > > >> > >> support. > >>>> > >> > > > > > >> > >> > > > Since we are already planning on > converting > >>>> we > >>>> > >> > > definitely > >>>> > >> > > > > > don't > >>>> > >> > > > > > >> > >> want to > >>>> > >> > > > > > >> > >> > > add > >>>> > >> > > > > > >> > >> > > > a ton more of these--they are total tech > >>>> debt. > >>>> > >> > > > > > >> > >> > > > > >>>> > >> > > > > > >> > >> > > > What does it mean that the changes are > >>>> isolated > >>>> > >> from > >>>> > >> > > the > >>>> > >> > > > > > >> current > >>>> > >> > > > > > >> > >> code > >>>> > >> > > > > > >> > >> > > base? > >>>> > >> > > > > > >> > >> > > > > >>>> > >> > > > > > >> > >> > > > I actually didn't understand the > remaining > >>>> > >> comments, > >>>> > >> > > > which > >>>> > >> > > > > of > >>>> > >> > > > > > >> the > >>>> > >> > > > > > >> > >> > points > >>>> > >> > > > > > >> > >> > > > are you responding to? > >>>> > >> > > > > > >> > >> > > > > >>>> > >> > > > > > >> > >> > > > Maybe one sticking point here is that it > >>>> seems like > >>>> > >> > you > >>>> > >> > > > > want > >>>> > >> > > > > > to > >>>> > >> > > > > > >> > make > >>>> > >> > > > > > >> > >> > some > >>>> > >> > > > > > >> > >> > > > kind of tool, and you have made a 1-1 > mapping > >>>> > >> between > >>>> > >> > > > > > commands > >>>> > >> > > > > > >> you > >>>> > >> > > > > > >> > >> > > imagine > >>>> > >> > > > > > >> > >> > > > in the tool and protocol additions. I > want > >>>> to make > >>>> > >> > sure > >>>> > >> > > > we > >>>> > >> > > > > > >> don't > >>>> > >> > > > > > >> > do > >>>> > >> > > > > > >> > >> > that. > >>>> > >> > > > > > >> > >> > > > The protocol needs to be really really > well > >>>> thought > >>>> > >> > out > >>>> > >> > > > > > against > >>>> > >> > > > > > >> > many > >>>> > >> > > > > > >> > >> > use > >>>> > >> > > > > > >> > >> > > > cases so it should make perfect logical > >>>> sense in > >>>> > >> the > >>>> > >> > > > > absence > >>>> > >> > > > > > of > >>>> > >> > > > > > >> > >> knowing > >>>> > >> > > > > > >> > >> > > the > >>>> > >> > > > > > >> > >> > > > command line tool, right? > >>>> > >> > > > > > >> > >> > > > > >>>> > >> > > > > > >> > >> > > > -Jay > >>>> > >> > > > > > >> > >> > > > > >>>> > >> > > > > > >> > >> > > > On Wed, Feb 11, 2015 at 11:57 AM, Andrii > >>>> Biletskyi > >>>> > >> < > >>>> > >> > > > > > >> > >> > > > andrii.bilets...@stealth.ly> wrote: > >>>> > >> > > > > > >> > >> > > > > >>>> > >> > > > > > >> > >> > > > > Hey Jay, > >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > I would like to continue this > discussion > >>>> as it > >>>> > >> seem > >>>> > >> > > > there > >>>> > >> > > > > > is > >>>> > >> > > > > > >> no > >>>> > >> > > > > > >> > >> > > progress > >>>> > >> > > > > > >> > >> > > > > here. > >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > First of all, could you please explain > >>>> what did > >>>> > >> you > >>>> > >> > > > mean > >>>> > >> > > > > in > >>>> > >> > > > > > >> 2? > >>>> > >> > > > > > >> > How > >>>> > >> > > > > > >> > >> > > > exactly > >>>> > >> > > > > > >> > >> > > > > are we going to migrate to the new java > >>>> protocol > >>>> > >> > > > > > definitions. > >>>> > >> > > > > > >> > And > >>>> > >> > > > > > >> > >> why > >>>> > >> > > > > > >> > >> > > > it's > >>>> > >> > > > > > >> > >> > > > > a blocker for centralized CLI? > >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > I agree with you, this feature includes > >>>> lots of > >>>> > >> > > stuff, > >>>> > >> > > > > but > >>>> > >> > > > > > >> > >> thankfully > >>>> > >> > > > > > >> > >> > > > > almost all changes are isolated from > the > >>>> current > >>>> > >> > code > >>>> > >> > > > > base, > >>>> > >> > > > > > >> > >> > > > > so the main thing, I think, we need to > >>>> agree is > >>>> > >> > RQ/RP > >>>> > >> > > > > > format. > >>>> > >> > > > > > >> > >> > > > > So how can we start discussion about > the > >>>> concrete > >>>> > >> > > > > messages > >>>> > >> > > > > > >> > format? > >>>> > >> > > > > > >> > >> > > > > Can we take ( > >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > >>>> > >> > > > > > >> > >> > > > >>>> > >> > > > > > >> > >> > > >>>> > >> > > > > > >> > >> > >>>> > >> > > > > > >> > > >>>> > >> > > > > > >> > >>>> > >> > > > > > > >>>> > >> > > > > > >>>> > >> > > > > >>>> > >> > > > >>>> > >> > > >>>> > >> > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-ProposedRQ/RPFormat > >>>> > >> > > > > > >> > >> > > > > ) > >>>> > >> > > > > > >> > >> > > > > as starting point? > >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > We had some doubts earlier whether it > worth > >>>> > >> > > introducing > >>>> > >> > > > > one > >>>> > >> > > > > > >> > >> generic > >>>> > >> > > > > > >> > >> > > Admin > >>>> > >> > > > > > >> > >> > > > > Request for all commands ( > >>>> > >> > > > > > >> > >> > > > > >>>> https://issues.apache.org/jira/browse/KAFKA-1694 > >>>> > >> > > > > > >> > >> > > > > ) > >>>> > >> > > > > > >> > >> > > > > but then everybody agreed it would be > >>>> better to > >>>> > >> > have > >>>> > >> > > > > > separate > >>>> > >> > > > > > >> > >> message > >>>> > >> > > > > > >> > >> > > for > >>>> > >> > > > > > >> > >> > > > > each admin command. The Request part is > >>>> really > >>>> > >> > > dictated > >>>> > >> > > > > > from > >>>> > >> > > > > > >> the > >>>> > >> > > > > > >> > >> > > command > >>>> > >> > > > > > >> > >> > > > > (e.g. TopicCommand) arguments itself, > so > >>>> the > >>>> > >> > proposed > >>>> > >> > > > > > version > >>>> > >> > > > > > >> > >> should > >>>> > >> > > > > > >> > >> > be > >>>> > >> > > > > > >> > >> > > > > fine (let's put aside for now remarks > about > >>>> > >> > Optional > >>>> > >> > > > > type, > >>>> > >> > > > > > >> > >> batching, > >>>> > >> > > > > > >> > >> > > > > configs normalization - I agree with > all of > >>>> > >> them). > >>>> > >> > > > > > >> > >> > > > > So the second part is Response. I see > >>>> there are > >>>> > >> two > >>>> > >> > > > cases > >>>> > >> > > > > > >> here. > >>>> > >> > > > > > >> > >> > > > > a) "Mutate" requests - > Create/Alter/... ; > >>>> b) > >>>> > >> "Get" > >>>> > >> > > > > > requests - > >>>> > >> > > > > > >> > >> > > > > List/Describe... > >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > a) should only hold request result > >>>> (regardless > >>>> > >> what > >>>> > >> > > we > >>>> > >> > > > > > decide > >>>> > >> > > > > > >> > >> about > >>>> > >> > > > > > >> > >> > > > > blocking/non-blocking commands > execution). > >>>> > >> > > > > > >> > >> > > > > Usually we provide error code in > response > >>>> but > >>>> > >> since > >>>> > >> > > we > >>>> > >> > > > > will > >>>> > >> > > > > > >> use > >>>> > >> > > > > > >> > >> this > >>>> > >> > > > > > >> > >> > in > >>>> > >> > > > > > >> > >> > > > > interactive shell we need some human > >>>> readable > >>>> > >> error > >>>> > >> > > > > > >> description > >>>> > >> > > > > > >> > - > >>>> > >> > > > > > >> > >> so > >>>> > >> > > > > > >> > >> > I > >>>> > >> > > > > > >> > >> > > > > added errorDesription field where you > can > >>>> at > >>>> > >> least > >>>> > >> > > > leave > >>>> > >> > > > > > >> > >> > > > > exception.getMessage. > >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > b) in addition to previous item message > >>>> should > >>>> > >> hold > >>>> > >> > > > > command > >>>> > >> > > > > > >> > >> specific > >>>> > >> > > > > > >> > >> > > > > response data. We can discuss in detail > >>>> each of > >>>> > >> > them > >>>> > >> > > > but > >>>> > >> > > > > > >> let's > >>>> > >> > > > > > >> > for > >>>> > >> > > > > > >> > >> > now > >>>> > >> > > > > > >> > >> > > > > agree about the overall pattern. > >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > Thanks, > >>>> > >> > > > > > >> > >> > > > > Andrii Biletskyi > >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > On Fri, Jan 23, 2015 at 6:59 AM, Jay > Kreps > >>>> < > >>>> > >> > > > > > >> jay.kr...@gmail.com > >>>> > >> > > > > > >> > > > >>>> > >> > > > > > >> > >> > > wrote: > >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > > Hey Joe, > >>>> > >> > > > > > >> > >> > > > > > > >>>> > >> > > > > > >> > >> > > > > > This is great. A few comments on > KIP-4 > >>>> > >> > > > > > >> > >> > > > > > > >>>> > >> > > > > > >> > >> > > > > > 1. This is much needed functionality, > >>>> but there > >>>> > >> > > are a > >>>> > >> > > > > lot > >>>> > >> > > > > > >> of > >>>> > >> > > > > > >> > >> the so > >>>> > >> > > > > > >> > >> > > > let's > >>>> > >> > > > > > >> > >> > > > > > really think these protocols > through. We > >>>> really > >>>> > >> > > want > >>>> > >> > > > to > >>>> > >> > > > > > >> end up > >>>> > >> > > > > > >> > >> > with a > >>>> > >> > > > > > >> > >> > > > set > >>>> > >> > > > > > >> > >> > > > > > of well thought-out, orthoganol apis. > >>>> For this > >>>> > >> > > > reason I > >>>> > >> > > > > > >> think > >>>> > >> > > > > > >> > >> it is > >>>> > >> > > > > > >> > >> > > > > really > >>>> > >> > > > > > >> > >> > > > > > important to think through the end > state > >>>> even > >>>> > >> if > >>>> > >> > > that > >>>> > >> > > > > > >> includes > >>>> > >> > > > > > >> > >> APIs > >>>> > >> > > > > > >> > >> > > we > >>>> > >> > > > > > >> > >> > > > > > won't implement in the first phase. > >>>> > >> > > > > > >> > >> > > > > > > >>>> > >> > > > > > >> > >> > > > > > 2. Let's please please please wait > until > >>>> we > >>>> > >> have > >>>> > >> > > > > switched > >>>> > >> > > > > > >> the > >>>> > >> > > > > > >> > >> > server > >>>> > >> > > > > > >> > >> > > > over > >>>> > >> > > > > > >> > >> > > > > > to the new java protocol > definitions. If > >>>> we add > >>>> > >> > > > upteen > >>>> > >> > > > > > >> more ad > >>>> > >> > > > > > >> > >> hoc > >>>> > >> > > > > > >> > >> > > > scala > >>>> > >> > > > > > >> > >> > > > > > objects that is just generating more > >>>> work for > >>>> > >> the > >>>> > >> > > > > > >> conversion > >>>> > >> > > > > > >> > we > >>>> > >> > > > > > >> > >> > know > >>>> > >> > > > > > >> > >> > > we > >>>> > >> > > > > > >> > >> > > > > > have to do. > >>>> > >> > > > > > >> > >> > > > > > > >>>> > >> > > > > > >> > >> > > > > > 3. This proposal introduces a new > type of > >>>> > >> > optional > >>>> > >> > > > > > >> parameter. > >>>> > >> > > > > > >> > >> This > >>>> > >> > > > > > >> > >> > is > >>>> > >> > > > > > >> > >> > > > > > inconsistent with everything else in > the > >>>> > >> protocol > >>>> > >> > > > where > >>>> > >> > > > > > we > >>>> > >> > > > > > >> use > >>>> > >> > > > > > >> > >> -1 > >>>> > >> > > > > > >> > >> > or > >>>> > >> > > > > > >> > >> > > > some > >>>> > >> > > > > > >> > >> > > > > > other marker value. You could argue > >>>> either way > >>>> > >> > but > >>>> > >> > > > > let's > >>>> > >> > > > > > >> stick > >>>> > >> > > > > > >> > >> with > >>>> > >> > > > > > >> > >> > > > that > >>>> > >> > > > > > >> > >> > > > > > for consistency. For clients that > >>>> implemented > >>>> > >> the > >>>> > >> > > > > > protocol > >>>> > >> > > > > > >> in > >>>> > >> > > > > > >> > a > >>>> > >> > > > > > >> > >> > > better > >>>> > >> > > > > > >> > >> > > > > way > >>>> > >> > > > > > >> > >> > > > > > than our scala code these basic > >>>> primitives are > >>>> > >> > hard > >>>> > >> > > > to > >>>> > >> > > > > > >> change. > >>>> > >> > > > > > >> > >> > > > > > > >>>> > >> > > > > > >> > >> > > > > > 4. ClusterMetadata: This seems to > >>>> duplicate > >>>> > >> > > > > > >> > TopicMetadataRequest > >>>> > >> > > > > > >> > >> > > which > >>>> > >> > > > > > >> > >> > > > > has > >>>> > >> > > > > > >> > >> > > > > > brokers, topics, and partitions. I > think > >>>> we > >>>> > >> > should > >>>> > >> > > > > rename > >>>> > >> > > > > > >> that > >>>> > >> > > > > > >> > >> > > request > >>>> > >> > > > > > >> > >> > > > > > ClusterMetadataRequest (or just > >>>> > >> MetadataRequest) > >>>> > >> > > and > >>>> > >> > > > > > >> include > >>>> > >> > > > > > >> > >> the id > >>>> > >> > > > > > >> > >> > > of > >>>> > >> > > > > > >> > >> > > > > the > >>>> > >> > > > > > >> > >> > > > > > controller. Or are there other > things we > >>>> could > >>>> > >> > add > >>>> > >> > > > > here? > >>>> > >> > > > > > >> > >> > > > > > > >>>> > >> > > > > > >> > >> > > > > > 5. We have a tendency to try to make > a > >>>> lot of > >>>> > >> > > > requests > >>>> > >> > > > > > that > >>>> > >> > > > > > >> > can > >>>> > >> > > > > > >> > >> > only > >>>> > >> > > > > > >> > >> > > go > >>>> > >> > > > > > >> > >> > > > > to > >>>> > >> > > > > > >> > >> > > > > > particular nodes. This adds a lot of > >>>> burden for > >>>> > >> > > > client > >>>> > >> > > > > > >> > >> > > implementations > >>>> > >> > > > > > >> > >> > > > > (it > >>>> > >> > > > > > >> > >> > > > > > sounds easy but each discovery can > fail > >>>> in many > >>>> > >> > > parts > >>>> > >> > > > > so > >>>> > >> > > > > > it > >>>> > >> > > > > > >> > >> ends up > >>>> > >> > > > > > >> > >> > > > > being a > >>>> > >> > > > > > >> > >> > > > > > full state machine to do right). I > think > >>>> we > >>>> > >> > should > >>>> > >> > > > > > consider > >>>> > >> > > > > > >> > >> making > >>>> > >> > > > > > >> > >> > > > admin > >>>> > >> > > > > > >> > >> > > > > > commands and ideally as many of the > >>>> other apis > >>>> > >> as > >>>> > >> > > > > > possible > >>>> > >> > > > > > >> > >> > available > >>>> > >> > > > > > >> > >> > > on > >>>> > >> > > > > > >> > >> > > > > all > >>>> > >> > > > > > >> > >> > > > > > brokers and just redirect to the > >>>> controller on > >>>> > >> > the > >>>> > >> > > > > broker > >>>> > >> > > > > > >> > side. > >>>> > >> > > > > > >> > >> > > Perhaps > >>>> > >> > > > > > >> > >> > > > > > there would be a general way to > >>>> encapsulate > >>>> > >> this > >>>> > >> > > > > > re-routing > >>>> > >> > > > > > >> > >> > behavior. > >>>> > >> > > > > > >> > >> > > > > > > >>>> > >> > > > > > >> > >> > > > > > 6. We should probably normalize the > key > >>>> value > >>>> > >> > pairs > >>>> > >> > > > > used > >>>> > >> > > > > > >> for > >>>> > >> > > > > > >> > >> > configs > >>>> > >> > > > > > >> > >> > > > > rather > >>>> > >> > > > > > >> > >> > > > > > than embedding a new formatting. So > two > >>>> strings > >>>> > >> > > > rather > >>>> > >> > > > > > than > >>>> > >> > > > > > >> > one > >>>> > >> > > > > > >> > >> > with > >>>> > >> > > > > > >> > >> > > an > >>>> > >> > > > > > >> > >> > > > > > internal equals sign. > >>>> > >> > > > > > >> > >> > > > > > > >>>> > >> > > > > > >> > >> > > > > > 7. Is the postcondition of these APIs > >>>> that the > >>>> > >> > > > command > >>>> > >> > > > > > has > >>>> > >> > > > > > >> > >> begun or > >>>> > >> > > > > > >> > >> > > > that > >>>> > >> > > > > > >> > >> > > > > > the command has been completed? It > is a > >>>> lot > >>>> > >> more > >>>> > >> > > > usable > >>>> > >> > > > > > if > >>>> > >> > > > > > >> the > >>>> > >> > > > > > >> > >> > > command > >>>> > >> > > > > > >> > >> > > > > has > >>>> > >> > > > > > >> > >> > > > > > been completed so you know that if > you > >>>> create a > >>>> > >> > > topic > >>>> > >> > > > > and > >>>> > >> > > > > > >> then > >>>> > >> > > > > > >> > >> > > publish > >>>> > >> > > > > > >> > >> > > > to > >>>> > >> > > > > > >> > >> > > > > > it you won't get an exception about > >>>> there being > >>>> > >> > no > >>>> > >> > > > such > >>>> > >> > > > > > >> topic. > >>>> > >> > > > > > >> > >> > > > > > > >>>> > >> > > > > > >> > >> > > > > > 8. Describe topic and list topics > >>>> duplicate a > >>>> > >> lot > >>>> > >> > > of > >>>> > >> > > > > > stuff > >>>> > >> > > > > > >> in > >>>> > >> > > > > > >> > >> the > >>>> > >> > > > > > >> > >> > > > > metadata > >>>> > >> > > > > > >> > >> > > > > > request. Is there a reason to give > back > >>>> topics > >>>> > >> > > marked > >>>> > >> > > > > for > >>>> > >> > > > > > >> > >> > deletion? I > >>>> > >> > > > > > >> > >> > > > > feel > >>>> > >> > > > > > >> > >> > > > > > like if we just make the > post-condition > >>>> of the > >>>> > >> > > delete > >>>> > >> > > > > > >> command > >>>> > >> > > > > > >> > be > >>>> > >> > > > > > >> > >> > that > >>>> > >> > > > > > >> > >> > > > the > >>>> > >> > > > > > >> > >> > > > > > topic is deleted that will get rid of > >>>> the need > >>>> > >> > for > >>>> > >> > > > this > >>>> > >> > > > > > >> right? > >>>> > >> > > > > > >> > >> And > >>>> > >> > > > > > >> > >> > it > >>>> > >> > > > > > >> > >> > > > > will > >>>> > >> > > > > > >> > >> > > > > > be much more intuitive. > >>>> > >> > > > > > >> > >> > > > > > > >>>> > >> > > > > > >> > >> > > > > > 9. Should we consider batching these > >>>> requests? > >>>> > >> We > >>>> > >> > > > have > >>>> > >> > > > > > >> > generally > >>>> > >> > > > > > >> > >> > > tried > >>>> > >> > > > > > >> > >> > > > to > >>>> > >> > > > > > >> > >> > > > > > allow multiple operations to be > batched. > >>>> My > >>>> > >> > > suspicion > >>>> > >> > > > > is > >>>> > >> > > > > > >> that > >>>> > >> > > > > > >> > >> > without > >>>> > >> > > > > > >> > >> > > > > this > >>>> > >> > > > > > >> > >> > > > > > we will get a lot of code that does > >>>> something > >>>> > >> > like > >>>> > >> > > > > > >> > >> > > > > > for(topic: > adminClient.listTopics()) > >>>> > >> > > > > > >> > >> > > > > > > adminClient.describeTopic(topic) > >>>> > >> > > > > > >> > >> > > > > > this code will work great when you > test > >>>> on 5 > >>>> > >> > topics > >>>> > >> > > > but > >>>> > >> > > > > > >> not do > >>>> > >> > > > > > >> > >> as > >>>> > >> > > > > > >> > >> > > well > >>>> > >> > > > > > >> > >> > > > if > >>>> > >> > > > > > >> > >> > > > > > you have 50k. > >>>> > >> > > > > > >> > >> > > > > > > >>>> > >> > > > > > >> > >> > > > > > 10. I think we should also discuss > how > >>>> we want > >>>> > >> to > >>>> > >> > > > > expose > >>>> > >> > > > > > a > >>>> > >> > > > > > >> > >> > > programmatic > >>>> > >> > > > > > >> > >> > > > > JVM > >>>> > >> > > > > > >> > >> > > > > > client api for these operations. > >>>> Currently > >>>> > >> people > >>>> > >> > > > rely > >>>> > >> > > > > on > >>>> > >> > > > > > >> > >> > AdminUtils > >>>> > >> > > > > > >> > >> > > > > which > >>>> > >> > > > > > >> > >> > > > > > is totally sketchy. I think we > probably > >>>> need > >>>> > >> > > another > >>>> > >> > > > > > client > >>>> > >> > > > > > >> > >> under > >>>> > >> > > > > > >> > >> > > > > clients/ > >>>> > >> > > > > > >> > >> > > > > > that exposes administrative > >>>> functionality. We > >>>> > >> > will > >>>> > >> > > > need > >>>> > >> > > > > > >> this > >>>> > >> > > > > > >> > >> just > >>>> > >> > > > > > >> > >> > to > >>>> > >> > > > > > >> > >> > > > > > properly test the new apis, I > suspect. We > >>>> > >> should > >>>> > >> > > > figure > >>>> > >> > > > > > out > >>>> > >> > > > > > >> > that > >>>> > >> > > > > > >> > >> > API. > >>>> > >> > > > > > >> > >> > > > > > > >>>> > >> > > > > > >> > >> > > > > > 11. The other information that would > be > >>>> really > >>>> > >> > > useful > >>>> > >> > > > > to > >>>> > >> > > > > > >> get > >>>> > >> > > > > > >> > >> would > >>>> > >> > > > > > >> > >> > be > >>>> > >> > > > > > >> > >> > > > > > information about partitions--how > much > >>>> data is > >>>> > >> in > >>>> > >> > > the > >>>> > >> > > > > > >> > partition, > >>>> > >> > > > > > >> > >> > what > >>>> > >> > > > > > >> > >> > > > are > >>>> > >> > > > > > >> > >> > > > > > the segment offsets, what is the > log-end > >>>> offset > >>>> > >> > > (i.e. > >>>> > >> > > > > > last > >>>> > >> > > > > > >> > >> offset), > >>>> > >> > > > > > >> > >> > > > what > >>>> > >> > > > > > >> > >> > > > > is > >>>> > >> > > > > > >> > >> > > > > > the compaction point, etc. I think > that > >>>> done > >>>> > >> > right > >>>> > >> > > > this > >>>> > >> > > > > > >> would > >>>> > >> > > > > > >> > be > >>>> > >> > > > > > >> > >> > the > >>>> > >> > > > > > >> > >> > > > > > successor to the very awkward > >>>> OffsetRequest we > >>>> > >> > have > >>>> > >> > > > > > today. > >>>> > >> > > > > > >> > >> > > > > > > >>>> > >> > > > > > >> > >> > > > > > -Jay > >>>> > >> > > > > > >> > >> > > > > > > >>>> > >> > > > > > >> > >> > > > > > On Wed, Jan 21, 2015 at 10:27 PM, Joe > >>>> Stein < > >>>> > >> > > > > > >> > >> joe.st...@stealth.ly> > >>>> > >> > > > > > >> > >> > > > > wrote: > >>>> > >> > > > > > >> > >> > > > > > > >>>> > >> > > > > > >> > >> > > > > > > Hi, created a KIP > >>>> > >> > > > > > >> > >> > > > > > > > >>>> > >> > > > > > >> > >> > > > > > > > >>>> > >> > > > > > >> > >> > > > > > > >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > >>>> > >> > > > > > >> > >> > > > >>>> > >> > > > > > >> > >> > > >>>> > >> > > > > > >> > >> > >>>> > >> > > > > > >> > > >>>> > >> > > > > > >> > >>>> > >> > > > > > > >>>> > >> > > > > > >>>> > >> > > > > >>>> > >> > > > >>>> > >> > > >>>> > >> > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > >>>> > >> > > > > > >> > >> > > > > > > > >>>> > >> > > > > > >> > >> > > > > > > JIRA > >>>> > >> > > > > https://issues.apache.org/jira/browse/KAFKA-1694 > >>>> > >> > > > > > >> > >> > > > > > > > >>>> > >> > > > > > >> > >> > > > > > > > >>>> /******************************************* > >>>> > >> > > > > > >> > >> > > > > > > Joe Stein > >>>> > >> > > > > > >> > >> > > > > > > Founder, Principal Consultant > >>>> > >> > > > > > >> > >> > > > > > > Big Data Open Source Security LLC > >>>> > >> > > > > > >> > >> > > > > > > http://www.stealth.ly > >>>> > >> > > > > > >> > >> > > > > > > Twitter: @allthingshadoop < > >>>> > >> > > > > > >> > >> > http://www.twitter.com/allthingshadoop > >>>> > >> > > > > > >> > >> > > > > >>>> > >> > > > > > >> > >> > > > > > > > >>>> ********************************************/ > >>>> > >> > > > > > >> > >> > > > > > > > >>>> > >> > > > > > >> > >> > > > > > > >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > >>>> > >> > > > > > >> > >> > > > >>>> > >> > > > > > >> > >> > > >>>> > >> > > > > > >> > >> > >>>> > >> > > > > > >> > >> > >>>> > >> > > > > > >> > >> > >>>> > >> > > > > > >> > >> -- > >>>> > >> > > > > > >> > >> -- Guozhang > >>>> > >> > > > > > >> > >> > >>>> > >> > > > > > >> > > > >>>> > >> > > > > > >> > > > >>>> > >> > > > > > >> > > >>>> > >> > > > > > >> > >>>> > >> > > > > > > > >>>> > >> > > > > > > > >>>> > >> > > > > > > >>>> > >> > > > > > >>>> > >> > > > > >>>> > >> > > > >>>> > >> > > >>>> > >> > > >>>> > >> > > >>>> > >> > -- > >>>> > >> > Jeff Holoman > >>>> > >> > Systems Engineer > >>>> > >> > > >>>> > >> > >>>> > >>>> > >>> > >>> > >>> -- > >>> -- Guozhang > >>> > >> > >> > >> > >> -- > >> -- Guozhang >
