Joel, You are right, I removed ClusterMetadata because we have partially what we need in TopicMetadata. Also, as Jay pointed out earlier, we would like to have "orthogonal" API, but at the same time we need to be backward compatible.
But I like your idea and even have some other arguments for this option: There is also DescribeTopicRequest which was proposed in this KIP, it returns topic configs, partitions, replication factor plus partition ISR, ASR, leader replica. The later part is really already there in TopicMetadataRequest. So again we'll have to add stuff to TMR, not to duplicate some info in newly added requests. However, this way we'll end up with "monster" request which returns cluster metadata, topic replication and config info plus partition replication data. Seems logical to split TMR to - ClusterMetadata (brokers + controller, maybe smth else) - TopicMetadata (topic info + partition details) But since current TMR is involved in lots of places (including network client, as I understand) this might be very serious change and it probably makes sense to stick with current approach. Thanks, Andrii Biletskyi On Tue, Mar 17, 2015 at 5:29 PM, Joel Koshy <jjkosh...@gmail.com> wrote: > I may be missing some context but hopefully this will also be covered > today: I thought the earlier proposal where there was an explicit > ClusterMetadata request was clearer and explicit. During the course of > this thread I think the conclusion was that the main need was for > controller information and that can be rolled into the topic metadata > response but that seems a bit irrelevant to topic metadata. FWIW I > think the full broker-list is also irrelevant to topic metadata, but > it is already there and in use. I think there is still room for an > explicit ClusterMetadata request since there may be other > cluster-level information that we may want to add over time (and that > have nothing to do with topic metadata). > > On Tue, Mar 17, 2015 at 02:45:30PM +0200, Andrii Biletskyi wrote: > > Jun, > > > > 101. Okay, if you say that such use case is important. I also think > > using clientId for these purposes is fine - if we already have this field > > as part of all Wire protocol messages, why not use that. > > I will update KIP-4 page if nobody has other ideas (which may come up > > during the call today). > > > > 102.1 Agree, I'll update the KIP accordingly. I think we can add new, > > fine-grained error codes if some error code received in specific case > > won't give enough context to return a descriptive error message for user. > > > > Look forward to discussing all outstanding issues in detail today during > > the call. > > > > Thanks, > > Andrii Biletskyi > > > > > > > > On Mon, Mar 16, 2015 at 10:59 PM, Jun Rao <j...@confluent.io> wrote: > > > > > 101. There may be a use case where you only want the topics to be > created > > > manually by admins. Currently, you can do that by disabling auto topic > > > creation and issue topic creation from the TopicCommand. If we disable > auto > > > topic creation completely on the broker and don't have a way to > distinguish > > > between topic creation requests from the regular clients and the > admin, we > > > can't support manual topic creation any more. I was thinking that > another > > > way of distinguishing the clients making the topic creation requests is > > > using clientId. For example, the admin tool can set it to something > like > > > admin and the broker can treat that clientId specially. > > > > > > Also, there is a related discussion in KAFKA-2020. Currently, we do the > > > following in TopicMetadataResponse: > > > > > > 1. If leader is not available, we set the partition level error code to > > > LeaderNotAvailable. > > > 2. If a non-leader replica is not available, we take that replica out > of > > > the assigned replica list and isr in the response. As an indication for > > > doing that, we set the partition level error code to > ReplicaNotAvailable. > > > > > > This has a few problems. First, ReplicaNotAvailable probably shouldn't > be > > > an error, at least for the normal producer/consumer clients that just > want > > > to find out the leader. Second, it can happen that both the leader and > > > another replica are not available at the same time. There is no error > code > > > to indicate both. Third, even if a replica is not available, it's still > > > useful to return its replica id since some clients (e.g. admin tool) > may > > > still make use of it. > > > > > > One way to address this issue is to always return the replica id for > > > leader, assigned replicas, and isr regardless of whether the > corresponding > > > broker is live or not. Since we also return the list of live brokers, > the > > > client can figure out whether a leader or a replica is live or not and > act > > > accordingly. This way, we don't need to set the partition level error > code > > > when the leader or a replica is not available. This doesn't change the > wire > > > protocol, but does change the semantics. Since we are evolving the > protocol > > > of TopicMetadataRequest here, we can potentially piggyback the change. > > > > > > 102.1 For those types of errors due to invalid input, shouldn't we just > > > guard it at parameter validation time and throw > InvalidArgumentException > > > without even sending the request to the broker? > > > > > > Thanks, > > > > > > Jun > > > > > > > > > On Mon, Mar 16, 2015 at 10:37 AM, Andrii Biletskyi < > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > Jun, > > > > > > > > Answering your questions: > > > > > > > > 101. If I understand you correctly, you are saying future producer > > > versions > > > > (which > > > > will be ported to TMR_V1) won't be able to automatically create > topic (if > > > > we > > > > unconditionally remove topic creation from there). But we need to > this > > > > preserve logic. > > > > Ok, about your proposal: I'm not a big fan too, when it comes to > > > > differentiating > > > > clients directly in protocol schema. And also I'm not sure I > understand > > > at > > > > all why > > > > auto.create.topics.enable is a server side configuration. Can we > > > deprecate > > > > this setting > > > > in future versions, add this setting to producer and based on that > upon > > > > receiving > > > > UnknownTopic create topic explicitly by a separate producer call via > > > > adminClient? > > > > > > > > 102.1. Hm, yes. It's because we want to support batching and at the > same > > > > time we > > > > want to give descriptive error messages for clients. Since > AdminClient > > > > holds the context > > > > to construct such messages (e.g. AdminClient layer can know that > > > > InvalidArgumentsCode > > > > means two cases: either invalid number - e.g. -1; or > replication-factor > > > was > > > > provided while > > > > partitions argument wasn't) - I wrapped responses in Exceptions. But > I'm > > > > open to any > > > > other ideas, this was just initial version. > > > > 102.2. Yes, I agree. I'll change that to probably some other dto. > > > > > > > > Thanks, > > > > Andrii Biletskyi > > > > > > > > On Fri, Mar 13, 2015 at 7:16 PM, Jun Rao <j...@confluent.io> wrote: > > > > > > > > > Andrii, > > > > > > > > > > 101. That's what I was thinking too, but it may not be that > simple. In > > > > > TopicMetadataRequest_V1, > > > > > we can let it not trigger auto topic creation. Then, in the > producer > > > > side, > > > > > if it gets an UnknownTopicException, it can explicitly issue a > > > > > createTopicRequest for auto topic creation. On the consumer side, > it > > > will > > > > > never issue createTopicRequest. This works when auto topic > creation is > > > > > enabled on the broker side. However, I am not sure how things will > work > > > > > when auto topic creation is disabled on the broker side. In this > case, > > > we > > > > > want to have a way to manually create a topic, potentially through > > > admin > > > > > commands. However, then we need a way to distinguish > createTopicRequest > > > > > issued from the producer clients and the admin tools. May be we can > > > add a > > > > > new field in createTopicRequest and set it differently in the > producer > > > > > client and the admin client. However, I am not sure if that's the > best > > > > > approach. > > > > > > > > > > 2. Yes, refactoring existing requests is a non-trivial amount of > work. > > > I > > > > > posted some comments in KAFKA-1927. We will probably have to fix > > > > KAFKA-1927 > > > > > first, before adding the new logic in KAFKA-1694. Otherwise, the > > > changes > > > > > will be too big. > > > > > > > > > > 102. About the AdminClient: > > > > > 102.1. It's a bit weird that we return exception in the api. It > seems > > > > that > > > > > we should either return error code or throw an exception when > getting > > > the > > > > > response state. > > > > > 102.2. We probably shouldn't explicitly use the request object in > the > > > > api. > > > > > Not every request evolution requires an api change. > > > > > > > > > > Thanks, > > > > > > > > > > Jun > > > > > > > > > > > > > > > On Fri, Mar 13, 2015 at 4:08 AM, Andrii Biletskyi < > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > Jun, > > > > > > > > > > > > Thanks for you comments. Answers inline: > > > > > > > > > > > > 100. There are a few fields such as ReplicaAssignment, > > > > > > > ReassignPartitionRequest, > > > > > > > and PartitionsSerialized that are represented as a string, but > > > > contain > > > > > > > composite structures in json. Could we flatten them out > directly in > > > > the > > > > > > > protocol definition as arrays/records? > > > > > > > > > > > > > > > > > > Yes, now with Admin Client this looks a bit weird. My initial > > > > motivation > > > > > > was: > > > > > > ReassignPartitionCommand accepts input in json, we want to remain > > > > tools' > > > > > > interfaces unchanged, where possible. > > > > > > If we port it to deserialized format, in CLI (/tools project) we > will > > > > > have > > > > > > to add some > > > > > > json library since /tools is written in java and we'll need to > > > > > deserialize > > > > > > json file > > > > > > provided by a user. Can we quickly agree on what this library > should > > > be > > > > > > (Jackson, GSON, whatever)? > > > > > > > > > > > > 101. Does TopicMetadataRequest v1 still trigger auto topic > creation? > > > > This > > > > > > > will be a bit weird now that we have a separate topic creation > api. > > > > > Have > > > > > > > you thought about how the new createTopicRequest and > > > > > TopicMetadataRequest > > > > > > > v1 will be used in the producer/consumer client, in addition to > > > admin > > > > > > > tools? For example, ideally, we don't want TopicMetadataRequest > > > from > > > > > the > > > > > > > consumer to trigger auto topic creation. > > > > > > > > > > > > > > > > > > I agree, this strange logic should be fixed. I'm not confident in > > > this > > > > > > Kafka part so > > > > > > correct me if I'm wrong, but it doesn't look like a hard thing to > > > do, I > > > > > > think we can > > > > > > leverage AdminClient for that in Producer and unconditionally > remove > > > > > topic > > > > > > creation from the TopicMetadataRequest_V1. > > > > > > > > > > > > 2. I think Jay meant getting rid of scala classes > > > > > > > like HeartbeatRequestAndHeader and HeartbeatResponseAndHeader. > We > > > did > > > > > > that > > > > > > > as a stop-gap thing when adding the new requests for the > consumers. > > > > > > > However, the long term plan is to get rid of all those and just > > > reuse > > > > > the > > > > > > > java request/response in the client. Since this KIP proposes to > > > add a > > > > > > > significant number of new requests, perhaps we should bite the > > > bullet > > > > > to > > > > > > > clean up the existing scala requests first before adding new > ones? > > > > > > > > > > > > > > > > > > > Yes, looks like I misunderstood the point of ...RequestAndHeader. > > > > Okay, I > > > > > > will > > > > > > rework that. The only thing is that I don't see any example how > it > > > was > > > > > done > > > > > > for at > > > > > > least one existing protocol message. Thus, as I understand, I > have to > > > > > think > > > > > > how we > > > > > > are going to do it. > > > > > > Re porting all existing RQ/RP in this patch. Sounds reasonable, > but > > > if > > > > > it's > > > > > > an *obligatory* > > > > > > requirement to have Admin KIP done, I'm afraid this can be a > serious > > > > > > blocker for us. > > > > > > There are 13 protocol messages and all that would require not > only > > > unit > > > > > > tests but quite > > > > > > intensive manual testing, no? I'm afraid I'm not the right guy to > > > cover > > > > > > pretty much all > > > > > > Kafka core internals :). Let me know your thoughts on this item. > Btw > > > > > there > > > > > > is a ticket to > > > > > > follow-up this issue ( > > > https://issues.apache.org/jira/browse/KAFKA-2006 > > > > ). > > > > > > > > > > > > Thanks, > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > > > On Fri, Mar 13, 2015 at 6:40 AM, Jun Rao <j...@confluent.io> > wrote: > > > > > > > > > > > > > Andrii, > > > > > > > > > > > > > > > > > > > > > A few more comments. > > > > > > > > > > > > > > 100. There are a few fields such as ReplicaAssignment, > > > > > > > ReassignPartitionRequest, > > > > > > > and PartitionsSerialized that are represented as a string, but > > > > contain > > > > > > > composite structures in json. Could we flatten them out > directly in > > > > the > > > > > > > protocol definition as arrays/records? > > > > > > > > > > > > > > 101. Does TopicMetadataRequest v1 still trigger auto topic > > > creation? > > > > > This > > > > > > > will be a bit weird now that we have a separate topic creation > api. > > > > > Have > > > > > > > you thought about how the new createTopicRequest and > > > > > TopicMetadataRequest > > > > > > > v1 will be used in the producer/consumer client, in addition to > > > admin > > > > > > > tools? For example, ideally, we don't want TopicMetadataRequest > > > from > > > > > the > > > > > > > consumer to trigger auto topic creation. > > > > > > > > > > > > > > 2. I think Jay meant getting rid of scala classes > > > > > > > like HeartbeatRequestAndHeader and HeartbeatResponseAndHeader. > We > > > did > > > > > > that > > > > > > > as a stop-gap thing when adding the new requests for the > consumers. > > > > > > > However, the long term plan is to get rid of all those and just > > > reuse > > > > > the > > > > > > > java request/response in the client. Since this KIP proposes to > > > add a > > > > > > > significant number of new requests, perhaps we should bite the > > > bullet > > > > > to > > > > > > > clean up the existing scala requests first before adding new > ones? > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 12, 2015 at 3:37 PM, Andrii Biletskyi < > > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > As said above - I list again all comments from this thread > so we > > > > > > > > can see what's left and finalize all pending issues. > > > > > > > > > > > > > > > > Comments from Jay: > > > > > > > > 1. This is much needed functionality, but there are a lot of > the > > > so > > > > > > let's > > > > > > > > really think these protocols through. We really want to end > up > > > > with a > > > > > > set > > > > > > > > of well thought-out, orthoganol apis. For this reason I > think it > > > is > > > > > > > really > > > > > > > > important to think through the end state even if that > includes > > > APIs > > > > > we > > > > > > > > won't implement in the first phase. > > > > > > > > > > > > > > > > A: Definitely behind this. Would appreciate if there are > concrete > > > > > > > comments > > > > > > > > how this can be improved. > > > > > > > > > > > > > > > > 2. Let's please please please wait until we have switched the > > > > server > > > > > > over > > > > > > > > to the new java protocol definitions. If we add upteen more > ad > > > hoc > > > > > > scala > > > > > > > > objects that is just generating more work for the conversion > we > > > > know > > > > > we > > > > > > > > have to do. > > > > > > > > > > > > > > > > A: Fixed in the latest patch - removed scala protocol > classes. > > > > > > > > > > > > > > > > 3. This proposal introduces a new type of optional parameter. > > > This > > > > is > > > > > > > > inconsistent with everything else in the protocol where we > use -1 > > > > or > > > > > > some > > > > > > > > other marker value. You could argue either way but let's > stick > > > with > > > > > > that > > > > > > > > for consistency. For clients that implemented the protocol > in a > > > > > better > > > > > > > way > > > > > > > > than our scala code these basic primitives are hard to > change. > > > > > > > > > > > > > > > > A: Fixed in the latest patch - removed MaybeOf type and > changed > > > > > > protocol > > > > > > > > accordingly. > > > > > > > > > > > > > > > > 4. ClusterMetadata: This seems to duplicate > TopicMetadataRequest > > > > > which > > > > > > > has > > > > > > > > brokers, topics, and partitions. I think we should rename > that > > > > > request > > > > > > > > ClusterMetadataRequest (or just MetadataRequest) and include > the > > > id > > > > > of > > > > > > > the > > > > > > > > controller. Or are there other things we could add here? > > > > > > > > > > > > > > > > A: I agree. Updated the KIP. Let's extends TopicMetadata to > > > > version 2 > > > > > > and > > > > > > > > include controller. > > > > > > > > > > > > > > > > 5. We have a tendency to try to make a lot of requests that > can > > > > only > > > > > go > > > > > > > to > > > > > > > > particular nodes. This adds a lot of burden for client > > > > > implementations > > > > > > > (it > > > > > > > > sounds easy but each discovery can fail in many parts so it > ends > > > up > > > > > > > being a > > > > > > > > full state machine to do right). I think we should consider > > > making > > > > > > admin > > > > > > > > commands and ideally as many of the other apis as possible > > > > available > > > > > on > > > > > > > all > > > > > > > > brokers and just redirect to the controller on the broker > side. > > > > > Perhaps > > > > > > > > there would be a general way to encapsulate this re-routing > > > > behavior. > > > > > > > > > > > > > > > > A: It's a very interesting idea, but seems there are some > > > concerns > > > > > > about > > > > > > > > this > > > > > > > > feature (like performance considerations, how this will > > > complicate > > > > > > server > > > > > > > > etc). > > > > > > > > I believe this shouldn't be a blocker. If this feature is > > > > implemented > > > > > > at > > > > > > > > some > > > > > > > > point it won't affect Admin changes - at least no changes to > > > public > > > > > API > > > > > > > > will be required. > > > > > > > > > > > > > > > > 6. We should probably normalize the key value pairs used for > > > > configs > > > > > > > rather > > > > > > > > than embedding a new formatting. So two strings rather than > one > > > > with > > > > > an > > > > > > > > internal equals sign. > > > > > > > > > > > > > > > > A: Fixed in the latest patch - normalized configs and changed > > > > > protocol > > > > > > > > accordingly. > > > > > > > > > > > > > > > > 7. Is the postcondition of these APIs that the command has > begun > > > or > > > > > > that > > > > > > > > the command has been completed? It is a lot more usable if > the > > > > > command > > > > > > > has > > > > > > > > been completed so you know that if you create a topic and > then > > > > > publish > > > > > > to > > > > > > > > it you won't get an exception about there being no such > topic. > > > > > > > > > > > > > > > > A: For long running requests (like reassign partitions) - the > > > post > > > > > > > > condition is > > > > > > > > command has begun - so we don't block the client. In case of > your > > > > > > > example - > > > > > > > > topic commands, this will be refactored and topic commands > will > > > be > > > > > > > executed > > > > > > > > immediately, since the Controller will serve Admin requests > > > > > > > > (follow-up ticket KAFKA-1777). > > > > > > > > > > > > > > > > 8. Describe topic and list topics duplicate a lot of stuff > in the > > > > > > > metadata > > > > > > > > request. Is there a reason to give back topics marked for > > > > deletion? I > > > > > > > feel > > > > > > > > like if we just make the post-condition of the delete > command be > > > > that > > > > > > the > > > > > > > > topic is deleted that will get rid of the need for this > right? > > > And > > > > it > > > > > > > will > > > > > > > > be much more intuitive. > > > > > > > > > > > > > > > > A: Fixed in the latest patch - removed topics marked for > deletion > > > > in > > > > > > > > ListTopicsRequest. > > > > > > > > > > > > > > > > 9. Should we consider batching these requests? We have > generally > > > > > tried > > > > > > to > > > > > > > > allow multiple operations to be batched. My suspicion is that > > > > without > > > > > > > this > > > > > > > > we will get a lot of code that does something like > > > > > > > > for(topic: adminClient.listTopics()) > > > > > > > > adminClient.describeTopic(topic) > > > > > > > > this code will work great when you test on 5 topics but not > do as > > > > > well > > > > > > if > > > > > > > > you have 50k. > > > > > > > > > > > > > > > > A: Updated the KIP - please check "Topic Admin Schema" > section. > > > > > > > > > > > > > > > > 10. I think we should also discuss how we want to expose a > > > > > programmatic > > > > > > > JVM > > > > > > > > client api for these operations. Currently people rely on > > > > AdminUtils > > > > > > > which > > > > > > > > is totally sketchy. I think we probably need another client > under > > > > > > > clients/ > > > > > > > > that exposes administrative functionality. We will need this > just > > > > to > > > > > > > > properly test the new apis, I suspect. We should figure out > that > > > > API. > > > > > > > > > > > > > > > > A: Updated the KIP - please check "Admin Client" section > with an > > > > > > initial > > > > > > > > API proposal. > > > > > > > > > > > > > > > > 11. The other information that would be really useful to get > > > would > > > > be > > > > > > > > information about partitions--how much data is in the > partition, > > > > what > > > > > > are > > > > > > > > the segment offsets, what is the log-end offset (i.e. last > > > offset), > > > > > > what > > > > > > > is > > > > > > > > the compaction point, etc. I think that done right this > would be > > > > the > > > > > > > > successor to the very awkward OffsetRequest we have today. > > > > > > > > > > > > > > > > A: I removed ConsumerGroupOffsetsRequest in the latest > patch. I > > > > > believe > > > > > > > > this should > > > > > > > > be resolved in a separate KIP / jira ticket. > > > > > > > > > > > > > > > > 12. Generally we can do good error handling without needing > > > custom > > > > > > > > server-side > > > > > > > > messages. I.e. generally the client has the context to know > that > > > if > > > > > it > > > > > > > got > > > > > > > > an error that the topic doesn't exist to say "Topic X doesn't > > > > exist" > > > > > > > rather > > > > > > > > than "error code 14" (or whatever). Maybe there are specific > > > cases > > > > > > where > > > > > > > > this is hard? If we want to add server-side error messages we > > > > really > > > > > do > > > > > > > > need to do this in a consistent way across the protocol. > > > > > > > > > > > > > > > > A: Updated the KIP - please check "Protocol Errors" section. > I > > > > added > > > > > > the > > > > > > > > comprehensive, fine-grained list of error codes. > > > > > > > > > > > > > > > > Comments from Guozhang: > > > > > > > > 13. Describe topic request: it would be great to go beyond > just > > > > > > batching > > > > > > > on > > > > > > > > topic name regex for this request. For example, a very > common use > > > > > case > > > > > > of > > > > > > > > the topic command is to list all topics whose config A's > value is > > > > B. > > > > > > With > > > > > > > > topic name regex then we have to first retrieve __all__ > topics's > > > > > > > > description info and then filter at the client end, which > will > > > be a > > > > > > huge > > > > > > > > burden on ZK. > > > > > > > > AND > > > > > > > > 14. Config K-Vs in create topic: this is related to the > previous > > > > > point; > > > > > > > > maybe we can add another metadata K-V or just a metadata > string > > > > along > > > > > > > side > > > > > > > > with config K-V in create topic like we did for offset commit > > > > > request. > > > > > > > This > > > > > > > > field can be quite useful in storing information like > "owner" of > > > > the > > > > > > > topic > > > > > > > > who issue the create command, etc, which is quite important > for a > > > > > > > > multi-tenant setting. Then in the describe topic request we > can > > > > also > > > > > > > batch > > > > > > > > on regex of the metadata field. > > > > > > > > > > > > > > > > A: As discussed it is very interesting but can be implemented > > > later > > > > > > after > > > > > > > > we have some basic functionality there. > > > > > > > > > > > > > > > > 15. Today all the admin operations are async in the sense > that > > > > > command > > > > > > > will > > > > > > > > return once it is written in ZK, and that is why we need > extra > > > > > > > verification > > > > > > > > like testUtil.waitForTopicCreated() / verify partition > > > reassignment > > > > > > > > request, etc. With admin requests we could add a flag to > enable / > > > > > > disable > > > > > > > > synchronous requests; when it is turned on, the response > will not > > > > > > return > > > > > > > > until the request has been completed. And for async requests > we > > > can > > > > > > add a > > > > > > > > "token" field in the response, and then only need a general > > > "admin > > > > > > > > verification request" with the given token to check if the > async > > > > > > request > > > > > > > > has been completed. > > > > > > > > > > > > > > > > A: I see your point. My idea was to provide specific > > > > Verify...Request > > > > > > per > > > > > > > > each > > > > > > > > long running request, where needed. We can do it the way you > > > > suggest. > > > > > > The > > > > > > > > only > > > > > > > > concern is that introducing a token we again will make schema > > > > > > "dynamic". > > > > > > > We > > > > > > > > wanted > > > > > > > > to do similar thing introducing single AdminRequest for all > topic > > > > > > > commands > > > > > > > > but rejected > > > > > > > > this idea because we wanted to have schema defined. So this > is > > > > more a > > > > > > > > choice between: > > > > > > > > a) have fixed schema but introduce each time new > Verify...Request > > > > for > > > > > > > > long-running requests > > > > > > > > b) use one request for verification but generalize it with > token > > > > > > > > I'm fine with whatever decision community come to. Just let > me > > > know > > > > > > your > > > > > > > > thoughts. > > > > > > > > > > > > > > > > Comment from Gwen: > > > > > > > > 16. Specifically for ownership, I think the plan is to add > ACL > > > (it > > > > > > sounds > > > > > > > > like you are describing ACL) via an external system (Argus, > > > > Sentry). > > > > > > > > I remember KIP-11 described this, but I can't find the KIP > any > > > > > longer. > > > > > > > > > > > > > > > > A: Okay, no problem. Not sure though how we are going to > handle > > > it. > > > > > > Wait > > > > > > > > which KIP > > > > > > > > will be committed first and include changes to TopicMetadata > from > > > > the > > > > > > > later > > > > > > > > one? > > > > > > > > Anyway, I added this note to "Open Questions" section so we > don't > > > > > miss > > > > > > > this > > > > > > > > piece. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > On Fri, Mar 13, 2015 at 12:34 AM, Andrii Biletskyi < > > > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > Today I uploaded the patch that covers some of the > discussed > > > and > > > > > > agreed > > > > > > > > > items: > > > > > > > > > - removed MaybeOf optional type > > > > > > > > > - switched to java protocol definitions > > > > > > > > > - simplified messages (normalized configs, removed topic > marked > > > > for > > > > > > > > > deletion) > > > > > > > > > > > > > > > > > > I also updated the KIP-4 with respective changes and wrote > down > > > > my > > > > > > > > > proposal for > > > > > > > > > pending items: > > > > > > > > > - Batch Admin Operations -> updated Wire Protocol schema > > > proposal > > > > > > > > > - Remove ClusterMetadata -> changed to extend > > > > TopicMetadataRequest > > > > > > > > > - Admin Client -> updated my initial proposal to reflect > > > batching > > > > > > > > > - Error codes -> proposed fine-grained error code instead > of > > > > > > > > > AdminRequestFailed > > > > > > > > > > > > > > > > > > I will also send a separate email to cover all comments > from > > > this > > > > > > > thread. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 12, 2015 at 9:26 PM, Gwen Shapira < > > > > > gshap...@cloudera.com > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > >> Found KIP-11 ( > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface > > > > > > > > >> ) > > > > > > > > >> It actually specifies changes to the Metadata protocol, so > > > > making > > > > > > sure > > > > > > > > >> both KIPs are consistent in this regard will be good. > > > > > > > > >> > > > > > > > > >> On Thu, Mar 12, 2015 at 12:21 PM, Gwen Shapira < > > > > > > gshap...@cloudera.com > > > > > > > > > > > > > > > > >> wrote: > > > > > > > > >> > Specifically for ownership, I think the plan is to add > ACL > > > (it > > > > > > > sounds > > > > > > > > >> > like you are describing ACL) via an external system > (Argus, > > > > > > Sentry). > > > > > > > > >> > I remember KIP-11 described this, but I can't find the > KIP > > > any > > > > > > > longer. > > > > > > > > >> > > > > > > > > > >> > Regardless, I think KIP-4 focuses on getting information > > > that > > > > > > > already > > > > > > > > >> > exists from Kafka brokers, not on adding information > that > > > > > perhaps > > > > > > > > >> > should exist but doesn't yet? > > > > > > > > >> > > > > > > > > > >> > Gwen > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > On Thu, Mar 12, 2015 at 6:37 AM, Guozhang Wang < > > > > > > wangg...@gmail.com> > > > > > > > > >> wrote: > > > > > > > > >> >> Folks, > > > > > > > > >> >> > > > > > > > > >> >> Just want to elaborate a bit more on the create-topic > > > > metadata > > > > > > and > > > > > > > > >> batching > > > > > > > > >> >> describe-topic based on config / metadata in my > previous > > > > email > > > > > as > > > > > > > we > > > > > > > > >> work > > > > > > > > >> >> on KAFKA-1694. The main motivation is to have some > sort of > > > > > topic > > > > > > > > >> management > > > > > > > > >> >> mechanisms, which I think is quite important in a > > > > multi-tenant > > > > > / > > > > > > > > cloud > > > > > > > > >> >> architecture: today anyone can create topics in a > shared > > > > Kafka > > > > > > > > >> cluster, but > > > > > > > > >> >> there is no concept or "ownership" of topics that are > > > created > > > > > by > > > > > > > > >> different > > > > > > > > >> >> users. For example, at LinkedIn we basically > distinguish > > > > topic > > > > > > > owners > > > > > > > > >> via > > > > > > > > >> >> some casual topic name prefix, which is a bit awkward > and > > > > does > > > > > > not > > > > > > > > fly > > > > > > > > >> as > > > > > > > > >> >> we scale our customers. It would be great to use > > > > > describe-topics > > > > > > > such > > > > > > > > >> as: > > > > > > > > >> >> > > > > > > > > >> >> Describe all topics that is created by me. > > > > > > > > >> >> > > > > > > > > >> >> Describe all topics whose retention time is overriden > to X. > > > > > > > > >> >> > > > > > > > > >> >> Describe all topics whose writable group include user Y > > > (this > > > > > is > > > > > > > > >> related to > > > > > > > > >> >> authorization), etc.. > > > > > > > > >> >> > > > > > > > > >> >> One possible way to achieve this is to add a metadata > file > > > in > > > > > the > > > > > > > > >> >> create-topic request, whose value will also be written > ZK > > > as > > > > we > > > > > > > > create > > > > > > > > >> the > > > > > > > > >> >> topic; then describe-topics can choose to batch topics > > > based > > > > on > > > > > > 1) > > > > > > > > name > > > > > > > > >> >> regex, 2) config K-V matching, 3) metadata regex, etc. > > > > > > > > >> >> > > > > > > > > >> >> Thoughts? > > > > > > > > >> >> > > > > > > > > >> >> Guozhang > > > > > > > > >> >> > > > > > > > > >> >> On Thu, Mar 5, 2015 at 4:37 PM, Guozhang Wang < > > > > > > wangg...@gmail.com> > > > > > > > > >> wrote: > > > > > > > > >> >> > > > > > > > > >> >>> Thanks for the updated wiki. A few comments below: > > > > > > > > >> >>> > > > > > > > > >> >>> 1. Error description in response: I think if some > > > errorCode > > > > > > could > > > > > > > > >> indicate > > > > > > > > >> >>> several different error cases then we should really > change > > > > it > > > > > to > > > > > > > > >> multiple > > > > > > > > >> >>> codes. In general the errorCode itself would be > precise > > > and > > > > > > > > >> sufficient for > > > > > > > > >> >>> describing the server side errors. > > > > > > > > >> >>> > > > > > > > > >> >>> 2. Describe topic request: it would be great to go > beyond > > > > just > > > > > > > > >> batching on > > > > > > > > >> >>> topic name regex for this request. For example, a very > > > > common > > > > > > use > > > > > > > > >> case of > > > > > > > > >> >>> the topic command is to list all topics whose config > A's > > > > value > > > > > > is > > > > > > > B. > > > > > > > > >> With > > > > > > > > >> >>> topic name regex then we have to first retrieve > __all__ > > > > > topics's > > > > > > > > >> >>> description info and then filter at the client end, > which > > > > will > > > > > > be > > > > > > > a > > > > > > > > >> huge > > > > > > > > >> >>> burden on ZK. > > > > > > > > >> >>> > > > > > > > > >> >>> 3. Config K-Vs in create topic: this is related to the > > > > > previous > > > > > > > > point; > > > > > > > > >> >>> maybe we can add another metadata K-V or just a > metadata > > > > > string > > > > > > > > along > > > > > > > > >> side > > > > > > > > >> >>> with config K-V in create topic like we did for offset > > > > commit > > > > > > > > >> request. This > > > > > > > > >> >>> field can be quite useful in storing information like > > > > "owner" > > > > > of > > > > > > > the > > > > > > > > >> topic > > > > > > > > >> >>> who issue the create command, etc, which is quite > > > important > > > > > for > > > > > > a > > > > > > > > >> >>> multi-tenant setting. Then in the describe topic > request > > > we > > > > > can > > > > > > > also > > > > > > > > >> batch > > > > > > > > >> >>> on regex of the metadata field. > > > > > > > > >> >>> > > > > > > > > >> >>> 4. Today all the admin operations are async in the > sense > > > > that > > > > > > > > command > > > > > > > > >> will > > > > > > > > >> >>> return once it is written in ZK, and that is why we > need > > > > extra > > > > > > > > >> verification > > > > > > > > >> >>> like testUtil.waitForTopicCreated() / verify partition > > > > > > > reassignment > > > > > > > > >> >>> request, etc. With admin requests we could add a flag > to > > > > > enable > > > > > > / > > > > > > > > >> disable > > > > > > > > >> >>> synchronous requests; when it is turned on, the > response > > > > will > > > > > > not > > > > > > > > >> return > > > > > > > > >> >>> until the request has been completed. And for async > > > requests > > > > > we > > > > > > > can > > > > > > > > >> add a > > > > > > > > >> >>> "token" field in the response, and then only need a > > > general > > > > > > "admin > > > > > > > > >> >>> verification request" with the given token to check > if the > > > > > async > > > > > > > > >> request > > > > > > > > >> >>> has been completed. > > > > > > > > >> >>> > > > > > > > > >> >>> 5. +1 for extending Metadata request to include > > > controller / > > > > > > > > >> coordinator > > > > > > > > >> >>> information, and then we can remove the > ConsumerMetadata / > > > > > > > > >> ClusterMetadata > > > > > > > > >> >>> requests. > > > > > > > > >> >>> > > > > > > > > >> >>> Guozhang > > > > > > > > >> >>> > > > > > > > > >> >>> On Tue, Mar 3, 2015 at 10:23 AM, Joel Koshy < > > > > > > jjkosh...@gmail.com> > > > > > > > > >> wrote: > > > > > > > > >> >>> > > > > > > > > >> >>>> Thanks for sending that out Joe - I don't think I > will be > > > > > able > > > > > > to > > > > > > > > >> make > > > > > > > > >> >>>> it today, so if notes can be sent out afterward that > > > would > > > > be > > > > > > > > great. > > > > > > > > >> >>>> > > > > > > > > >> >>>> On Mon, Mar 02, 2015 at 09:16:13AM -0800, Gwen > Shapira > > > > wrote: > > > > > > > > >> >>>> > Thanks for sending this out Joe. Looking forward to > > > > > chatting > > > > > > > with > > > > > > > > >> >>>> everyone :) > > > > > > > > >> >>>> > > > > > > > > > >> >>>> > On Mon, Mar 2, 2015 at 6:46 AM, Joe Stein < > > > > > > > joe.st...@stealth.ly> > > > > > > > > >> wrote: > > > > > > > > >> >>>> > > Hey, I just sent out a google hangout invite to > all > > > > pmc, > > > > > > > > >> committers > > > > > > > > >> >>>> and > > > > > > > > >> >>>> > > everyone I found working on a KIP. If I missed > anyone > > > > in > > > > > > the > > > > > > > > >> invite > > > > > > > > >> >>>> please > > > > > > > > >> >>>> > > let me know and can update it, np. > > > > > > > > >> >>>> > > > > > > > > > > >> >>>> > > We should do this every Tuesday @ 2pm Eastern > Time. > > > > Maybe > > > > > > we > > > > > > > > can > > > > > > > > >> get > > > > > > > > >> >>>> INFRA > > > > > > > > >> >>>> > > help to make a google account so we can manage > > > better? > > > > > > > > >> >>>> > > > > > > > > > > >> >>>> > > To discuss > > > > > > > > >> >>>> > > > > > > > > > > >> >>>> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > > > > > > >> >>>> > > in progress and related JIRA that are > interdependent > > > > and > > > > > > > common > > > > > > > > >> work. > > > > > > > > >> >>>> > > > > > > > > > > >> >>>> > > ~ Joe Stein > > > > > > > > >> >>>> > > > > > > > > > > >> >>>> > > On Tue, Feb 24, 2015 at 2:59 PM, Jay Kreps < > > > > > > > > jay.kr...@gmail.com> > > > > > > > > >> >>>> wrote: > > > > > > > > >> >>>> > > > > > > > > > > >> >>>> > >> Let's stay on Google hangouts that will also > record > > > > and > > > > > > make > > > > > > > > the > > > > > > > > >> >>>> sessions > > > > > > > > >> >>>> > >> available on youtube. > > > > > > > > >> >>>> > >> > > > > > > > > >> >>>> > >> -Jay > > > > > > > > >> >>>> > >> > > > > > > > > >> >>>> > >> On Tue, Feb 24, 2015 at 11:49 AM, Jeff Holoman < > > > > > > > > >> >>>> jholo...@cloudera.com> > > > > > > > > >> >>>> > >> wrote: > > > > > > > > >> >>>> > >> > > > > > > > > >> >>>> > >> > Jay / Joe > > > > > > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > We're happy to send out a Webex for this > purpose. > > > We > > > > > > could > > > > > > > > >> record > > > > > > > > >> >>>> the > > > > > > > > >> >>>> > >> > sessions if there is interest and publish them > > > out. > > > > > > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > Thanks > > > > > > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > Jeff > > > > > > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > On Tue, Feb 24, 2015 at 11:28 AM, Jay Kreps < > > > > > > > > >> jay.kr...@gmail.com> > > > > > > > > >> >>>> wrote: > > > > > > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > > Let's try to get the technical hang-ups > sorted > > > > out, > > > > > > > > though. > > > > > > > > >> I > > > > > > > > >> >>>> really > > > > > > > > >> >>>> > >> > think > > > > > > > > >> >>>> > >> > > there is some benefit to live discussion vs > > > > > writing. I > > > > > > > am > > > > > > > > >> >>>> hopeful that > > > > > > > > >> >>>> > >> if > > > > > > > > >> >>>> > >> > > we post instructions and give ourselves a > few > > > > > attempts > > > > > > > we > > > > > > > > >> can > > > > > > > > >> >>>> get it > > > > > > > > >> >>>> > >> > > working. > > > > > > > > >> >>>> > >> > > > > > > > > > > >> >>>> > >> > > Tuesday at that time would work for me...any > > > > > > objections? > > > > > > > > >> >>>> > >> > > > > > > > > > > >> >>>> > >> > > -Jay > > > > > > > > >> >>>> > >> > > > > > > > > > > >> >>>> > >> > > On Tue, Feb 24, 2015 at 8:18 AM, Joe Stein < > > > > > > > > >> joe.st...@stealth.ly > > > > > > > > >> >>>> > > > > > > > > > >> >>>> > >> wrote: > > > > > > > > >> >>>> > >> > > > > > > > > > > >> >>>> > >> > > > Weekly would be great maybe like every > > > Tuesday ~ > > > > > 1pm > > > > > > > ET > > > > > > > > / > > > > > > > > >> 10am > > > > > > > > >> >>>> PT > > > > > > > > >> >>>> > >> ???? > > > > > > > > >> >>>> > >> > > > > > > > > > > > >> >>>> > >> > > > I don't mind google hangout but there is > > > always > > > > > some > > > > > > > > >> issue or > > > > > > > > >> >>>> > >> whatever > > > > > > > > >> >>>> > >> > so > > > > > > > > >> >>>> > >> > > > we know the apache irc channel works. We > can > > > > start > > > > > > > there > > > > > > > > >> and > > > > > > > > >> >>>> see how > > > > > > > > >> >>>> > >> it > > > > > > > > >> >>>> > >> > > > goes? We can pull transcripts too and > > > associate > > > > to > > > > > > > > >> tickets if > > > > > > > > >> >>>> need be > > > > > > > > >> >>>> > >> > > makes > > > > > > > > >> >>>> > >> > > > it helpful for things. > > > > > > > > >> >>>> > >> > > > > > > > > > > > >> >>>> > >> > > > ~ Joestein > > > > > > > > >> >>>> > >> > > > > > > > > > > > >> >>>> > >> > > > On Tue, Feb 24, 2015 at 11:10 AM, Jay > Kreps < > > > > > > > > >> >>>> jay.kr...@gmail.com> > > > > > > > > >> >>>> > >> > wrote: > > > > > > > > >> >>>> > >> > > > > > > > > > > > >> >>>> > >> > > > > We'd talked about doing a Google > Hangout to > > > > chat > > > > > > > about > > > > > > > > >> this. > > > > > > > > >> >>>> What > > > > > > > > >> >>>> > >> > about > > > > > > > > >> >>>> > >> > > > > generalizing that a little further...I > > > > actually > > > > > > > think > > > > > > > > it > > > > > > > > >> >>>> would be > > > > > > > > >> >>>> > >> > good > > > > > > > > >> >>>> > >> > > > for > > > > > > > > >> >>>> > >> > > > > everyone spending a reasonable chunk of > > > their > > > > > week > > > > > > > on > > > > > > > > >> Kafka > > > > > > > > >> >>>> stuff > > > > > > > > >> >>>> > >> to > > > > > > > > >> >>>> > >> > > > maybe > > > > > > > > >> >>>> > >> > > > > sync up once a week. I think we could > use > > > time > > > > > to > > > > > > > talk > > > > > > > > >> >>>> through > > > > > > > > >> >>>> > >> design > > > > > > > > >> >>>> > >> > > > > stuff, make sure we are on top of code > > > > reviews, > > > > > > talk > > > > > > > > >> through > > > > > > > > >> >>>> any > > > > > > > > >> >>>> > >> > tricky > > > > > > > > >> >>>> > >> > > > > issues, etc. > > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> >>>> > >> > > > > We can make it publicly available so > that > > > any > > > > > one > > > > > > > can > > > > > > > > >> follow > > > > > > > > >> >>>> along > > > > > > > > >> >>>> > >> > who > > > > > > > > >> >>>> > >> > > > > likes. > > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> >>>> > >> > > > > Any interest in doing this? If so I'll > try > > > to > > > > > set > > > > > > it > > > > > > > > up > > > > > > > > >> >>>> starting > > > > > > > > >> >>>> > >> next > > > > > > > > >> >>>> > >> > > > week. > > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> >>>> > >> > > > > -Jay > > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> >>>> > >> > > > > On Tue, Feb 24, 2015 at 3:57 AM, Andrii > > > > > Biletskyi > > > > > > < > > > > > > > > >> >>>> > >> > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > Hi all, > > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > I've updated KIP page, fixed / aligned > > > > > document > > > > > > > > >> structure. > > > > > > > > >> >>>> Also I > > > > > > > > >> >>>> > >> > > added > > > > > > > > >> >>>> > >> > > > > > some > > > > > > > > >> >>>> > >> > > > > > very initial proposal for AdminClient > so > > > we > > > > > have > > > > > > > > >> something > > > > > > > > >> >>>> to > > > > > > > > >> >>>> > >> start > > > > > > > > >> >>>> > >> > > > from > > > > > > > > >> >>>> > >> > > > > > while > > > > > > > > >> >>>> > >> > > > > > discussing the KIP. > > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > > > > > > > > >> >>>> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > Thanks, > > > > > > > > >> >>>> > >> > > > > > Andrii Biletskyi > > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > On Wed, Feb 18, 2015 at 9:01 PM, > Andrii > > > > > > Biletskyi > > > > > > > < > > > > > > > > >> >>>> > >> > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > > Jay, > > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> >>>> > >> > > > > > > Re error messages: you are right, in > > > most > > > > > > cases > > > > > > > > >> client > > > > > > > > >> >>>> will > > > > > > > > >> >>>> > >> have > > > > > > > > >> >>>> > >> > > > enough > > > > > > > > >> >>>> > >> > > > > > > context to show descriptive error > > > message. > > > > > My > > > > > > > > >> concern is > > > > > > > > >> >>>> that > > > > > > > > >> >>>> > >> we > > > > > > > > >> >>>> > >> > > will > > > > > > > > >> >>>> > >> > > > > > have > > > > > > > > >> >>>> > >> > > > > > > to > > > > > > > > >> >>>> > >> > > > > > > add lots of new error codes for each > > > > > possible > > > > > > > > >> error. Of > > > > > > > > >> >>>> course, > > > > > > > > >> >>>> > >> > we > > > > > > > > >> >>>> > >> > > > > could > > > > > > > > >> >>>> > >> > > > > > > reuse > > > > > > > > >> >>>> > >> > > > > > > some of existing like > > > > > > > UknownTopicOrPartitionCode, > > > > > > > > >> but we > > > > > > > > >> >>>> will > > > > > > > > >> >>>> > >> > also > > > > > > > > >> >>>> > >> > > > need > > > > > > > > >> >>>> > >> > > > > > to > > > > > > > > >> >>>> > >> > > > > > > add smth like: > TopicAlreadyExistsCode, > > > > > > > > >> >>>> TopicConfigInvalid (both > > > > > > > > >> >>>> > >> > for > > > > > > > > >> >>>> > >> > > > > topic > > > > > > > > >> >>>> > >> > > > > > > name and config, and probably user > would > > > > > like > > > > > > to > > > > > > > > >> know > > > > > > > > >> >>>> what > > > > > > > > >> >>>> > >> > exactly > > > > > > > > >> >>>> > >> > > > > > > is wrong in his config), > > > > > > > InvalidReplicaAssignment, > > > > > > > > >> >>>> > >> InternalError > > > > > > > > >> >>>> > >> > > > (e.g. > > > > > > > > >> >>>> > >> > > > > > > zookeeper failure) etc. > > > > > > > > >> >>>> > >> > > > > > > And this is only for TopicCommand, > we > > > will > > > > > > also > > > > > > > > >> need to > > > > > > > > >> >>>> add > > > > > > > > >> >>>> > >> > similar > > > > > > > > >> >>>> > >> > > > > stuff > > > > > > > > >> >>>> > >> > > > > > > for > > > > > > > > >> >>>> > >> > > > > > > ReassignPartitions, > PreferredReplica. So > > > > > we'll > > > > > > > end > > > > > > > > >> up > > > > > > > > >> >>>> with a > > > > > > > > >> >>>> > >> > large > > > > > > > > >> >>>> > >> > > > list > > > > > > > > >> >>>> > >> > > > > > of > > > > > > > > >> >>>> > >> > > > > > > error codes, used only in Admin > > > protocol. > > > > > > > > >> >>>> > >> > > > > > > Having said that, I agree my > proposal is > > > > not > > > > > > > > >> consistent > > > > > > > > >> >>>> with > > > > > > > > >> >>>> > >> > other > > > > > > > > >> >>>> > >> > > > > cases. > > > > > > > > >> >>>> > >> > > > > > > Maybe we can find better solution or > > > > > something > > > > > > > > >> >>>> in-between. > > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> >>>> > >> > > > > > > Re Hangout chat: I think it is a > great > > > > idea. > > > > > > > This > > > > > > > > >> way we > > > > > > > > >> >>>> can > > > > > > > > >> >>>> > >> move > > > > > > > > >> >>>> > >> > > on > > > > > > > > >> >>>> > >> > > > > > > faster. > > > > > > > > >> >>>> > >> > > > > > > Let's agree somehow on date/time so > > > people > > > > > can > > > > > > > > join. > > > > > > > > >> >>>> Will work > > > > > > > > >> >>>> > >> > for > > > > > > > > >> >>>> > >> > > me > > > > > > > > >> >>>> > >> > > > > > this > > > > > > > > >> >>>> > >> > > > > > > and > > > > > > > > >> >>>> > >> > > > > > > next week almost anytime if agreed > in > > > > > advance. > > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> >>>> > >> > > > > > > Thanks, > > > > > > > > >> >>>> > >> > > > > > > Andrii > > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> >>>> > >> > > > > > > On Wed, Feb 18, 2015 at 7:09 PM, Jay > > > > Kreps < > > > > > > > > >> >>>> > >> jay.kr...@gmail.com> > > > > > > > > >> >>>> > >> > > > > wrote: > > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> >>>> > >> > > > > > >> Hey Andrii, > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> >>>> > >> > > > > > >> Generally we can do good error > handling > > > > > > without > > > > > > > > >> needing > > > > > > > > >> >>>> custom > > > > > > > > >> >>>> > >> > > > > > server-side > > > > > > > > >> >>>> > >> > > > > > >> messages. I.e. generally the > client has > > > > the > > > > > > > > >> context to > > > > > > > > >> >>>> know > > > > > > > > >> >>>> > >> that > > > > > > > > >> >>>> > >> > > if > > > > > > > > >> >>>> > >> > > > it > > > > > > > > >> >>>> > >> > > > > > got > > > > > > > > >> >>>> > >> > > > > > >> an error that the topic doesn't > exist > > > to > > > > > say > > > > > > > > >> "Topic X > > > > > > > > >> >>>> doesn't > > > > > > > > >> >>>> > >> > > exist" > > > > > > > > >> >>>> > >> > > > > > >> rather > > > > > > > > >> >>>> > >> > > > > > >> than "error code 14" (or whatever). > > > Maybe > > > > > > there > > > > > > > > are > > > > > > > > >> >>>> specific > > > > > > > > >> >>>> > >> > cases > > > > > > > > >> >>>> > >> > > > > where > > > > > > > > >> >>>> > >> > > > > > >> this is hard? If we want to add > > > > server-side > > > > > > > error > > > > > > > > >> >>>> messages we > > > > > > > > >> >>>> > >> > > really > > > > > > > > >> >>>> > >> > > > > do > > > > > > > > >> >>>> > >> > > > > > >> need to do this in a consistent way > > > > across > > > > > > the > > > > > > > > >> protocol. > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> >>>> > >> > > > > > >> I still have a bunch of open > questions > > > > here > > > > > > > from > > > > > > > > my > > > > > > > > >> >>>> previous > > > > > > > > >> >>>> > >> > > list. I > > > > > > > > >> >>>> > >> > > > > > will > > > > > > > > >> >>>> > >> > > > > > >> be out for the next few days for > Strata > > > > > > though. > > > > > > > > >> Maybe > > > > > > > > >> >>>> we could > > > > > > > > >> >>>> > >> > do > > > > > > > > >> >>>> > >> > > a > > > > > > > > >> >>>> > >> > > > > > Google > > > > > > > > >> >>>> > >> > > > > > >> Hangout chat on any open issues > some > > > time > > > > > > > towards > > > > > > > > >> the > > > > > > > > >> >>>> end of > > > > > > > > >> >>>> > >> > next > > > > > > > > >> >>>> > >> > > > week > > > > > > > > >> >>>> > >> > > > > > for > > > > > > > > >> >>>> > >> > > > > > >> anyone interested in this ticket? I > > > have > > > > a > > > > > > > > feeling > > > > > > > > >> that > > > > > > > > >> >>>> might > > > > > > > > >> >>>> > >> > > > progress > > > > > > > > >> >>>> > >> > > > > > >> things a little faster than > email--I > > > > think > > > > > we > > > > > > > > >> could talk > > > > > > > > >> >>>> > >> through > > > > > > > > >> >>>> > >> > > > those > > > > > > > > >> >>>> > >> > > > > > >> issues I brought up fairly > quickly... > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> >>>> > >> > > > > > >> -Jay > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> >>>> > >> > > > > > >> On Wed, Feb 18, 2015 at 7:27 AM, > Andrii > > > > > > > > Biletskyi < > > > > > > > > >> >>>> > >> > > > > > >> andrii.bilets...@stealth.ly> > wrote: > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> >>>> > >> > > > > > >> > Hi all, > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > >> >>>> > >> > > > > > >> > I'm trying to address some of the > > > > issues > > > > > > > which > > > > > > > > >> were > > > > > > > > >> >>>> > >> mentioned > > > > > > > > >> >>>> > >> > > > > earlier > > > > > > > > >> >>>> > >> > > > > > >> about > > > > > > > > >> >>>> > >> > > > > > >> > Admin RQ/RP format. One of those > was > > > > > about > > > > > > > > >> batching > > > > > > > > >> >>>> > >> > operations. > > > > > > > > >> >>>> > >> > > > What > > > > > > > > >> >>>> > >> > > > > > if > > > > > > > > >> >>>> > >> > > > > > >> we > > > > > > > > >> >>>> > >> > > > > > >> > follow TopicCommand approach and > let > > > > > people > > > > > > > > >> specify > > > > > > > > >> >>>> > >> topic-name > > > > > > > > >> >>>> > >> > > by > > > > > > > > >> >>>> > >> > > > > > >> regexp - > > > > > > > > >> >>>> > >> > > > > > >> > would that cover most of the use > > > cases? > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > >> >>>> > >> > > > > > >> > Secondly, is what information > should > > > we > > > > > > > > generally > > > > > > > > >> >>>> provide in > > > > > > > > >> >>>> > >> > > Admin > > > > > > > > >> >>>> > >> > > > > > >> > responses. > > > > > > > > >> >>>> > >> > > > > > >> > I realize that Admin commands > don't > > > > imply > > > > > > > they > > > > > > > > >> will > > > > > > > > >> >>>> be used > > > > > > > > >> >>>> > >> > only > > > > > > > > >> >>>> > >> > > > in > > > > > > > > >> >>>> > >> > > > > > CLI > > > > > > > > >> >>>> > >> > > > > > >> > but, > > > > > > > > >> >>>> > >> > > > > > >> > it seems to me, CLI is a very > > > important > > > > > > > client > > > > > > > > >> of this > > > > > > > > >> >>>> > >> > feature. > > > > > > > > >> >>>> > >> > > In > > > > > > > > >> >>>> > >> > > > > > this > > > > > > > > >> >>>> > >> > > > > > >> > case, > > > > > > > > >> >>>> > >> > > > > > >> > seems logical, we would like to > > > provide > > > > > > users > > > > > > > > >> with > > > > > > > > >> >>>> rich > > > > > > > > >> >>>> > >> > > experience > > > > > > > > >> >>>> > >> > > > > in > > > > > > > > >> >>>> > >> > > > > > >> terms > > > > > > > > >> >>>> > >> > > > > > >> > of > > > > > > > > >> >>>> > >> > > > > > >> > getting results / errors of the > > > > executed > > > > > > > > >> commands. > > > > > > > > >> >>>> Usually > > > > > > > > >> >>>> > >> we > > > > > > > > >> >>>> > >> > > > supply > > > > > > > > >> >>>> > >> > > > > > >> with > > > > > > > > >> >>>> > >> > > > > > >> > responses only errorCode, which > looks > > > > > very > > > > > > > > >> limiting, > > > > > > > > >> >>>> in case > > > > > > > > >> >>>> > >> > of > > > > > > > > >> >>>> > >> > > > CLI > > > > > > > > >> >>>> > >> > > > > we > > > > > > > > >> >>>> > >> > > > > > >> may > > > > > > > > >> >>>> > >> > > > > > >> > want to print human readable > error > > > > > > > description. > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > >> >>>> > >> > > > > > >> > So, taking into account previous > item > > > > > about > > > > > > > > >> batching, > > > > > > > > >> >>>> what > > > > > > > > >> >>>> > >> do > > > > > > > > >> >>>> > >> > > you > > > > > > > > >> >>>> > >> > > > > > think > > > > > > > > >> >>>> > >> > > > > > >> > about > > > > > > > > >> >>>> > >> > > > > > >> > having smth like: > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > >> >>>> > >> > > > > > >> > ('create' doesn't support regexp) > > > > > > > > >> >>>> > >> > > > > > >> > CreateTopicRequest => TopicName > > > > > Partitions > > > > > > > > >> Replicas > > > > > > > > >> >>>> > >> > > > > ReplicaAssignment > > > > > > > > >> >>>> > >> > > > > > >> > [Config] > > > > > > > > >> >>>> > >> > > > > > >> > CreateTopicResponse => ErrorCode > > > > > > > > ErrorDescription > > > > > > > > >> >>>> > >> > > > > > >> > ErrorCode => int16 > > > > > > > > >> >>>> > >> > > > > > >> > ErrorDescription => string > (empty > > > if > > > > > > > > >> successful) > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > >> >>>> > >> > > > > > >> > AlterTopicRequest -> > TopicNameRegexp > > > > > > > Partitions > > > > > > > > >> >>>> > >> > > ReplicaAssignment > > > > > > > > >> >>>> > >> > > > > > >> > [AddedConfig] [DeletedConfig] > > > > > > > > >> >>>> > >> > > > > > >> > AlterTopicResponse -> [TopicName > > > > > ErrorCode > > > > > > > > >> >>>> ErrorDescription] > > > > > > > > >> >>>> > >> > > > > > >> > CommandErrorCode > > > > CommandErrorDescription > > > > > > > > >> >>>> > >> > > > > > >> > CommandErrorCode => int16 > > > > > > > > >> >>>> > >> > > > > > >> > CommandErrorDescription => > string > > > > > > (nonempty > > > > > > > > in > > > > > > > > >> case > > > > > > > > >> >>>> of > > > > > > > > >> >>>> > >> fatal > > > > > > > > >> >>>> > >> > > > > error, > > > > > > > > >> >>>> > >> > > > > > >> e.g. > > > > > > > > >> >>>> > >> > > > > > >> > we couldn't get topics by regexp) > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > >> >>>> > >> > > > > > >> > DescribeTopicRequest -> > > > TopicNameRegexp > > > > > > > > >> >>>> > >> > > > > > >> > DescribeTopicResponse -> > [TopicName > > > > > > > > >> TopicDescription > > > > > > > > >> >>>> > >> ErrorCode > > > > > > > > >> >>>> > >> > > > > > >> > ErrorDescription] > CommandErrorCode > > > > > > > > >> >>>> CommandErrorDescription > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > >> >>>> > >> > > > > > >> > Also, any thoughts about our > > > discussion > > > > > > > > regarding > > > > > > > > >> >>>> re-routing > > > > > > > > >> >>>> > >> > > > > facility? > > > > > > > > >> >>>> > >> > > > > > >> In > > > > > > > > >> >>>> > >> > > > > > >> > my > > > > > > > > >> >>>> > >> > > > > > >> > understanding, it is like between > > > > > > augmenting > > > > > > > > >> >>>> > >> > > TopicMetadataRequest > > > > > > > > >> >>>> > >> > > > > > >> > (to include at least > controllerId) > > > and > > > > > > > > >> implementing > > > > > > > > >> >>>> new > > > > > > > > >> >>>> > >> > generic > > > > > > > > >> >>>> > >> > > > > > >> re-routing > > > > > > > > >> >>>> > >> > > > > > >> > facility so sending messages to > > > > > controller > > > > > > > will > > > > > > > > >> be > > > > > > > > >> >>>> handled > > > > > > > > >> >>>> > >> by > > > > > > > > >> >>>> > >> > > it. > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > >> >>>> > >> > > > > > >> > Thanks, > > > > > > > > >> >>>> > >> > > > > > >> > Andrii Biletskyi > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > >> >>>> > >> > > > > > >> > On Mon, Feb 16, 2015 at 5:26 PM, > > > Andrii > > > > > > > > >> Biletskyi < > > > > > > > > >> >>>> > >> > > > > > >> > andrii.bilets...@stealth.ly> > wrote: > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > >> >>>> > >> > > > > > >> > > @Guozhang: > > > > > > > > >> >>>> > >> > > > > > >> > > Thanks for your comments, I've > > > > answered > > > > > > > some > > > > > > > > of > > > > > > > > >> >>>> those. The > > > > > > > > >> >>>> > >> > > main > > > > > > > > >> >>>> > >> > > > > > thing > > > > > > > > >> >>>> > >> > > > > > >> is > > > > > > > > >> >>>> > >> > > > > > >> > > having merged request for > > > > > > > > >> >>>> create-alter-delete-describe - I > > > > > > > > >> >>>> > >> > > have > > > > > > > > >> >>>> > >> > > > > some > > > > > > > > >> >>>> > >> > > > > > >> > > concerns about this approach. > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > > @*Jay*: > > > > > > > > >> >>>> > >> > > > > > >> > > I see that introduced > > > > > > ClusterMetadaRequest > > > > > > > is > > > > > > > > >> also > > > > > > > > >> >>>> one of > > > > > > > > >> >>>> > >> > the > > > > > > > > >> >>>> > >> > > > > > >> concerns. > > > > > > > > >> >>>> > >> > > > > > >> > We > > > > > > > > >> >>>> > >> > > > > > >> > > can solve it if we implement > > > > re-routing > > > > > > > > >> facility. > > > > > > > > >> >>>> But I > > > > > > > > >> >>>> > >> > agree > > > > > > > > >> >>>> > >> > > > with > > > > > > > > >> >>>> > >> > > > > > >> > > Guozhang - it will make > clients' > > > > > > internals > > > > > > > a > > > > > > > > >> little > > > > > > > > >> >>>> bit > > > > > > > > >> >>>> > >> > easier > > > > > > > > >> >>>> > >> > > > but > > > > > > > > >> >>>> > >> > > > > > >> this > > > > > > > > >> >>>> > >> > > > > > >> > > seems to be a complex logic to > > > > > implement > > > > > > > and > > > > > > > > >> >>>> support then. > > > > > > > > >> >>>> > >> > > > > > Especially > > > > > > > > >> >>>> > >> > > > > > >> for > > > > > > > > >> >>>> > >> > > > > > >> > > Fetch and Produce (even if we > add > > > > > > > re-routing > > > > > > > > >> later > > > > > > > > >> >>>> for > > > > > > > > >> >>>> > >> these > > > > > > > > >> >>>> > >> > > > > > >> requests). > > > > > > > > >> >>>> > >> > > > > > >> > > Also people will tend to avoid > this > > > > > > > > re-routing > > > > > > > > >> >>>> facility > > > > > > > > >> >>>> > >> and > > > > > > > > >> >>>> > >> > > hold > > > > > > > > >> >>>> > >> > > > > > local > > > > > > > > >> >>>> > >> > > > > > >> > > cluster cache to ensure their > > > > > > high-priority > > > > > > > > >> requests > > > > > > > > >> >>>> > >> (which > > > > > > > > >> >>>> > >> > > some > > > > > > > > >> >>>> > >> > > > > of > > > > > > > > >> >>>> > >> > > > > > >> the > > > > > > > > >> >>>> > >> > > > > > >> > > admin requests are) not sent to > > > some > > > > > busy > > > > > > > > >> broker > > > > > > > > >> >>>> where > > > > > > > > >> >>>> > >> they > > > > > > > > >> >>>> > >> > > wait > > > > > > > > >> >>>> > >> > > > > to > > > > > > > > >> >>>> > >> > > > > > be > > > > > > > > >> >>>> > >> > > > > > >> > > routed to the correct one. > > > > > > > > >> >>>> > >> > > > > > >> > > As pointed out by Jun here ( > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > > > > > > > > >> >>>> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-1772?focusedCommentId=14234530&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14234530 > > > > > > > > >> >>>> > >> > > > > > >> > ) > > > > > > > > >> >>>> > >> > > > > > >> > > to solve the issue we might > > > > introduce a > > > > > > > > message > > > > > > > > >> >>>> type to > > > > > > > > >> >>>> > >> get > > > > > > > > >> >>>> > >> > > > > cluster > > > > > > > > >> >>>> > >> > > > > > >> > state. > > > > > > > > >> >>>> > >> > > > > > >> > > But I agree we can just update > > > > > > > > >> >>>> TopicMetadataResponse to > > > > > > > > >> >>>> > >> > > include > > > > > > > > >> >>>> > >> > > > > > >> > > controllerId (and probably smth > > > > else). > > > > > > > > >> >>>> > >> > > > > > >> > > What are you thougths? > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > > Thanks, > > > > > > > > >> >>>> > >> > > > > > >> > > Andrii > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > > On Thu, Feb 12, 2015 at 8:31 > AM, > > > > > Guozhang > > > > > > > > Wang > > > > > > > > >> < > > > > > > > > >> >>>> > >> > > > > wangg...@gmail.com> > > > > > > > > >> >>>> > >> > > > > > >> > wrote: > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > >> I think for the topics > commands we > > > > can > > > > > > > > >> actually > > > > > > > > >> >>>> merge > > > > > > > > >> >>>> > >> > > > > > >> > >> create/alter/delete/describe > as > > > one > > > > > > > request > > > > > > > > >> type > > > > > > > > >> >>>> since > > > > > > > > >> >>>> > >> > their > > > > > > > > >> >>>> > >> > > > > > formats > > > > > > > > >> >>>> > >> > > > > > >> are > > > > > > > > >> >>>> > >> > > > > > >> > >> very much similar, and keep > > > > > list-topics > > > > > > > and > > > > > > > > >> others > > > > > > > > >> >>>> like > > > > > > > > >> >>>> > >> > > > > > >> > >> partition-reassignment / > > > > > > > > >> preferred-leader-election > > > > > > > > >> >>>> as > > > > > > > > >> >>>> > >> > > separate > > > > > > > > >> >>>> > >> > > > > > >> request > > > > > > > > >> >>>> > >> > > > > > >> > >> types, I also left some other > > > > comments > > > > > > on > > > > > > > > the > > > > > > > > >> RB ( > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > https://reviews.apache.org/r/29301/ > > > > ). > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> On Wed, Feb 11, 2015 at 2:04 > PM, > > > Jay > > > > > > > Kreps < > > > > > > > > >> >>>> > >> > > > jay.kr...@gmail.com> > > > > > > > > >> >>>> > >> > > > > > >> wrote: > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > Yeah I totally agree that we > > > don't > > > > > > want > > > > > > > to > > > > > > > > >> just > > > > > > > > >> >>>> have > > > > > > > > >> >>>> > >> one > > > > > > > > >> >>>> > >> > > "do > > > > > > > > >> >>>> > >> > > > > > admin > > > > > > > > >> >>>> > >> > > > > > >> > >> stuff" > > > > > > > > >> >>>> > >> > > > > > >> > >> > command that has the union > of > > > all > > > > > > > > >> parameters. > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > >> >>>> > >> > > > > > >> > >> > What I am saying is that > command > > > > > line > > > > > > > > tools > > > > > > > > >> are > > > > > > > > >> >>>> one > > > > > > > > >> >>>> > >> > client > > > > > > > > >> >>>> > >> > > of > > > > > > > > >> >>>> > >> > > > > the > > > > > > > > >> >>>> > >> > > > > > >> > >> > administrative apis, but > these > > > > will > > > > > be > > > > > > > > used > > > > > > > > >> in a > > > > > > > > >> >>>> number > > > > > > > > >> >>>> > >> > of > > > > > > > > >> >>>> > >> > > > > > >> scenarios > > > > > > > > >> >>>> > >> > > > > > >> > so > > > > > > > > >> >>>> > >> > > > > > >> > >> > they should make logical > sense > > > > even > > > > > in > > > > > > > the > > > > > > > > >> >>>> absence of > > > > > > > > >> >>>> > >> the > > > > > > > > >> >>>> > >> > > > > command > > > > > > > > >> >>>> > >> > > > > > >> line > > > > > > > > >> >>>> > >> > > > > > >> > >> > tool. Hence comments like > trying > > > > to > > > > > > > > clarify > > > > > > > > >> the > > > > > > > > >> >>>> > >> > > relationship > > > > > > > > >> >>>> > >> > > > > > >> between > > > > > > > > >> >>>> > >> > > > > > >> > >> > ClusterMetadata and > > > > > > > TopicMetadata...these > > > > > > > > >> kinds > > > > > > > > >> >>>> of > > > > > > > > >> >>>> > >> things > > > > > > > > >> >>>> > >> > > > > really > > > > > > > > >> >>>> > >> > > > > > >> need > > > > > > > > >> >>>> > >> > > > > > >> > >> to be > > > > > > > > >> >>>> > >> > > > > > >> > >> > thought through. > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > >> >>>> > >> > > > > > >> > >> > Hope that makes sense. > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > >> >>>> > >> > > > > > >> > >> > -Jay > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > >> >>>> > >> > > > > > >> > >> > On Wed, Feb 11, 2015 at > 1:41 PM, > > > > > > Andrii > > > > > > > > >> >>>> Biletskyi < > > > > > > > > >> >>>> > >> > > > > > >> > >> > andrii.bilets...@stealth.ly > > > > > > wrote: > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > Jay, > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > Thanks for answering. You > > > > > understood > > > > > > > > >> >>>> correctly, most > > > > > > > > >> >>>> > >> of > > > > > > > > >> >>>> > >> > > my > > > > > > > > >> >>>> > >> > > > > > >> comments > > > > > > > > >> >>>> > >> > > > > > >> > >> were > > > > > > > > >> >>>> > >> > > > > > >> > >> > > related to your point 1) - > > > about > > > > > > "well > > > > > > > > >> >>>> thought-out" > > > > > > > > >> >>>> > >> > apis. > > > > > > > > >> >>>> > >> > > > > Also, > > > > > > > > >> >>>> > >> > > > > > >> yes, > > > > > > > > >> >>>> > >> > > > > > >> > >> as I > > > > > > > > >> >>>> > >> > > > > > >> > >> > > understood we would like > to > > > > > > introduce > > > > > > > a > > > > > > > > >> single > > > > > > > > >> >>>> > >> unified > > > > > > > > >> >>>> > >> > > CLI > > > > > > > > >> >>>> > >> > > > > tool > > > > > > > > >> >>>> > >> > > > > > >> with > > > > > > > > >> >>>> > >> > > > > > >> > >> > > centralized server-side > > > request > > > > > > > handling > > > > > > > > >> for > > > > > > > > >> >>>> lots of > > > > > > > > >> >>>> > >> > > > existing > > > > > > > > >> >>>> > >> > > > > > >> ones > > > > > > > > >> >>>> > >> > > > > > >> > >> (incl. > > > > > > > > >> >>>> > >> > > > > > >> > >> > > TopicCommand, > > > > CommitOffsetChecker, > > > > > > > > >> >>>> > >> ReassignPartitions, > > > > > > > > >> >>>> > >> > > smth > > > > > > > > >> >>>> > >> > > > > > else > > > > > > > > >> >>>> > >> > > > > > >> if > > > > > > > > >> >>>> > >> > > > > > >> > >> added > > > > > > > > >> >>>> > >> > > > > > >> > >> > > in future). In our > previous > > > > > > > discussion ( > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > >> >>>> https://issues.apache.org/jira/browse/KAFKA-1694) > > > > > > > > >> >>>> > >> > people > > > > > > > > >> >>>> > >> > > > > said > > > > > > > > >> >>>> > >> > > > > > >> > they'd > > > > > > > > >> >>>> > >> > > > > > >> > >> > > rather > > > > > > > > >> >>>> > >> > > > > > >> > >> > > have a separate message > for > > > each > > > > > > > > command, > > > > > > > > >> so, > > > > > > > > >> >>>> yes, > > > > > > > > >> >>>> > >> this > > > > > > > > >> >>>> > >> > > > way I > > > > > > > > >> >>>> > >> > > > > > >> came > > > > > > > > >> >>>> > >> > > > > > >> > to > > > > > > > > >> >>>> > >> > > > > > >> > >> 1-1 > > > > > > > > >> >>>> > >> > > > > > >> > >> > > mapping between commands > in > > > the > > > > > tool > > > > > > > and > > > > > > > > >> >>>> protocol > > > > > > > > >> >>>> > >> > > > additions. > > > > > > > > >> >>>> > >> > > > > > But > > > > > > > > >> >>>> > >> > > > > > >> I > > > > > > > > >> >>>> > >> > > > > > >> > >> might > > > > > > > > >> >>>> > >> > > > > > >> > >> > be > > > > > > > > >> >>>> > >> > > > > > >> > >> > > wrong. > > > > > > > > >> >>>> > >> > > > > > >> > >> > > At the end I just try to > start > > > > > > > > discussion > > > > > > > > >> how > > > > > > > > >> >>>> at > > > > > > > > >> >>>> > >> least > > > > > > > > >> >>>> > >> > > > > > generally > > > > > > > > >> >>>> > >> > > > > > >> > this > > > > > > > > >> >>>> > >> > > > > > >> > >> > > protocol should look like. > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > Thanks, > > > > > > > > >> >>>> > >> > > > > > >> > >> > > Andrii > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > On Wed, Feb 11, 2015 at > 11:10 > > > > PM, > > > > > > Jay > > > > > > > > >> Kreps < > > > > > > > > >> >>>> > >> > > > > > jay.kr...@gmail.com > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > >> >>>> > >> > > > > > >> > >> wrote: > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > Hey Andrii, > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > To answer your earlier > > > > question > > > > > we > > > > > > > > just > > > > > > > > >> >>>> really > > > > > > > > >> >>>> > >> can't > > > > > > > > >> >>>> > >> > be > > > > > > > > >> >>>> > >> > > > > > adding > > > > > > > > >> >>>> > >> > > > > > >> any > > > > > > > > >> >>>> > >> > > > > > >> > >> more > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > scala protocol objects. > > > These > > > > > > things > > > > > > > > are > > > > > > > > >> >>>> super hard > > > > > > > > >> >>>> > >> > to > > > > > > > > >> >>>> > >> > > > > > maintain > > > > > > > > >> >>>> > >> > > > > > >> > >> because > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > they hand code the byte > > > > parsing > > > > > > and > > > > > > > > >> don't > > > > > > > > >> >>>> have good > > > > > > > > >> >>>> > >> > > > > > versioning > > > > > > > > >> >>>> > >> > > > > > >> > >> support. > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > Since we are already > > > planning > > > > on > > > > > > > > >> converting > > > > > > > > >> >>>> we > > > > > > > > >> >>>> > >> > > definitely > > > > > > > > >> >>>> > >> > > > > > don't > > > > > > > > >> >>>> > >> > > > > > >> > >> want to > > > > > > > > >> >>>> > >> > > > > > >> > >> > > add > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > a ton more of > these--they > > > are > > > > > > total > > > > > > > > tech > > > > > > > > >> >>>> debt. > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > What does it mean that > the > > > > > changes > > > > > > > are > > > > > > > > >> >>>> isolated > > > > > > > > >> >>>> > >> from > > > > > > > > >> >>>> > >> > > the > > > > > > > > >> >>>> > >> > > > > > >> current > > > > > > > > >> >>>> > >> > > > > > >> > >> code > > > > > > > > >> >>>> > >> > > > > > >> > >> > > base? > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > I actually didn't > understand > > > > the > > > > > > > > >> remaining > > > > > > > > >> >>>> > >> comments, > > > > > > > > >> >>>> > >> > > > which > > > > > > > > >> >>>> > >> > > > > of > > > > > > > > >> >>>> > >> > > > > > >> the > > > > > > > > >> >>>> > >> > > > > > >> > >> > points > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > are you responding to? > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > Maybe one sticking point > > > here > > > > is > > > > > > > that > > > > > > > > it > > > > > > > > >> >>>> seems like > > > > > > > > >> >>>> > >> > you > > > > > > > > >> >>>> > >> > > > > want > > > > > > > > >> >>>> > >> > > > > > to > > > > > > > > >> >>>> > >> > > > > > >> > make > > > > > > > > >> >>>> > >> > > > > > >> > >> > some > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > kind of tool, and you > have > > > > made > > > > > a > > > > > > > 1-1 > > > > > > > > >> mapping > > > > > > > > >> >>>> > >> between > > > > > > > > >> >>>> > >> > > > > > commands > > > > > > > > >> >>>> > >> > > > > > >> you > > > > > > > > >> >>>> > >> > > > > > >> > >> > > imagine > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > in the tool and protocol > > > > > > additions. > > > > > > > I > > > > > > > > >> want > > > > > > > > >> >>>> to make > > > > > > > > >> >>>> > >> > sure > > > > > > > > >> >>>> > >> > > > we > > > > > > > > >> >>>> > >> > > > > > >> don't > > > > > > > > >> >>>> > >> > > > > > >> > do > > > > > > > > >> >>>> > >> > > > > > >> > >> > that. > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > The protocol needs to be > > > > really > > > > > > > really > > > > > > > > >> well > > > > > > > > >> >>>> thought > > > > > > > > >> >>>> > >> > out > > > > > > > > >> >>>> > >> > > > > > against > > > > > > > > >> >>>> > >> > > > > > >> > many > > > > > > > > >> >>>> > >> > > > > > >> > >> > use > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > cases so it should make > > > > perfect > > > > > > > > logical > > > > > > > > >> >>>> sense in > > > > > > > > >> >>>> > >> the > > > > > > > > >> >>>> > >> > > > > absence > > > > > > > > >> >>>> > >> > > > > > of > > > > > > > > >> >>>> > >> > > > > > >> > >> knowing > > > > > > > > >> >>>> > >> > > > > > >> > >> > > the > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > command line tool, > right? > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > -Jay > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > On Wed, Feb 11, 2015 at > > > 11:57 > > > > > AM, > > > > > > > > Andrii > > > > > > > > >> >>>> Biletskyi > > > > > > > > >> >>>> > >> < > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > andrii.bilets...@stealth.ly > > > > > > > > > > wrote: > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > Hey Jay, > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > I would like to > continue > > > > this > > > > > > > > >> discussion > > > > > > > > >> >>>> as it > > > > > > > > >> >>>> > >> seem > > > > > > > > >> >>>> > >> > > > there > > > > > > > > >> >>>> > >> > > > > > is > > > > > > > > >> >>>> > >> > > > > > >> no > > > > > > > > >> >>>> > >> > > > > > >> > >> > > progress > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > here. > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > First of all, could > you > > > > please > > > > > > > > explain > > > > > > > > >> >>>> what did > > > > > > > > >> >>>> > >> you > > > > > > > > >> >>>> > >> > > > mean > > > > > > > > >> >>>> > >> > > > > in > > > > > > > > >> >>>> > >> > > > > > >> 2? > > > > > > > > >> >>>> > >> > > > > > >> > How > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > exactly > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > are we going to > migrate to > > > > the > > > > > > new > > > > > > > > >> java > > > > > > > > >> >>>> protocol > > > > > > > > >> >>>> > >> > > > > > definitions. > > > > > > > > >> >>>> > >> > > > > > >> > And > > > > > > > > >> >>>> > >> > > > > > >> > >> why > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > it's > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > a blocker for > centralized > > > > CLI? > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > I agree with you, this > > > > feature > > > > > > > > >> includes > > > > > > > > >> >>>> lots of > > > > > > > > >> >>>> > >> > > stuff, > > > > > > > > >> >>>> > >> > > > > but > > > > > > > > >> >>>> > >> > > > > > >> > >> thankfully > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > almost all changes are > > > > > isolated > > > > > > > from > > > > > > > > >> the > > > > > > > > >> >>>> current > > > > > > > > >> >>>> > >> > code > > > > > > > > >> >>>> > >> > > > > base, > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > so the main thing, I > > > think, > > > > we > > > > > > > need > > > > > > > > to > > > > > > > > >> >>>> agree is > > > > > > > > >> >>>> > >> > RQ/RP > > > > > > > > >> >>>> > >> > > > > > format. > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > So how can we start > > > > discussion > > > > > > > about > > > > > > > > >> the > > > > > > > > >> >>>> concrete > > > > > > > > >> >>>> > >> > > > > messages > > > > > > > > >> >>>> > >> > > > > > >> > format? > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > Can we take ( > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > > > > > > > > >> >>>> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-ProposedRQ/RPFormat > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > ) > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > as starting point? > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > We had some doubts > earlier > > > > > > whether > > > > > > > > it > > > > > > > > >> worth > > > > > > > > >> >>>> > >> > > introducing > > > > > > > > >> >>>> > >> > > > > one > > > > > > > > >> >>>> > >> > > > > > >> > >> generic > > > > > > > > >> >>>> > >> > > > > > >> > >> > > Admin > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > Request for all > commands ( > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > >> >>>> https://issues.apache.org/jira/browse/KAFKA-1694 > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > ) > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > but then everybody > agreed > > > it > > > > > > would > > > > > > > > be > > > > > > > > >> >>>> better to > > > > > > > > >> >>>> > >> > have > > > > > > > > >> >>>> > >> > > > > > separate > > > > > > > > >> >>>> > >> > > > > > >> > >> message > > > > > > > > >> >>>> > >> > > > > > >> > >> > > for > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > each admin command. > The > > > > > Request > > > > > > > part > > > > > > > > >> is > > > > > > > > >> >>>> really > > > > > > > > >> >>>> > >> > > dictated > > > > > > > > >> >>>> > >> > > > > > from > > > > > > > > >> >>>> > >> > > > > > >> the > > > > > > > > >> >>>> > >> > > > > > >> > >> > > command > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > (e.g. TopicCommand) > > > > arguments > > > > > > > > itself, > > > > > > > > >> so > > > > > > > > >> >>>> the > > > > > > > > >> >>>> > >> > proposed > > > > > > > > >> >>>> > >> > > > > > version > > > > > > > > >> >>>> > >> > > > > > >> > >> should > > > > > > > > >> >>>> > >> > > > > > >> > >> > be > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > fine (let's put aside > for > > > > now > > > > > > > > remarks > > > > > > > > >> about > > > > > > > > >> >>>> > >> > Optional > > > > > > > > >> >>>> > >> > > > > type, > > > > > > > > >> >>>> > >> > > > > > >> > >> batching, > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > configs normalization > - I > > > > > agree > > > > > > > with > > > > > > > > >> all of > > > > > > > > >> >>>> > >> them). > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > So the second part is > > > > > Response. > > > > > > I > > > > > > > > see > > > > > > > > >> >>>> there are > > > > > > > > >> >>>> > >> two > > > > > > > > >> >>>> > >> > > > cases > > > > > > > > >> >>>> > >> > > > > > >> here. > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > a) "Mutate" requests - > > > > > > > > >> Create/Alter/... ; > > > > > > > > >> >>>> b) > > > > > > > > >> >>>> > >> "Get" > > > > > > > > >> >>>> > >> > > > > > requests - > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > List/Describe... > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > a) should only hold > > > request > > > > > > result > > > > > > > > >> >>>> (regardless > > > > > > > > >> >>>> > >> what > > > > > > > > >> >>>> > >> > > we > > > > > > > > >> >>>> > >> > > > > > decide > > > > > > > > >> >>>> > >> > > > > > >> > >> about > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > blocking/non-blocking > > > > commands > > > > > > > > >> execution). > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > Usually we provide > error > > > > code > > > > > in > > > > > > > > >> response > > > > > > > > >> >>>> but > > > > > > > > >> >>>> > >> since > > > > > > > > >> >>>> > >> > > we > > > > > > > > >> >>>> > >> > > > > will > > > > > > > > >> >>>> > >> > > > > > >> use > > > > > > > > >> >>>> > >> > > > > > >> > >> this > > > > > > > > >> >>>> > >> > > > > > >> > >> > in > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > interactive shell we > need > > > > some > > > > > > > human > > > > > > > > >> >>>> readable > > > > > > > > >> >>>> > >> error > > > > > > > > >> >>>> > >> > > > > > >> description > > > > > > > > >> >>>> > >> > > > > > >> > - > > > > > > > > >> >>>> > >> > > > > > >> > >> so > > > > > > > > >> >>>> > >> > > > > > >> > >> > I > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > added errorDesription > > > field > > > > > > where > > > > > > > > you > > > > > > > > >> can > > > > > > > > >> >>>> at > > > > > > > > >> >>>> > >> least > > > > > > > > >> >>>> > >> > > > leave > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > exception.getMessage. > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > b) in addition to > previous > > > > > item > > > > > > > > >> message > > > > > > > > >> >>>> should > > > > > > > > >> >>>> > >> hold > > > > > > > > >> >>>> > >> > > > > command > > > > > > > > >> >>>> > >> > > > > > >> > >> specific > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > response data. We can > > > > discuss > > > > > in > > > > > > > > >> detail > > > > > > > > >> >>>> each of > > > > > > > > >> >>>> > >> > them > > > > > > > > >> >>>> > >> > > > but > > > > > > > > >> >>>> > >> > > > > > >> let's > > > > > > > > >> >>>> > >> > > > > > >> > for > > > > > > > > >> >>>> > >> > > > > > >> > >> > now > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > agree about the > overall > > > > > pattern. > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > Thanks, > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > Andrii Biletskyi > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > On Fri, Jan 23, 2015 > at > > > 6:59 > > > > > AM, > > > > > > > Jay > > > > > > > > >> Kreps > > > > > > > > >> >>>> < > > > > > > > > >> >>>> > >> > > > > > >> jay.kr...@gmail.com > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > wrote: > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > Hey Joe, > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > This is great. A few > > > > > comments > > > > > > on > > > > > > > > >> KIP-4 > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 1. This is much > needed > > > > > > > > >> functionality, > > > > > > > > >> >>>> but there > > > > > > > > >> >>>> > >> > > are a > > > > > > > > >> >>>> > >> > > > > lot > > > > > > > > >> >>>> > >> > > > > > >> of > > > > > > > > >> >>>> > >> > > > > > >> > >> the so > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > let's > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > really think these > > > > protocols > > > > > > > > >> through. We > > > > > > > > >> >>>> really > > > > > > > > >> >>>> > >> > > want > > > > > > > > >> >>>> > >> > > > to > > > > > > > > >> >>>> > >> > > > > > >> end up > > > > > > > > >> >>>> > >> > > > > > >> > >> > with a > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > set > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > of well thought-out, > > > > > > orthoganol > > > > > > > > >> apis. > > > > > > > > >> >>>> For this > > > > > > > > >> >>>> > >> > > > reason I > > > > > > > > >> >>>> > >> > > > > > >> think > > > > > > > > >> >>>> > >> > > > > > >> > >> it is > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > really > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > important to think > > > through > > > > > the > > > > > > > end > > > > > > > > >> state > > > > > > > > >> >>>> even > > > > > > > > >> >>>> > >> if > > > > > > > > >> >>>> > >> > > that > > > > > > > > >> >>>> > >> > > > > > >> includes > > > > > > > > >> >>>> > >> > > > > > >> > >> APIs > > > > > > > > >> >>>> > >> > > > > > >> > >> > > we > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > won't implement in > the > > > > first > > > > > > > > phase. > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 2. Let's please > please > > > > > please > > > > > > > wait > > > > > > > > >> until > > > > > > > > >> >>>> we > > > > > > > > >> >>>> > >> have > > > > > > > > >> >>>> > >> > > > > switched > > > > > > > > >> >>>> > >> > > > > > >> the > > > > > > > > >> >>>> > >> > > > > > >> > >> > server > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > over > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > to the new java > protocol > > > > > > > > >> definitions. If > > > > > > > > >> >>>> we add > > > > > > > > >> >>>> > >> > > > upteen > > > > > > > > >> >>>> > >> > > > > > >> more ad > > > > > > > > >> >>>> > >> > > > > > >> > >> hoc > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > scala > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > objects that is just > > > > > > generating > > > > > > > > more > > > > > > > > >> >>>> work for > > > > > > > > >> >>>> > >> the > > > > > > > > >> >>>> > >> > > > > > >> conversion > > > > > > > > >> >>>> > >> > > > > > >> > we > > > > > > > > >> >>>> > >> > > > > > >> > >> > know > > > > > > > > >> >>>> > >> > > > > > >> > >> > > we > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > have to do. > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 3. This proposal > > > > introduces > > > > > a > > > > > > > new > > > > > > > > >> type of > > > > > > > > >> >>>> > >> > optional > > > > > > > > >> >>>> > >> > > > > > >> parameter. > > > > > > > > >> >>>> > >> > > > > > >> > >> This > > > > > > > > >> >>>> > >> > > > > > >> > >> > is > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > inconsistent with > > > > everything > > > > > > > else > > > > > > > > >> in the > > > > > > > > >> >>>> > >> protocol > > > > > > > > >> >>>> > >> > > > where > > > > > > > > >> >>>> > >> > > > > > we > > > > > > > > >> >>>> > >> > > > > > >> use > > > > > > > > >> >>>> > >> > > > > > >> > >> -1 > > > > > > > > >> >>>> > >> > > > > > >> > >> > or > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > some > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > other marker value. > You > > > > > could > > > > > > > > argue > > > > > > > > >> >>>> either way > > > > > > > > >> >>>> > >> > but > > > > > > > > >> >>>> > >> > > > > let's > > > > > > > > >> >>>> > >> > > > > > >> stick > > > > > > > > >> >>>> > >> > > > > > >> > >> with > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > that > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > for consistency. For > > > > clients > > > > > > > that > > > > > > > > >> >>>> implemented > > > > > > > > >> >>>> > >> the > > > > > > > > >> >>>> > >> > > > > > protocol > > > > > > > > >> >>>> > >> > > > > > >> in > > > > > > > > >> >>>> > >> > > > > > >> > a > > > > > > > > >> >>>> > >> > > > > > >> > >> > > better > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > way > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > than our scala code > > > these > > > > > > basic > > > > > > > > >> >>>> primitives are > > > > > > > > >> >>>> > >> > hard > > > > > > > > >> >>>> > >> > > > to > > > > > > > > >> >>>> > >> > > > > > >> change. > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 4. ClusterMetadata: > This > > > > > seems > > > > > > > to > > > > > > > > >> >>>> duplicate > > > > > > > > >> >>>> > >> > > > > > >> > TopicMetadataRequest > > > > > > > > >> >>>> > >> > > > > > >> > >> > > which > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > has > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > brokers, topics, and > > > > > > > partitions. I > > > > > > > > >> think > > > > > > > > >> >>>> we > > > > > > > > >> >>>> > >> > should > > > > > > > > >> >>>> > >> > > > > rename > > > > > > > > >> >>>> > >> > > > > > >> that > > > > > > > > >> >>>> > >> > > > > > >> > >> > > request > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > ClusterMetadataRequest > > > (or > > > > > > just > > > > > > > > >> >>>> > >> MetadataRequest) > > > > > > > > >> >>>> > >> > > and > > > > > > > > >> >>>> > >> > > > > > >> include > > > > > > > > >> >>>> > >> > > > > > >> > >> the id > > > > > > > > >> >>>> > >> > > > > > >> > >> > > of > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > the > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > controller. Or are > there > > > > > other > > > > > > > > >> things we > > > > > > > > >> >>>> could > > > > > > > > >> >>>> > >> > add > > > > > > > > >> >>>> > >> > > > > here? > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 5. We have a > tendency to > > > > try > > > > > > to > > > > > > > > >> make a > > > > > > > > >> >>>> lot of > > > > > > > > >> >>>> > >> > > > requests > > > > > > > > >> >>>> > >> > > > > > that > > > > > > > > >> >>>> > >> > > > > > >> > can > > > > > > > > >> >>>> > >> > > > > > >> > >> > only > > > > > > > > >> >>>> > >> > > > > > >> > >> > > go > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > to > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > particular nodes. > This > > > > adds > > > > > a > > > > > > > lot > > > > > > > > of > > > > > > > > >> >>>> burden for > > > > > > > > >> >>>> > >> > > > client > > > > > > > > >> >>>> > >> > > > > > >> > >> > > implementations > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > (it > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > sounds easy but each > > > > > discovery > > > > > > > can > > > > > > > > >> fail > > > > > > > > >> >>>> in many > > > > > > > > >> >>>> > >> > > parts > > > > > > > > >> >>>> > >> > > > > so > > > > > > > > >> >>>> > >> > > > > > it > > > > > > > > >> >>>> > >> > > > > > >> > >> ends up > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > being a > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > full state machine > to do > > > > > > > right). I > > > > > > > > >> think > > > > > > > > >> >>>> we > > > > > > > > >> >>>> > >> > should > > > > > > > > >> >>>> > >> > > > > > consider > > > > > > > > >> >>>> > >> > > > > > >> > >> making > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > admin > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > commands and > ideally as > > > > many > > > > > > of > > > > > > > > the > > > > > > > > >> >>>> other apis > > > > > > > > >> >>>> > >> as > > > > > > > > >> >>>> > >> > > > > > possible > > > > > > > > >> >>>> > >> > > > > > >> > >> > available > > > > > > > > >> >>>> > >> > > > > > >> > >> > > on > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > all > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > brokers and just > > > redirect > > > > to > > > > > > the > > > > > > > > >> >>>> controller on > > > > > > > > >> >>>> > >> > the > > > > > > > > >> >>>> > >> > > > > broker > > > > > > > > >> >>>> > >> > > > > > >> > side. > > > > > > > > >> >>>> > >> > > > > > >> > >> > > Perhaps > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > there would be a > general > > > > way > > > > > > to > > > > > > > > >> >>>> encapsulate > > > > > > > > >> >>>> > >> this > > > > > > > > >> >>>> > >> > > > > > re-routing > > > > > > > > >> >>>> > >> > > > > > >> > >> > behavior. > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 6. We should > probably > > > > > > normalize > > > > > > > > the > > > > > > > > >> key > > > > > > > > >> >>>> value > > > > > > > > >> >>>> > >> > pairs > > > > > > > > >> >>>> > >> > > > > used > > > > > > > > >> >>>> > >> > > > > > >> for > > > > > > > > >> >>>> > >> > > > > > >> > >> > configs > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > rather > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > than embedding a new > > > > > > formatting. > > > > > > > > So > > > > > > > > >> two > > > > > > > > >> >>>> strings > > > > > > > > >> >>>> > >> > > > rather > > > > > > > > >> >>>> > >> > > > > > than > > > > > > > > >> >>>> > >> > > > > > >> > one > > > > > > > > >> >>>> > >> > > > > > >> > >> > with > > > > > > > > >> >>>> > >> > > > > > >> > >> > > an > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > internal equals > sign. > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 7. Is the > postcondition > > > of > > > > > > these > > > > > > > > >> APIs > > > > > > > > >> >>>> that the > > > > > > > > >> >>>> > >> > > > command > > > > > > > > >> >>>> > >> > > > > > has > > > > > > > > >> >>>> > >> > > > > > >> > >> begun or > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > that > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > the command has been > > > > > > completed? > > > > > > > It > > > > > > > > >> is a > > > > > > > > >> >>>> lot > > > > > > > > >> >>>> > >> more > > > > > > > > >> >>>> > >> > > > usable > > > > > > > > >> >>>> > >> > > > > > if > > > > > > > > >> >>>> > >> > > > > > >> the > > > > > > > > >> >>>> > >> > > > > > >> > >> > > command > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > has > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > been completed so > you > > > know > > > > > > that > > > > > > > if > > > > > > > > >> you > > > > > > > > >> >>>> create a > > > > > > > > >> >>>> > >> > > topic > > > > > > > > >> >>>> > >> > > > > and > > > > > > > > >> >>>> > >> > > > > > >> then > > > > > > > > >> >>>> > >> > > > > > >> > >> > > publish > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > to > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > it you won't get an > > > > > exception > > > > > > > > about > > > > > > > > >> >>>> there being > > > > > > > > >> >>>> > >> > no > > > > > > > > >> >>>> > >> > > > such > > > > > > > > >> >>>> > >> > > > > > >> topic. > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 8. Describe topic > and > > > list > > > > > > > topics > > > > > > > > >> >>>> duplicate a > > > > > > > > >> >>>> > >> lot > > > > > > > > >> >>>> > >> > > of > > > > > > > > >> >>>> > >> > > > > > stuff > > > > > > > > >> >>>> > >> > > > > > >> in > > > > > > > > >> >>>> > >> > > > > > >> > >> the > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > metadata > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > request. Is there a > > > reason > > > > > to > > > > > > > give > > > > > > > > >> back > > > > > > > > >> >>>> topics > > > > > > > > >> >>>> > >> > > marked > > > > > > > > >> >>>> > >> > > > > for > > > > > > > > >> >>>> > >> > > > > > >> > >> > deletion? I > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > feel > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > like if we just > make the > > > > > > > > >> post-condition > > > > > > > > >> >>>> of the > > > > > > > > >> >>>> > >> > > delete > > > > > > > > >> >>>> > >> > > > > > >> command > > > > > > > > >> >>>> > >> > > > > > >> > be > > > > > > > > >> >>>> > >> > > > > > >> > >> > that > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > the > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > topic is deleted > that > > > will > > > > > get > > > > > > > rid > > > > > > > > >> of > > > > > > > > >> >>>> the need > > > > > > > > >> >>>> > >> > for > > > > > > > > >> >>>> > >> > > > this > > > > > > > > >> >>>> > >> > > > > > >> right? > > > > > > > > >> >>>> > >> > > > > > >> > >> And > > > > > > > > >> >>>> > >> > > > > > >> > >> > it > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > will > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > be much more > intuitive. > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 9. Should we > consider > > > > > batching > > > > > > > > these > > > > > > > > >> >>>> requests? > > > > > > > > >> >>>> > >> We > > > > > > > > >> >>>> > >> > > > have > > > > > > > > >> >>>> > >> > > > > > >> > generally > > > > > > > > >> >>>> > >> > > > > > >> > >> > > tried > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > to > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > allow multiple > > > operations > > > > to > > > > > > be > > > > > > > > >> batched. > > > > > > > > >> >>>> My > > > > > > > > >> >>>> > >> > > suspicion > > > > > > > > >> >>>> > >> > > > > is > > > > > > > > >> >>>> > >> > > > > > >> that > > > > > > > > >> >>>> > >> > > > > > >> > >> > without > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > this > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > we will get a lot of > > > code > > > > > that > > > > > > > > does > > > > > > > > >> >>>> something > > > > > > > > >> >>>> > >> > like > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > for(topic: > > > > > > > > >> adminClient.listTopics()) > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > >> adminClient.describeTopic(topic) > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > this code will work > > > great > > > > > when > > > > > > > you > > > > > > > > >> test > > > > > > > > >> >>>> on 5 > > > > > > > > >> >>>> > >> > topics > > > > > > > > >> >>>> > >> > > > but > > > > > > > > >> >>>> > >> > > > > > >> not do > > > > > > > > >> >>>> > >> > > > > > >> > >> as > > > > > > > > >> >>>> > >> > > > > > >> > >> > > well > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > if > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > you have 50k. > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 10. I think we > should > > > also > > > > > > > discuss > > > > > > > > >> how > > > > > > > > >> >>>> we want > > > > > > > > >> >>>> > >> to > > > > > > > > >> >>>> > >> > > > > expose > > > > > > > > >> >>>> > >> > > > > > a > > > > > > > > >> >>>> > >> > > > > > >> > >> > > programmatic > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > JVM > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > client api for these > > > > > > operations. > > > > > > > > >> >>>> Currently > > > > > > > > >> >>>> > >> people > > > > > > > > >> >>>> > >> > > > rely > > > > > > > > >> >>>> > >> > > > > on > > > > > > > > >> >>>> > >> > > > > > >> > >> > AdminUtils > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > which > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > is totally sketchy. > I > > > > think > > > > > we > > > > > > > > >> probably > > > > > > > > >> >>>> need > > > > > > > > >> >>>> > >> > > another > > > > > > > > >> >>>> > >> > > > > > client > > > > > > > > >> >>>> > >> > > > > > >> > >> under > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > clients/ > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > that exposes > > > > administrative > > > > > > > > >> >>>> functionality. We > > > > > > > > >> >>>> > >> > will > > > > > > > > >> >>>> > >> > > > need > > > > > > > > >> >>>> > >> > > > > > >> this > > > > > > > > >> >>>> > >> > > > > > >> > >> just > > > > > > > > >> >>>> > >> > > > > > >> > >> > to > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > properly test the > new > > > > apis, > > > > > I > > > > > > > > >> suspect. We > > > > > > > > >> >>>> > >> should > > > > > > > > >> >>>> > >> > > > figure > > > > > > > > >> >>>> > >> > > > > > out > > > > > > > > >> >>>> > >> > > > > > >> > that > > > > > > > > >> >>>> > >> > > > > > >> > >> > API. > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 11. The other > > > information > > > > > that > > > > > > > > >> would be > > > > > > > > >> >>>> really > > > > > > > > >> >>>> > >> > > useful > > > > > > > > >> >>>> > >> > > > > to > > > > > > > > >> >>>> > >> > > > > > >> get > > > > > > > > >> >>>> > >> > > > > > >> > >> would > > > > > > > > >> >>>> > >> > > > > > >> > >> > be > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > information about > > > > > > > partitions--how > > > > > > > > >> much > > > > > > > > >> >>>> data is > > > > > > > > >> >>>> > >> in > > > > > > > > >> >>>> > >> > > the > > > > > > > > >> >>>> > >> > > > > > >> > partition, > > > > > > > > >> >>>> > >> > > > > > >> > >> > what > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > are > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > the segment offsets, > > > what > > > > is > > > > > > the > > > > > > > > >> log-end > > > > > > > > >> >>>> offset > > > > > > > > >> >>>> > >> > > (i.e. > > > > > > > > >> >>>> > >> > > > > > last > > > > > > > > >> >>>> > >> > > > > > >> > >> offset), > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > what > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > is > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > the compaction > point, > > > > etc. I > > > > > > > think > > > > > > > > >> that > > > > > > > > >> >>>> done > > > > > > > > >> >>>> > >> > right > > > > > > > > >> >>>> > >> > > > this > > > > > > > > >> >>>> > >> > > > > > >> would > > > > > > > > >> >>>> > >> > > > > > >> > be > > > > > > > > >> >>>> > >> > > > > > >> > >> > the > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > successor to the > very > > > > > awkward > > > > > > > > >> >>>> OffsetRequest we > > > > > > > > >> >>>> > >> > have > > > > > > > > >> >>>> > >> > > > > > today. > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > -Jay > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > On Wed, Jan 21, > 2015 at > > > > > 10:27 > > > > > > > PM, > > > > > > > > >> Joe > > > > > > > > >> >>>> Stein < > > > > > > > > >> >>>> > >> > > > > > >> > >> joe.st...@stealth.ly> > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > wrote: > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > Hi, created a KIP > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > > > > > > > > >> >>>> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > JIRA > > > > > > > > >> >>>> > >> > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-1694 > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > > >> >>>> /******************************************* > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > Joe Stein > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > Founder, > Principal > > > > > > Consultant > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > Big Data Open > Source > > > > > > Security > > > > > > > > LLC > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > http://www.stealth.ly > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > Twitter: > > > > > @allthingshadoop < > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > http://www.twitter.com/allthingshadoop > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > > >> >>>> ********************************************/ > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> -- > > > > > > > > >> >>>> > >> > > > > > >> > >> -- Guozhang > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > > >> >>>> > >> > > > > > > > > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > -- > > > > > > > > >> >>>> > >> > Jeff Holoman > > > > > > > > >> >>>> > >> > Systems Engineer > > > > > > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > > > > > > > > >> >>>> > > > > > > > > >> >>>> > > > > > > > > >> >>> > > > > > > > > >> >>> > > > > > > > > >> >>> -- > > > > > > > > >> >>> -- Guozhang > > > > > > > > >> >>> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> -- > > > > > > > > >> >> -- Guozhang > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
