Got it. Thanks for clarifying!
On Wed, Mar 18, 2015 at 11:54 AM, Andrii Biletskyi <andrii.bilets...@stealth.ly> wrote: > Gwen, > > Yes, looks like KAFKA-1927 will leave TopicMetadataRequest/Response. > But I believe, KIP is still tightly related with KAFKA-1927 since we are > not only > going to update TopicMetadataRequest there but we will introduce a bunch > of new requests too. And it probably makes sense to do those correctly from > scratch - without introducing scala request objects. As I understand you'll > have this common infrastructure code done in KAFKA-1927. > > Thanks, > Andrii Biletskyi > > On Wed, Mar 18, 2015 at 8:38 PM, Gwen Shapira <gshap...@cloudera.com> wrote: > >> On Wed, Mar 18, 2015 at 9:34 AM, Jun Rao <j...@confluent.io> wrote: >> > Andri, >> > >> > Thanks for the summary. >> > >> > 1. I just realized that in order to start working on KAFKA-1927, we will >> > need to merge the changes to OffsetCommitRequest (from 0.8.2) to trunk. >> > This is planned to be done as part of KAFKA-1634. So, we will need >> Guozhang >> > and Joel's help to wrap this up. >> >> I mentioned this in a separate thread, but it may be more relevant here: >> It looks like the SimpleConsumer API exposes TopicMetadataRequest and >> TopicMetadataResponse. >> This means that KAFKA-1927 doesn't remove this duplication. >> >> So I'm not sure we actually need KAFKA-1927 before implementing this KIP. >> This doesn't mean I'm stopping work on KAFKA-1927, but perhaps it >> means we can proceed in parallel? >> >> > 2. Thinking about this a bit more, if the semantic of those "write" >> > requests is async (i.e., after the client gets a response, it just means >> > that the operation is initiated, but not necessarily completed), we don't >> > really need to forward the requests to the controller. Instead, the >> > receiving broker can just write the operation to ZK as the admin command >> > line tool previously does. This will simplify the implementation. >> > >> > 8. There is another implementation detail for describe topic. Ideally, we >> > want to read the topic config from the broker cache, instead of >> ZooKeeper. >> > Currently, every broker reads the topic-level config for all topics. >> > However, it ignores those for topics not hosted on itself. So, we may >> need >> > to change TopicConfigManager a bit so that it caches the configs for all >> > topics. >> > >> > Thanks, >> > >> > Jun >> > >> > >> > On Tue, Mar 17, 2015 at 1:13 PM, Andrii Biletskyi < >> > andrii.bilets...@stealth.ly> wrote: >> > >> >> Guys, >> >> >> >> Thanks for a great discussion! >> >> Here are the actions points: >> >> >> >> 1. Q: Get rid of all scala requests objects, use java protocol >> definitions. >> >> A: Gwen kindly took that (KAFKA-1927). It's important to speed up >> >> review procedure >> >> there since this ticket blocks other important changes. >> >> >> >> 2. Q: Generic re-reroute facility vs client maintaining cluster state. >> >> A: Jay has added pseudo code to KAFKA-1912 - need to consider >> whether >> >> this will be >> >> easy to implement as a server-side feature (comments are >> >> welcomed!). >> >> >> >> 3. Q: Controller field in wire protocol. >> >> A: This might be useful for clients, add this to >> TopicMetadataResponse >> >> (already in KIP). >> >> >> >> 4. Q: Decoupling topic creation from TMR. >> >> A: I will add proposed by Jun solution (using clientId for that) to >> the >> >> KIP. >> >> >> >> 5. Q: Bumping new versions of TMR vs grabbing all protocol changes in >> one >> >> version. >> >> A: It was decided to try to gather all changes to protocol (before >> >> release). >> >> In case of TMR it worth checking: KAFKA-2020 and KIP-13 (quotas) >> >> >> >> 6. Q: JSON lib is needed to deserialize user's input in CLI tool. >> >> A: Use jackson for that, /tools project is a separate jar so >> shouldn't >> >> be a big deal. >> >> >> >> 7. Q: VerifyReassingPartitions vs generic status check command. >> >> A: For long-running requests like reassign partitions *progress* >> check >> >> request is useful, >> >> it makes sense to introduce it. >> >> >> >> Please add, correct me if I missed something. >> >> >> >> Thanks, >> >> Andrii Biletskyi >> >> >> >> On Tue, Mar 17, 2015 at 6:20 PM, Andrii Biletskyi < >> >> andrii.bilets...@stealth.ly> wrote: >> >> >> >> > Joel, >> >> > >> >> > You are right, I removed ClusterMetadata because we have partially >> >> > what we need in TopicMetadata. Also, as Jay pointed out earlier, we >> >> > would like to have "orthogonal" API, but at the same time we need >> >> > to be backward compatible. >> >> > >> >> > But I like your idea and even have some other arguments for this >> option: >> >> > There is also DescribeTopicRequest which was proposed in this KIP, >> >> > it returns topic configs, partitions, replication factor plus >> partition >> >> > ISR, ASR, >> >> > leader replica. The later part is really already there in >> >> > TopicMetadataRequest. >> >> > So again we'll have to add stuff to TMR, not to duplicate some info in >> >> > newly added requests. However, this way we'll end up with "monster" >> >> > request which returns cluster metadata, topic replication and config >> info >> >> > plus partition replication data. Seems logical to split TMR to >> >> > - ClusterMetadata (brokers + controller, maybe smth else) >> >> > - TopicMetadata (topic info + partition details) >> >> > But since current TMR is involved in lots of places (including network >> >> > client, >> >> > as I understand) this might be very serious change and it probably >> makes >> >> > sense to stick with current approach. >> >> > >> >> > Thanks, >> >> > Andrii Biletskyi >> >> > >> >> > >> >> > On Tue, Mar 17, 2015 at 5:29 PM, Joel Koshy <jjkosh...@gmail.com> >> wrote: >> >> > >> >> >> I may be missing some context but hopefully this will also be covered >> >> >> today: I thought the earlier proposal where there was an explicit >> >> >> ClusterMetadata request was clearer and explicit. During the course >> of >> >> >> this thread I think the conclusion was that the main need was for >> >> >> controller information and that can be rolled into the topic metadata >> >> >> response but that seems a bit irrelevant to topic metadata. FWIW I >> >> >> think the full broker-list is also irrelevant to topic metadata, but >> >> >> it is already there and in use. I think there is still room for an >> >> >> explicit ClusterMetadata request since there may be other >> >> >> cluster-level information that we may want to add over time (and that >> >> >> have nothing to do with topic metadata). >> >> >> >> >> >> On Tue, Mar 17, 2015 at 02:45:30PM +0200, Andrii Biletskyi wrote: >> >> >> > Jun, >> >> >> > >> >> >> > 101. Okay, if you say that such use case is important. I also think >> >> >> > using clientId for these purposes is fine - if we already have this >> >> >> field >> >> >> > as part of all Wire protocol messages, why not use that. >> >> >> > I will update KIP-4 page if nobody has other ideas (which may come >> up >> >> >> > during the call today). >> >> >> > >> >> >> > 102.1 Agree, I'll update the KIP accordingly. I think we can add >> new, >> >> >> > fine-grained error codes if some error code received in specific >> case >> >> >> > won't give enough context to return a descriptive error message for >> >> >> user. >> >> >> > >> >> >> > Look forward to discussing all outstanding issues in detail today >> >> during >> >> >> > the call. >> >> >> > >> >> >> > Thanks, >> >> >> > Andrii Biletskyi >> >> >> > >> >> >> > >> >> >> > >> >> >> > On Mon, Mar 16, 2015 at 10:59 PM, Jun Rao <j...@confluent.io> >> wrote: >> >> >> > >> >> >> > > 101. There may be a use case where you only want the topics to be >> >> >> created >> >> >> > > manually by admins. Currently, you can do that by disabling auto >> >> topic >> >> >> > > creation and issue topic creation from the TopicCommand. If we >> >> >> disable auto >> >> >> > > topic creation completely on the broker and don't have a way to >> >> >> distinguish >> >> >> > > between topic creation requests from the regular clients and the >> >> >> admin, we >> >> >> > > can't support manual topic creation any more. I was thinking that >> >> >> another >> >> >> > > way of distinguishing the clients making the topic creation >> requests >> >> >> is >> >> >> > > using clientId. For example, the admin tool can set it to >> something >> >> >> like >> >> >> > > admin and the broker can treat that clientId specially. >> >> >> > > >> >> >> > > Also, there is a related discussion in KAFKA-2020. Currently, we >> do >> >> >> the >> >> >> > > following in TopicMetadataResponse: >> >> >> > > >> >> >> > > 1. If leader is not available, we set the partition level error >> code >> >> >> to >> >> >> > > LeaderNotAvailable. >> >> >> > > 2. If a non-leader replica is not available, we take that replica >> >> out >> >> >> of >> >> >> > > the assigned replica list and isr in the response. As an >> indication >> >> >> for >> >> >> > > doing that, we set the partition level error code to >> >> >> ReplicaNotAvailable. >> >> >> > > >> >> >> > > This has a few problems. First, ReplicaNotAvailable probably >> >> >> shouldn't be >> >> >> > > an error, at least for the normal producer/consumer clients that >> >> just >> >> >> want >> >> >> > > to find out the leader. Second, it can happen that both the >> leader >> >> and >> >> >> > > another replica are not available at the same time. There is no >> >> error >> >> >> code >> >> >> > > to indicate both. Third, even if a replica is not available, it's >> >> >> still >> >> >> > > useful to return its replica id since some clients (e.g. admin >> tool) >> >> >> may >> >> >> > > still make use of it. >> >> >> > > >> >> >> > > One way to address this issue is to always return the replica id >> for >> >> >> > > leader, assigned replicas, and isr regardless of whether the >> >> >> corresponding >> >> >> > > broker is live or not. Since we also return the list of live >> >> brokers, >> >> >> the >> >> >> > > client can figure out whether a leader or a replica is live or >> not >> >> >> and act >> >> >> > > accordingly. This way, we don't need to set the partition level >> >> error >> >> >> code >> >> >> > > when the leader or a replica is not available. This doesn't >> change >> >> >> the wire >> >> >> > > protocol, but does change the semantics. Since we are evolving >> the >> >> >> protocol >> >> >> > > of TopicMetadataRequest here, we can potentially piggyback the >> >> change. >> >> >> > > >> >> >> > > 102.1 For those types of errors due to invalid input, shouldn't >> we >> >> >> just >> >> >> > > guard it at parameter validation time and throw >> >> >> InvalidArgumentException >> >> >> > > without even sending the request to the broker? >> >> >> > > >> >> >> > > Thanks, >> >> >> > > >> >> >> > > Jun >> >> >> > > >> >> >> > > >> >> >> > > On Mon, Mar 16, 2015 at 10:37 AM, Andrii Biletskyi < >> >> >> > > andrii.bilets...@stealth.ly> wrote: >> >> >> > > >> >> >> > > > Jun, >> >> >> > > > >> >> >> > > > Answering your questions: >> >> >> > > > >> >> >> > > > 101. If I understand you correctly, you are saying future >> producer >> >> >> > > versions >> >> >> > > > (which >> >> >> > > > will be ported to TMR_V1) won't be able to automatically create >> >> >> topic (if >> >> >> > > > we >> >> >> > > > unconditionally remove topic creation from there). But we need >> to >> >> >> this >> >> >> > > > preserve logic. >> >> >> > > > Ok, about your proposal: I'm not a big fan too, when it comes >> to >> >> >> > > > differentiating >> >> >> > > > clients directly in protocol schema. And also I'm not sure I >> >> >> understand >> >> >> > > at >> >> >> > > > all why >> >> >> > > > auto.create.topics.enable is a server side configuration. Can >> we >> >> >> > > deprecate >> >> >> > > > this setting >> >> >> > > > in future versions, add this setting to producer and based on >> that >> >> >> upon >> >> >> > > > receiving >> >> >> > > > UnknownTopic create topic explicitly by a separate producer >> call >> >> via >> >> >> > > > adminClient? >> >> >> > > > >> >> >> > > > 102.1. Hm, yes. It's because we want to support batching and at >> >> the >> >> >> same >> >> >> > > > time we >> >> >> > > > want to give descriptive error messages for clients. Since >> >> >> AdminClient >> >> >> > > > holds the context >> >> >> > > > to construct such messages (e.g. AdminClient layer can know >> that >> >> >> > > > InvalidArgumentsCode >> >> >> > > > means two cases: either invalid number - e.g. -1; or >> >> >> replication-factor >> >> >> > > was >> >> >> > > > provided while >> >> >> > > > partitions argument wasn't) - I wrapped responses in >> Exceptions. >> >> >> But I'm >> >> >> > > > open to any >> >> >> > > > other ideas, this was just initial version. >> >> >> > > > 102.2. Yes, I agree. I'll change that to probably some other >> dto. >> >> >> > > > >> >> >> > > > Thanks, >> >> >> > > > Andrii Biletskyi >> >> >> > > > >> >> >> > > > On Fri, Mar 13, 2015 at 7:16 PM, Jun Rao <j...@confluent.io> >> >> wrote: >> >> >> > > > >> >> >> > > > > Andrii, >> >> >> > > > > >> >> >> > > > > 101. That's what I was thinking too, but it may not be that >> >> >> simple. In >> >> >> > > > > TopicMetadataRequest_V1, >> >> >> > > > > we can let it not trigger auto topic creation. Then, in the >> >> >> producer >> >> >> > > > side, >> >> >> > > > > if it gets an UnknownTopicException, it can explicitly issue >> a >> >> >> > > > > createTopicRequest for auto topic creation. On the consumer >> >> side, >> >> >> it >> >> >> > > will >> >> >> > > > > never issue createTopicRequest. This works when auto topic >> >> >> creation is >> >> >> > > > > enabled on the broker side. However, I am not sure how things >> >> >> will work >> >> >> > > > > when auto topic creation is disabled on the broker side. In >> this >> >> >> case, >> >> >> > > we >> >> >> > > > > want to have a way to manually create a topic, potentially >> >> through >> >> >> > > admin >> >> >> > > > > commands. However, then we need a way to distinguish >> >> >> createTopicRequest >> >> >> > > > > issued from the producer clients and the admin tools. May be >> we >> >> >> can >> >> >> > > add a >> >> >> > > > > new field in createTopicRequest and set it differently in the >> >> >> producer >> >> >> > > > > client and the admin client. However, I am not sure if that's >> >> the >> >> >> best >> >> >> > > > > approach. >> >> >> > > > > >> >> >> > > > > 2. Yes, refactoring existing requests is a non-trivial >> amount of >> >> >> work. >> >> >> > > I >> >> >> > > > > posted some comments in KAFKA-1927. We will probably have to >> fix >> >> >> > > > KAFKA-1927 >> >> >> > > > > first, before adding the new logic in KAFKA-1694. Otherwise, >> the >> >> >> > > changes >> >> >> > > > > will be too big. >> >> >> > > > > >> >> >> > > > > 102. About the AdminClient: >> >> >> > > > > 102.1. It's a bit weird that we return exception in the api. >> It >> >> >> seems >> >> >> > > > that >> >> >> > > > > we should either return error code or throw an exception when >> >> >> getting >> >> >> > > the >> >> >> > > > > response state. >> >> >> > > > > 102.2. We probably shouldn't explicitly use the request >> object >> >> in >> >> >> the >> >> >> > > > api. >> >> >> > > > > Not every request evolution requires an api change. >> >> >> > > > > >> >> >> > > > > Thanks, >> >> >> > > > > >> >> >> > > > > Jun >> >> >> > > > > >> >> >> > > > > >> >> >> > > > > On Fri, Mar 13, 2015 at 4:08 AM, Andrii Biletskyi < >> >> >> > > > > andrii.bilets...@stealth.ly> wrote: >> >> >> > > > > >> >> >> > > > > > Jun, >> >> >> > > > > > >> >> >> > > > > > Thanks for you comments. Answers inline: >> >> >> > > > > > >> >> >> > > > > > 100. There are a few fields such as ReplicaAssignment, >> >> >> > > > > > > ReassignPartitionRequest, >> >> >> > > > > > > and PartitionsSerialized that are represented as a >> string, >> >> but >> >> >> > > > contain >> >> >> > > > > > > composite structures in json. Could we flatten them out >> >> >> directly in >> >> >> > > > the >> >> >> > > > > > > protocol definition as arrays/records? >> >> >> > > > > > >> >> >> > > > > > >> >> >> > > > > > Yes, now with Admin Client this looks a bit weird. My >> initial >> >> >> > > > motivation >> >> >> > > > > > was: >> >> >> > > > > > ReassignPartitionCommand accepts input in json, we want to >> >> >> remain >> >> >> > > > tools' >> >> >> > > > > > interfaces unchanged, where possible. >> >> >> > > > > > If we port it to deserialized format, in CLI (/tools >> project) >> >> >> we will >> >> >> > > > > have >> >> >> > > > > > to add some >> >> >> > > > > > json library since /tools is written in java and we'll >> need to >> >> >> > > > > deserialize >> >> >> > > > > > json file >> >> >> > > > > > provided by a user. Can we quickly agree on what this >> library >> >> >> should >> >> >> > > be >> >> >> > > > > > (Jackson, GSON, whatever)? >> >> >> > > > > > >> >> >> > > > > > 101. Does TopicMetadataRequest v1 still trigger auto topic >> >> >> creation? >> >> >> > > > This >> >> >> > > > > > > will be a bit weird now that we have a separate topic >> >> >> creation api. >> >> >> > > > > Have >> >> >> > > > > > > you thought about how the new createTopicRequest and >> >> >> > > > > TopicMetadataRequest >> >> >> > > > > > > v1 will be used in the producer/consumer client, in >> addition >> >> >> to >> >> >> > > admin >> >> >> > > > > > > tools? For example, ideally, we don't want >> >> >> TopicMetadataRequest >> >> >> > > from >> >> >> > > > > the >> >> >> > > > > > > consumer to trigger auto topic creation. >> >> >> > > > > > >> >> >> > > > > > >> >> >> > > > > > I agree, this strange logic should be fixed. I'm not >> confident >> >> >> in >> >> >> > > this >> >> >> > > > > > Kafka part so >> >> >> > > > > > correct me if I'm wrong, but it doesn't look like a hard >> thing >> >> >> to >> >> >> > > do, I >> >> >> > > > > > think we can >> >> >> > > > > > leverage AdminClient for that in Producer and >> unconditionally >> >> >> remove >> >> >> > > > > topic >> >> >> > > > > > creation from the TopicMetadataRequest_V1. >> >> >> > > > > > >> >> >> > > > > > 2. I think Jay meant getting rid of scala classes >> >> >> > > > > > > like HeartbeatRequestAndHeader and >> >> >> HeartbeatResponseAndHeader. We >> >> >> > > did >> >> >> > > > > > that >> >> >> > > > > > > as a stop-gap thing when adding the new requests for the >> >> >> consumers. >> >> >> > > > > > > However, the long term plan is to get rid of all those >> and >> >> >> just >> >> >> > > reuse >> >> >> > > > > the >> >> >> > > > > > > java request/response in the client. Since this KIP >> proposes >> >> >> to >> >> >> > > add a >> >> >> > > > > > > significant number of new requests, perhaps we should >> bite >> >> the >> >> >> > > bullet >> >> >> > > > > to >> >> >> > > > > > > clean up the existing scala requests first before adding >> new >> >> >> ones? >> >> >> > > > > > > >> >> >> > > > > > >> >> >> > > > > > Yes, looks like I misunderstood the point of >> >> >> ...RequestAndHeader. >> >> >> > > > Okay, I >> >> >> > > > > > will >> >> >> > > > > > rework that. The only thing is that I don't see any example >> >> how >> >> >> it >> >> >> > > was >> >> >> > > > > done >> >> >> > > > > > for at >> >> >> > > > > > least one existing protocol message. Thus, as I >> understand, I >> >> >> have to >> >> >> > > > > think >> >> >> > > > > > how we >> >> >> > > > > > are going to do it. >> >> >> > > > > > Re porting all existing RQ/RP in this patch. Sounds >> >> reasonable, >> >> >> but >> >> >> > > if >> >> >> > > > > it's >> >> >> > > > > > an *obligatory* >> >> >> > > > > > requirement to have Admin KIP done, I'm afraid this can be >> a >> >> >> serious >> >> >> > > > > > blocker for us. >> >> >> > > > > > There are 13 protocol messages and all that would require >> not >> >> >> only >> >> >> > > unit >> >> >> > > > > > tests but quite >> >> >> > > > > > intensive manual testing, no? I'm afraid I'm not the right >> guy >> >> >> to >> >> >> > > cover >> >> >> > > > > > pretty much all >> >> >> > > > > > Kafka core internals :). Let me know your thoughts on this >> >> >> item. Btw >> >> >> > > > > there >> >> >> > > > > > is a ticket to >> >> >> > > > > > follow-up this issue ( >> >> >> > > https://issues.apache.org/jira/browse/KAFKA-2006 >> >> >> > > > ). >> >> >> > > > > > >> >> >> > > > > > Thanks, >> >> >> > > > > > Andrii Biletskyi >> >> >> > > > > > >> >> >> > > > > > >> >> >> > > > > > On Fri, Mar 13, 2015 at 6:40 AM, Jun Rao <j...@confluent.io >> > >> >> >> wrote: >> >> >> > > > > > >> >> >> > > > > > > Andrii, >> >> >> > > > > > > >> >> >> > > > > > > >> >> >> > > > > > > A few more comments. >> >> >> > > > > > > >> >> >> > > > > > > 100. There are a few fields such as ReplicaAssignment, >> >> >> > > > > > > ReassignPartitionRequest, >> >> >> > > > > > > and PartitionsSerialized that are represented as a >> string, >> >> but >> >> >> > > > contain >> >> >> > > > > > > composite structures in json. Could we flatten them out >> >> >> directly in >> >> >> > > > the >> >> >> > > > > > > protocol definition as arrays/records? >> >> >> > > > > > > >> >> >> > > > > > > 101. Does TopicMetadataRequest v1 still trigger auto >> topic >> >> >> > > creation? >> >> >> > > > > This >> >> >> > > > > > > will be a bit weird now that we have a separate topic >> >> >> creation api. >> >> >> > > > > Have >> >> >> > > > > > > you thought about how the new createTopicRequest and >> >> >> > > > > TopicMetadataRequest >> >> >> > > > > > > v1 will be used in the producer/consumer client, in >> addition >> >> >> to >> >> >> > > admin >> >> >> > > > > > > tools? For example, ideally, we don't want >> >> >> TopicMetadataRequest >> >> >> > > from >> >> >> > > > > the >> >> >> > > > > > > consumer to trigger auto topic creation. >> >> >> > > > > > > >> >> >> > > > > > > 2. I think Jay meant getting rid of scala classes >> >> >> > > > > > > like HeartbeatRequestAndHeader and >> >> >> HeartbeatResponseAndHeader. We >> >> >> > > did >> >> >> > > > > > that >> >> >> > > > > > > as a stop-gap thing when adding the new requests for the >> >> >> consumers. >> >> >> > > > > > > However, the long term plan is to get rid of all those >> and >> >> >> just >> >> >> > > reuse >> >> >> > > > > the >> >> >> > > > > > > java request/response in the client. Since this KIP >> proposes >> >> >> to >> >> >> > > add a >> >> >> > > > > > > significant number of new requests, perhaps we should >> bite >> >> the >> >> >> > > bullet >> >> >> > > > > to >> >> >> > > > > > > clean up the existing scala requests first before adding >> new >> >> >> ones? >> >> >> > > > > > > >> >> >> > > > > > > Thanks, >> >> >> > > > > > > >> >> >> > > > > > > Jun >> >> >> > > > > > > >> >> >> > > > > > > >> >> >> > > > > > > >> >> >> > > > > > > On Thu, Mar 12, 2015 at 3:37 PM, Andrii Biletskyi < >> >> >> > > > > > > andrii.bilets...@stealth.ly> wrote: >> >> >> > > > > > > >> >> >> > > > > > > > Hi, >> >> >> > > > > > > > >> >> >> > > > > > > > As said above - I list again all comments from this >> thread >> >> >> so we >> >> >> > > > > > > > can see what's left and finalize all pending issues. >> >> >> > > > > > > > >> >> >> > > > > > > > Comments from Jay: >> >> >> > > > > > > > 1. This is much needed functionality, but there are a >> lot >> >> >> of the >> >> >> > > so >> >> >> > > > > > let's >> >> >> > > > > > > > really think these protocols through. We really want to >> >> end >> >> >> up >> >> >> > > > with a >> >> >> > > > > > set >> >> >> > > > > > > > of well thought-out, orthoganol apis. For this reason I >> >> >> think it >> >> >> > > is >> >> >> > > > > > > really >> >> >> > > > > > > > important to think through the end state even if that >> >> >> includes >> >> >> > > APIs >> >> >> > > > > we >> >> >> > > > > > > > won't implement in the first phase. >> >> >> > > > > > > > >> >> >> > > > > > > > A: Definitely behind this. Would appreciate if there >> are >> >> >> concrete >> >> >> > > > > > > comments >> >> >> > > > > > > > how this can be improved. >> >> >> > > > > > > > >> >> >> > > > > > > > 2. Let's please please please wait until we have >> switched >> >> >> the >> >> >> > > > server >> >> >> > > > > > over >> >> >> > > > > > > > to the new java protocol definitions. If we add upteen >> >> more >> >> >> ad >> >> >> > > hoc >> >> >> > > > > > scala >> >> >> > > > > > > > objects that is just generating more work for the >> >> >> conversion we >> >> >> > > > know >> >> >> > > > > we >> >> >> > > > > > > > have to do. >> >> >> > > > > > > > >> >> >> > > > > > > > A: Fixed in the latest patch - removed scala protocol >> >> >> classes. >> >> >> > > > > > > > >> >> >> > > > > > > > 3. This proposal introduces a new type of optional >> >> >> parameter. >> >> >> > > This >> >> >> > > > is >> >> >> > > > > > > > inconsistent with everything else in the protocol >> where we >> >> >> use -1 >> >> >> > > > or >> >> >> > > > > > some >> >> >> > > > > > > > other marker value. You could argue either way but >> let's >> >> >> stick >> >> >> > > with >> >> >> > > > > > that >> >> >> > > > > > > > for consistency. For clients that implemented the >> protocol >> >> >> in a >> >> >> > > > > better >> >> >> > > > > > > way >> >> >> > > > > > > > than our scala code these basic primitives are hard to >> >> >> change. >> >> >> > > > > > > > >> >> >> > > > > > > > A: Fixed in the latest patch - removed MaybeOf type and >> >> >> changed >> >> >> > > > > > protocol >> >> >> > > > > > > > accordingly. >> >> >> > > > > > > > >> >> >> > > > > > > > 4. ClusterMetadata: This seems to duplicate >> >> >> TopicMetadataRequest >> >> >> > > > > which >> >> >> > > > > > > has >> >> >> > > > > > > > brokers, topics, and partitions. I think we should >> rename >> >> >> that >> >> >> > > > > request >> >> >> > > > > > > > ClusterMetadataRequest (or just MetadataRequest) and >> >> >> include the >> >> >> > > id >> >> >> > > > > of >> >> >> > > > > > > the >> >> >> > > > > > > > controller. Or are there other things we could add >> here? >> >> >> > > > > > > > >> >> >> > > > > > > > A: I agree. Updated the KIP. Let's extends >> TopicMetadata >> >> to >> >> >> > > > version 2 >> >> >> > > > > > and >> >> >> > > > > > > > include controller. >> >> >> > > > > > > > >> >> >> > > > > > > > 5. We have a tendency to try to make a lot of requests >> >> that >> >> >> can >> >> >> > > > only >> >> >> > > > > go >> >> >> > > > > > > to >> >> >> > > > > > > > particular nodes. This adds a lot of burden for client >> >> >> > > > > implementations >> >> >> > > > > > > (it >> >> >> > > > > > > > sounds easy but each discovery can fail in many parts >> so >> >> it >> >> >> ends >> >> >> > > up >> >> >> > > > > > > being a >> >> >> > > > > > > > full state machine to do right). I think we should >> >> consider >> >> >> > > making >> >> >> > > > > > admin >> >> >> > > > > > > > commands and ideally as many of the other apis as >> possible >> >> >> > > > available >> >> >> > > > > on >> >> >> > > > > > > all >> >> >> > > > > > > > brokers and just redirect to the controller on the >> broker >> >> >> side. >> >> >> > > > > Perhaps >> >> >> > > > > > > > there would be a general way to encapsulate this >> >> re-routing >> >> >> > > > behavior. >> >> >> > > > > > > > >> >> >> > > > > > > > A: It's a very interesting idea, but seems there are >> some >> >> >> > > concerns >> >> >> > > > > > about >> >> >> > > > > > > > this >> >> >> > > > > > > > feature (like performance considerations, how this will >> >> >> > > complicate >> >> >> > > > > > server >> >> >> > > > > > > > etc). >> >> >> > > > > > > > I believe this shouldn't be a blocker. If this feature >> is >> >> >> > > > implemented >> >> >> > > > > > at >> >> >> > > > > > > > some >> >> >> > > > > > > > point it won't affect Admin changes - at least no >> changes >> >> to >> >> >> > > public >> >> >> > > > > API >> >> >> > > > > > > > will be required. >> >> >> > > > > > > > >> >> >> > > > > > > > 6. We should probably normalize the key value pairs >> used >> >> for >> >> >> > > > configs >> >> >> > > > > > > rather >> >> >> > > > > > > > than embedding a new formatting. So two strings rather >> >> than >> >> >> one >> >> >> > > > with >> >> >> > > > > an >> >> >> > > > > > > > internal equals sign. >> >> >> > > > > > > > >> >> >> > > > > > > > A: Fixed in the latest patch - normalized configs and >> >> >> changed >> >> >> > > > > protocol >> >> >> > > > > > > > accordingly. >> >> >> > > > > > > > >> >> >> > > > > > > > 7. Is the postcondition of these APIs that the command >> has >> >> >> begun >> >> >> > > or >> >> >> > > > > > that >> >> >> > > > > > > > the command has been completed? It is a lot more >> usable if >> >> >> the >> >> >> > > > > command >> >> >> > > > > > > has >> >> >> > > > > > > > been completed so you know that if you create a topic >> and >> >> >> then >> >> >> > > > > publish >> >> >> > > > > > to >> >> >> > > > > > > > it you won't get an exception about there being no such >> >> >> topic. >> >> >> > > > > > > > >> >> >> > > > > > > > A: For long running requests (like reassign >> partitions) - >> >> >> the >> >> >> > > post >> >> >> > > > > > > > condition is >> >> >> > > > > > > > command has begun - so we don't block the client. In >> case >> >> >> of your >> >> >> > > > > > > example - >> >> >> > > > > > > > topic commands, this will be refactored and topic >> commands >> >> >> will >> >> >> > > be >> >> >> > > > > > > executed >> >> >> > > > > > > > immediately, since the Controller will serve Admin >> >> requests >> >> >> > > > > > > > (follow-up ticket KAFKA-1777). >> >> >> > > > > > > > >> >> >> > > > > > > > 8. Describe topic and list topics duplicate a lot of >> stuff >> >> >> in the >> >> >> > > > > > > metadata >> >> >> > > > > > > > request. Is there a reason to give back topics marked >> for >> >> >> > > > deletion? I >> >> >> > > > > > > feel >> >> >> > > > > > > > like if we just make the post-condition of the delete >> >> >> command be >> >> >> > > > that >> >> >> > > > > > the >> >> >> > > > > > > > topic is deleted that will get rid of the need for this >> >> >> right? >> >> >> > > And >> >> >> > > > it >> >> >> > > > > > > will >> >> >> > > > > > > > be much more intuitive. >> >> >> > > > > > > > >> >> >> > > > > > > > A: Fixed in the latest patch - removed topics marked >> for >> >> >> deletion >> >> >> > > > in >> >> >> > > > > > > > ListTopicsRequest. >> >> >> > > > > > > > >> >> >> > > > > > > > 9. Should we consider batching these requests? We have >> >> >> generally >> >> >> > > > > tried >> >> >> > > > > > to >> >> >> > > > > > > > allow multiple operations to be batched. My suspicion >> is >> >> >> that >> >> >> > > > without >> >> >> > > > > > > this >> >> >> > > > > > > > we will get a lot of code that does something like >> >> >> > > > > > > > for(topic: adminClient.listTopics()) >> >> >> > > > > > > > adminClient.describeTopic(topic) >> >> >> > > > > > > > this code will work great when you test on 5 topics but >> >> not >> >> >> do as >> >> >> > > > > well >> >> >> > > > > > if >> >> >> > > > > > > > you have 50k. >> >> >> > > > > > > > >> >> >> > > > > > > > A: Updated the KIP - please check "Topic Admin Schema" >> >> >> section. >> >> >> > > > > > > > >> >> >> > > > > > > > 10. I think we should also discuss how we want to >> expose a >> >> >> > > > > programmatic >> >> >> > > > > > > JVM >> >> >> > > > > > > > client api for these operations. Currently people rely >> on >> >> >> > > > AdminUtils >> >> >> > > > > > > which >> >> >> > > > > > > > is totally sketchy. I think we probably need another >> >> client >> >> >> under >> >> >> > > > > > > clients/ >> >> >> > > > > > > > that exposes administrative functionality. We will need >> >> >> this just >> >> >> > > > to >> >> >> > > > > > > > properly test the new apis, I suspect. We should figure >> >> out >> >> >> that >> >> >> > > > API. >> >> >> > > > > > > > >> >> >> > > > > > > > A: Updated the KIP - please check "Admin Client" >> section >> >> >> with an >> >> >> > > > > > initial >> >> >> > > > > > > > API proposal. >> >> >> > > > > > > > >> >> >> > > > > > > > 11. The other information that would be really useful >> to >> >> get >> >> >> > > would >> >> >> > > > be >> >> >> > > > > > > > information about partitions--how much data is in the >> >> >> partition, >> >> >> > > > what >> >> >> > > > > > are >> >> >> > > > > > > > the segment offsets, what is the log-end offset (i.e. >> last >> >> >> > > offset), >> >> >> > > > > > what >> >> >> > > > > > > is >> >> >> > > > > > > > the compaction point, etc. I think that done right this >> >> >> would be >> >> >> > > > the >> >> >> > > > > > > > successor to the very awkward OffsetRequest we have >> today. >> >> >> > > > > > > > >> >> >> > > > > > > > A: I removed ConsumerGroupOffsetsRequest in the latest >> >> >> patch. I >> >> >> > > > > believe >> >> >> > > > > > > > this should >> >> >> > > > > > > > be resolved in a separate KIP / jira ticket. >> >> >> > > > > > > > >> >> >> > > > > > > > 12. Generally we can do good error handling without >> >> needing >> >> >> > > custom >> >> >> > > > > > > > server-side >> >> >> > > > > > > > messages. I.e. generally the client has the context to >> >> know >> >> >> that >> >> >> > > if >> >> >> > > > > it >> >> >> > > > > > > got >> >> >> > > > > > > > an error that the topic doesn't exist to say "Topic X >> >> >> doesn't >> >> >> > > > exist" >> >> >> > > > > > > rather >> >> >> > > > > > > > than "error code 14" (or whatever). Maybe there are >> >> specific >> >> >> > > cases >> >> >> > > > > > where >> >> >> > > > > > > > this is hard? If we want to add server-side error >> messages >> >> >> we >> >> >> > > > really >> >> >> > > > > do >> >> >> > > > > > > > need to do this in a consistent way across the >> protocol. >> >> >> > > > > > > > >> >> >> > > > > > > > A: Updated the KIP - please check "Protocol Errors" >> >> >> section. I >> >> >> > > > added >> >> >> > > > > > the >> >> >> > > > > > > > comprehensive, fine-grained list of error codes. >> >> >> > > > > > > > >> >> >> > > > > > > > Comments from Guozhang: >> >> >> > > > > > > > 13. Describe topic request: it would be great to go >> beyond >> >> >> just >> >> >> > > > > > batching >> >> >> > > > > > > on >> >> >> > > > > > > > topic name regex for this request. For example, a very >> >> >> common use >> >> >> > > > > case >> >> >> > > > > > of >> >> >> > > > > > > > the topic command is to list all topics whose config >> A's >> >> >> value is >> >> >> > > > B. >> >> >> > > > > > With >> >> >> > > > > > > > topic name regex then we have to first retrieve __all__ >> >> >> topics's >> >> >> > > > > > > > description info and then filter at the client end, >> which >> >> >> will >> >> >> > > be a >> >> >> > > > > > huge >> >> >> > > > > > > > burden on ZK. >> >> >> > > > > > > > AND >> >> >> > > > > > > > 14. Config K-Vs in create topic: this is related to the >> >> >> previous >> >> >> > > > > point; >> >> >> > > > > > > > maybe we can add another metadata K-V or just a >> metadata >> >> >> string >> >> >> > > > along >> >> >> > > > > > > side >> >> >> > > > > > > > with config K-V in create topic like we did for offset >> >> >> commit >> >> >> > > > > request. >> >> >> > > > > > > This >> >> >> > > > > > > > field can be quite useful in storing information like >> >> >> "owner" of >> >> >> > > > the >> >> >> > > > > > > topic >> >> >> > > > > > > > who issue the create command, etc, which is quite >> >> important >> >> >> for a >> >> >> > > > > > > > multi-tenant setting. Then in the describe topic >> request >> >> we >> >> >> can >> >> >> > > > also >> >> >> > > > > > > batch >> >> >> > > > > > > > on regex of the metadata field. >> >> >> > > > > > > > >> >> >> > > > > > > > A: As discussed it is very interesting but can be >> >> >> implemented >> >> >> > > later >> >> >> > > > > > after >> >> >> > > > > > > > we have some basic functionality there. >> >> >> > > > > > > > >> >> >> > > > > > > > 15. Today all the admin operations are async in the >> sense >> >> >> that >> >> >> > > > > command >> >> >> > > > > > > will >> >> >> > > > > > > > return once it is written in ZK, and that is why we >> need >> >> >> extra >> >> >> > > > > > > verification >> >> >> > > > > > > > like testUtil.waitForTopicCreated() / verify partition >> >> >> > > reassignment >> >> >> > > > > > > > request, etc. With admin requests we could add a flag >> to >> >> >> enable / >> >> >> > > > > > disable >> >> >> > > > > > > > synchronous requests; when it is turned on, the >> response >> >> >> will not >> >> >> > > > > > return >> >> >> > > > > > > > until the request has been completed. And for async >> >> >> requests we >> >> >> > > can >> >> >> > > > > > add a >> >> >> > > > > > > > "token" field in the response, and then only need a >> >> general >> >> >> > > "admin >> >> >> > > > > > > > verification request" with the given token to check if >> the >> >> >> async >> >> >> > > > > > request >> >> >> > > > > > > > has been completed. >> >> >> > > > > > > > >> >> >> > > > > > > > A: I see your point. My idea was to provide specific >> >> >> > > > Verify...Request >> >> >> > > > > > per >> >> >> > > > > > > > each >> >> >> > > > > > > > long running request, where needed. We can do it the >> way >> >> you >> >> >> > > > suggest. >> >> >> > > > > > The >> >> >> > > > > > > > only >> >> >> > > > > > > > concern is that introducing a token we again will make >> >> >> schema >> >> >> > > > > > "dynamic". >> >> >> > > > > > > We >> >> >> > > > > > > > wanted >> >> >> > > > > > > > to do similar thing introducing single AdminRequest for >> >> all >> >> >> topic >> >> >> > > > > > > commands >> >> >> > > > > > > > but rejected >> >> >> > > > > > > > this idea because we wanted to have schema defined. So >> >> this >> >> >> is >> >> >> > > > more a >> >> >> > > > > > > > choice between: >> >> >> > > > > > > > a) have fixed schema but introduce each time new >> >> >> Verify...Request >> >> >> > > > for >> >> >> > > > > > > > long-running requests >> >> >> > > > > > > > b) use one request for verification but generalize it >> with >> >> >> token >> >> >> > > > > > > > I'm fine with whatever decision community come to. Just >> >> let >> >> >> me >> >> >> > > know >> >> >> > > > > > your >> >> >> > > > > > > > thoughts. >> >> >> > > > > > > > >> >> >> > > > > > > > Comment from Gwen: >> >> >> > > > > > > > 16. Specifically for ownership, I think the plan is to >> add >> >> >> ACL >> >> >> > > (it >> >> >> > > > > > sounds >> >> >> > > > > > > > like you are describing ACL) via an external system >> >> (Argus, >> >> >> > > > Sentry). >> >> >> > > > > > > > I remember KIP-11 described this, but I can't find the >> KIP >> >> >> any >> >> >> > > > > longer. >> >> >> > > > > > > > >> >> >> > > > > > > > A: Okay, no problem. Not sure though how we are going >> to >> >> >> handle >> >> >> > > it. >> >> >> > > > > > Wait >> >> >> > > > > > > > which KIP >> >> >> > > > > > > > will be committed first and include changes to >> >> >> TopicMetadata from >> >> >> > > > the >> >> >> > > > > > > later >> >> >> > > > > > > > one? >> >> >> > > > > > > > Anyway, I added this note to "Open Questions" section >> so >> >> we >> >> >> don't >> >> >> > > > > miss >> >> >> > > > > > > this >> >> >> > > > > > > > piece. >> >> >> > > > > > > > >> >> >> > > > > > > > Thanks, >> >> >> > > > > > > > Andrii Biletskyi >> >> >> > > > > > > > >> >> >> > > > > > > > On Fri, Mar 13, 2015 at 12:34 AM, Andrii Biletskyi < >> >> >> > > > > > > > andrii.bilets...@stealth.ly> wrote: >> >> >> > > > > > > > >> >> >> > > > > > > > > Hi all, >> >> >> > > > > > > > > >> >> >> > > > > > > > > Today I uploaded the patch that covers some of the >> >> >> discussed >> >> >> > > and >> >> >> > > > > > agreed >> >> >> > > > > > > > > items: >> >> >> > > > > > > > > - removed MaybeOf optional type >> >> >> > > > > > > > > - switched to java protocol definitions >> >> >> > > > > > > > > - simplified messages (normalized configs, removed >> topic >> >> >> marked >> >> >> > > > for >> >> >> > > > > > > > > deletion) >> >> >> > > > > > > > > >> >> >> > > > > > > > > I also updated the KIP-4 with respective changes and >> >> >> wrote down >> >> >> > > > my >> >> >> > > > > > > > > proposal for >> >> >> > > > > > > > > pending items: >> >> >> > > > > > > > > - Batch Admin Operations -> updated Wire Protocol >> schema >> >> >> > > proposal >> >> >> > > > > > > > > - Remove ClusterMetadata -> changed to extend >> >> >> > > > TopicMetadataRequest >> >> >> > > > > > > > > - Admin Client -> updated my initial proposal to >> reflect >> >> >> > > batching >> >> >> > > > > > > > > - Error codes -> proposed fine-grained error code >> >> instead >> >> >> of >> >> >> > > > > > > > > AdminRequestFailed >> >> >> > > > > > > > > >> >> >> > > > > > > > > I will also send a separate email to cover all >> comments >> >> >> from >> >> >> > > this >> >> >> > > > > > > thread. >> >> >> > > > > > > > > >> >> >> > > > > > > > > Thanks, >> >> >> > > > > > > > > Andrii Biletskyi >> >> >> > > > > > > > > >> >> >> > > > > > > > > >> >> >> > > > > > > > > On Thu, Mar 12, 2015 at 9:26 PM, Gwen Shapira < >> >> >> > > > > gshap...@cloudera.com >> >> >> > > > > > > >> >> >> > > > > > > > > wrote: >> >> >> > > > > > > > > >> >> >> > > > > > > > >> Found KIP-11 ( >> >> >> > > > > > > > >> >> >> >> > > > > > > > >> >> >> > > > > > > >> >> >> > > > > > >> >> >> > > > > >> >> >> > > > >> >> >> > > >> >> >> >> >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface >> >> >> > > > > > > > >> ) >> >> >> > > > > > > > >> It actually specifies changes to the Metadata >> protocol, >> >> >> so >> >> >> > > > making >> >> >> > > > > > sure >> >> >> > > > > > > > >> both KIPs are consistent in this regard will be >> good. >> >> >> > > > > > > > >> >> >> >> > > > > > > > >> On Thu, Mar 12, 2015 at 12:21 PM, Gwen Shapira < >> >> >> > > > > > gshap...@cloudera.com >> >> >> > > > > > > > >> >> >> > > > > > > > >> wrote: >> >> >> > > > > > > > >> > Specifically for ownership, I think the plan is to >> >> add >> >> >> ACL >> >> >> > > (it >> >> >> > > > > > > sounds >> >> >> > > > > > > > >> > like you are describing ACL) via an external >> system >> >> >> (Argus, >> >> >> > > > > > Sentry). >> >> >> > > > > > > > >> > I remember KIP-11 described this, but I can't find >> >> the >> >> >> KIP >> >> >> > > any >> >> >> > > > > > > longer. >> >> >> > > > > > > > >> > >> >> >> > > > > > > > >> > Regardless, I think KIP-4 focuses on getting >> >> >> information >> >> >> > > that >> >> >> > > > > > > already >> >> >> > > > > > > > >> > exists from Kafka brokers, not on adding >> information >> >> >> that >> >> >> > > > > perhaps >> >> >> > > > > > > > >> > should exist but doesn't yet? >> >> >> > > > > > > > >> > >> >> >> > > > > > > > >> > Gwen >> >> >> > > > > > > > >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > >> > On Thu, Mar 12, 2015 at 6:37 AM, Guozhang Wang < >> >> >> > > > > > wangg...@gmail.com> >> >> >> > > > > > > > >> wrote: >> >> >> > > > > > > > >> >> Folks, >> >> >> > > > > > > > >> >> >> >> >> > > > > > > > >> >> Just want to elaborate a bit more on the >> >> create-topic >> >> >> > > > metadata >> >> >> > > > > > and >> >> >> > > > > > > > >> batching >> >> >> > > > > > > > >> >> describe-topic based on config / metadata in my >> >> >> previous >> >> >> > > > email >> >> >> > > > > as >> >> >> > > > > > > we >> >> >> > > > > > > > >> work >> >> >> > > > > > > > >> >> on KAFKA-1694. The main motivation is to have >> some >> >> >> sort of >> >> >> > > > > topic >> >> >> > > > > > > > >> management >> >> >> > > > > > > > >> >> mechanisms, which I think is quite important in a >> >> >> > > > multi-tenant >> >> >> > > > > / >> >> >> > > > > > > > cloud >> >> >> > > > > > > > >> >> architecture: today anyone can create topics in a >> >> >> shared >> >> >> > > > Kafka >> >> >> > > > > > > > >> cluster, but >> >> >> > > > > > > > >> >> there is no concept or "ownership" of topics that >> >> are >> >> >> > > created >> >> >> > > > > by >> >> >> > > > > > > > >> different >> >> >> > > > > > > > >> >> users. For example, at LinkedIn we basically >> >> >> distinguish >> >> >> > > > topic >> >> >> > > > > > > owners >> >> >> > > > > > > > >> via >> >> >> > > > > > > > >> >> some casual topic name prefix, which is a bit >> >> awkward >> >> >> and >> >> >> > > > does >> >> >> > > > > > not >> >> >> > > > > > > > fly >> >> >> > > > > > > > >> as >> >> >> > > > > > > > >> >> we scale our customers. It would be great to use >> >> >> > > > > describe-topics >> >> >> > > > > > > such >> >> >> > > > > > > > >> as: >> >> >> > > > > > > > >> >> >> >> >> > > > > > > > >> >> Describe all topics that is created by me. >> >> >> > > > > > > > >> >> >> >> >> > > > > > > > >> >> Describe all topics whose retention time is >> >> overriden >> >> >> to X. >> >> >> > > > > > > > >> >> >> >> >> > > > > > > > >> >> Describe all topics whose writable group include >> >> user >> >> >> Y >> >> >> > > (this >> >> >> > > > > is >> >> >> > > > > > > > >> related to >> >> >> > > > > > > > >> >> authorization), etc.. >> >> >> > > > > > > > >> >> >> >> >> > > > > > > > >> >> One possible way to achieve this is to add a >> >> metadata >> >> >> file >> >> >> > > in >> >> >> > > > > the >> >> >> > > > > > > > >> >> create-topic request, whose value will also be >> >> >> written ZK >> >> >> > > as >> >> >> > > > we >> >> >> > > > > > > > create >> >> >> > > > > > > > >> the >> >> >> > > > > > > > >> >> topic; then describe-topics can choose to batch >> >> topics >> >> >> > > based >> >> >> > > > on >> >> >> > > > > > 1) >> >> >> > > > > > > > name >> >> >> > > > > > > > >> >> regex, 2) config K-V matching, 3) metadata regex, >> >> etc. >> >> >> > > > > > > > >> >> >> >> >> > > > > > > > >> >> Thoughts? >> >> >> > > > > > > > >> >> >> >> >> > > > > > > > >> >> Guozhang >> >> >> > > > > > > > >> >> >> >> >> > > > > > > > >> >> On Thu, Mar 5, 2015 at 4:37 PM, Guozhang Wang < >> >> >> > > > > > wangg...@gmail.com> >> >> >> > > > > > > > >> wrote: >> >> >> > > > > > > > >> >> >> >> >> > > > > > > > >> >>> Thanks for the updated wiki. A few comments >> below: >> >> >> > > > > > > > >> >>> >> >> >> > > > > > > > >> >>> 1. Error description in response: I think if >> some >> >> >> > > errorCode >> >> >> > > > > > could >> >> >> > > > > > > > >> indicate >> >> >> > > > > > > > >> >>> several different error cases then we should >> really >> >> >> change >> >> >> > > > it >> >> >> > > > > to >> >> >> > > > > > > > >> multiple >> >> >> > > > > > > > >> >>> codes. In general the errorCode itself would be >> >> >> precise >> >> >> > > and >> >> >> > > > > > > > >> sufficient for >> >> >> > > > > > > > >> >>> describing the server side errors. >> >> >> > > > > > > > >> >>> >> >> >> > > > > > > > >> >>> 2. Describe topic request: it would be great to >> go >> >> >> beyond >> >> >> > > > just >> >> >> > > > > > > > >> batching on >> >> >> > > > > > > > >> >>> topic name regex for this request. For example, >> a >> >> >> very >> >> >> > > > common >> >> >> > > > > > use >> >> >> > > > > > > > >> case of >> >> >> > > > > > > > >> >>> the topic command is to list all topics whose >> >> config >> >> >> A's >> >> >> > > > value >> >> >> > > > > > is >> >> >> > > > > > > B. >> >> >> > > > > > > > >> With >> >> >> > > > > > > > >> >>> topic name regex then we have to first retrieve >> >> >> __all__ >> >> >> > > > > topics's >> >> >> > > > > > > > >> >>> description info and then filter at the client >> end, >> >> >> which >> >> >> > > > will >> >> >> > > > > > be >> >> >> > > > > > > a >> >> >> > > > > > > > >> huge >> >> >> > > > > > > > >> >>> burden on ZK. >> >> >> > > > > > > > >> >>> >> >> >> > > > > > > > >> >>> 3. Config K-Vs in create topic: this is related >> to >> >> >> the >> >> >> > > > > previous >> >> >> > > > > > > > point; >> >> >> > > > > > > > >> >>> maybe we can add another metadata K-V or just a >> >> >> metadata >> >> >> > > > > string >> >> >> > > > > > > > along >> >> >> > > > > > > > >> side >> >> >> > > > > > > > >> >>> with config K-V in create topic like we did for >> >> >> offset >> >> >> > > > commit >> >> >> > > > > > > > >> request. This >> >> >> > > > > > > > >> >>> field can be quite useful in storing information >> >> like >> >> >> > > > "owner" >> >> >> > > > > of >> >> >> > > > > > > the >> >> >> > > > > > > > >> topic >> >> >> > > > > > > > >> >>> who issue the create command, etc, which is >> quite >> >> >> > > important >> >> >> > > > > for >> >> >> > > > > > a >> >> >> > > > > > > > >> >>> multi-tenant setting. Then in the describe topic >> >> >> request >> >> >> > > we >> >> >> > > > > can >> >> >> > > > > > > also >> >> >> > > > > > > > >> batch >> >> >> > > > > > > > >> >>> on regex of the metadata field. >> >> >> > > > > > > > >> >>> >> >> >> > > > > > > > >> >>> 4. Today all the admin operations are async in >> the >> >> >> sense >> >> >> > > > that >> >> >> > > > > > > > command >> >> >> > > > > > > > >> will >> >> >> > > > > > > > >> >>> return once it is written in ZK, and that is >> why we >> >> >> need >> >> >> > > > extra >> >> >> > > > > > > > >> verification >> >> >> > > > > > > > >> >>> like testUtil.waitForTopicCreated() / verify >> >> >> partition >> >> >> > > > > > > reassignment >> >> >> > > > > > > > >> >>> request, etc. With admin requests we could add a >> >> >> flag to >> >> >> > > > > enable >> >> >> > > > > > / >> >> >> > > > > > > > >> disable >> >> >> > > > > > > > >> >>> synchronous requests; when it is turned on, the >> >> >> response >> >> >> > > > will >> >> >> > > > > > not >> >> >> > > > > > > > >> return >> >> >> > > > > > > > >> >>> until the request has been completed. And for >> async >> >> >> > > requests >> >> >> > > > > we >> >> >> > > > > > > can >> >> >> > > > > > > > >> add a >> >> >> > > > > > > > >> >>> "token" field in the response, and then only >> need a >> >> >> > > general >> >> >> > > > > > "admin >> >> >> > > > > > > > >> >>> verification request" with the given token to >> check >> >> >> if the >> >> >> > > > > async >> >> >> > > > > > > > >> request >> >> >> > > > > > > > >> >>> has been completed. >> >> >> > > > > > > > >> >>> >> >> >> > > > > > > > >> >>> 5. +1 for extending Metadata request to include >> >> >> > > controller / >> >> >> > > > > > > > >> coordinator >> >> >> > > > > > > > >> >>> information, and then we can remove the >> >> >> ConsumerMetadata / >> >> >> > > > > > > > >> ClusterMetadata >> >> >> > > > > > > > >> >>> requests. >> >> >> > > > > > > > >> >>> >> >> >> > > > > > > > >> >>> Guozhang >> >> >> > > > > > > > >> >>> >> >> >> > > > > > > > >> >>> On Tue, Mar 3, 2015 at 10:23 AM, Joel Koshy < >> >> >> > > > > > jjkosh...@gmail.com> >> >> >> > > > > > > > >> wrote: >> >> >> > > > > > > > >> >>> >> >> >> > > > > > > > >> >>>> Thanks for sending that out Joe - I don't >> think I >> >> >> will be >> >> >> > > > > able >> >> >> > > > > > to >> >> >> > > > > > > > >> make >> >> >> > > > > > > > >> >>>> it today, so if notes can be sent out afterward >> >> that >> >> >> > > would >> >> >> > > > be >> >> >> > > > > > > > great. >> >> >> > > > > > > > >> >>>> >> >> >> > > > > > > > >> >>>> On Mon, Mar 02, 2015 at 09:16:13AM -0800, Gwen >> >> >> Shapira >> >> >> > > > wrote: >> >> >> > > > > > > > >> >>>> > Thanks for sending this out Joe. Looking >> forward >> >> >> to >> >> >> > > > > chatting >> >> >> > > > > > > with >> >> >> > > > > > > > >> >>>> everyone :) >> >> >> > > > > > > > >> >>>> > >> >> >> > > > > > > > >> >>>> > On Mon, Mar 2, 2015 at 6:46 AM, Joe Stein < >> >> >> > > > > > > joe.st...@stealth.ly> >> >> >> > > > > > > > >> wrote: >> >> >> > > > > > > > >> >>>> > > Hey, I just sent out a google hangout >> invite >> >> to >> >> >> all >> >> >> > > > pmc, >> >> >> > > > > > > > >> committers >> >> >> > > > > > > > >> >>>> and >> >> >> > > > > > > > >> >>>> > > everyone I found working on a KIP. If I >> missed >> >> >> anyone >> >> >> > > > in >> >> >> > > > > > the >> >> >> > > > > > > > >> invite >> >> >> > > > > > > > >> >>>> please >> >> >> > > > > > > > >> >>>> > > let me know and can update it, np. >> >> >> > > > > > > > >> >>>> > > >> >> >> > > > > > > > >> >>>> > > We should do this every Tuesday @ 2pm >> Eastern >> >> >> Time. >> >> >> > > > Maybe >> >> >> > > > > > we >> >> >> > > > > > > > can >> >> >> > > > > > > > >> get >> >> >> > > > > > > > >> >>>> INFRA >> >> >> > > > > > > > >> >>>> > > help to make a google account so we can >> manage >> >> >> > > better? >> >> >> > > > > > > > >> >>>> > > >> >> >> > > > > > > > >> >>>> > > To discuss >> >> >> > > > > > > > >> >>>> > > >> >> >> > > > > > > > >> >>>> >> >> >> > > > > > > > >> >> >> >> > > > > > > > >> >> >> > > > > > > >> >> >> > > > > > >> >> >> > > > > >> >> >> > > > >> >> >> > > >> >> >> >> >> >> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals >> >> >> > > > > > > > >> >>>> > > in progress and related JIRA that are >> >> >> interdependent >> >> >> > > > and >> >> >> > > > > > > common >> >> >> > > > > > > > >> work. >> >> >> > > > > > > > >> >>>> > > >> >> >> > > > > > > > >> >>>> > > ~ Joe Stein >> >> >> > > > > > > > >> >>>> > > >> >> >> > > > > > > > >> >>>> > > On Tue, Feb 24, 2015 at 2:59 PM, Jay Kreps >> < >> >> >> > > > > > > > jay.kr...@gmail.com> >> >> >> > > > > > > > >> >>>> wrote: >> >> >> > > > > > > > >> >>>> > > >> >> >> > > > > > > > >> >>>> > >> Let's stay on Google hangouts that will >> also >> >> >> record >> >> >> > > > and >> >> >> > > > > > make >> >> >> > > > > > > > the >> >> >> > > > > > > > >> >>>> sessions >> >> >> > > > > > > > >> >>>> > >> available on youtube. >> >> >> > > > > > > > >> >>>> > >> >> >> >> > > > > > > > >> >>>> > >> -Jay >> >> >> > > > > > > > >> >>>> > >> >> >> >> > > > > > > > >> >>>> > >> On Tue, Feb 24, 2015 at 11:49 AM, Jeff >> >> Holoman >> >> >> < >> >> >> > > > > > > > >> >>>> jholo...@cloudera.com> >> >> >> > > > > > > > >> >>>> > >> wrote: >> >> >> > > > > > > > >> >>>> > >> >> >> >> > > > > > > > >> >>>> > >> > Jay / Joe >> >> >> > > > > > > > >> >>>> > >> > >> >> >> > > > > > > > >> >>>> > >> > We're happy to send out a Webex for this >> >> >> purpose. >> >> >> > > We >> >> >> > > > > > could >> >> >> > > > > > > > >> record >> >> >> > > > > > > > >> >>>> the >> >> >> > > > > > > > >> >>>> > >> > sessions if there is interest and >> publish >> >> >> them >> >> >> > > out. >> >> >> > > > > > > > >> >>>> > >> > >> >> >> > > > > > > > >> >>>> > >> > Thanks >> >> >> > > > > > > > >> >>>> > >> > >> >> >> > > > > > > > >> >>>> > >> > Jeff >> >> >> > > > > > > > >> >>>> > >> > >> >> >> > > > > > > > >> >>>> > >> > On Tue, Feb 24, 2015 at 11:28 AM, Jay >> >> Kreps < >> >> >> > > > > > > > >> jay.kr...@gmail.com> >> >> >> > > > > > > > >> >>>> wrote: >> >> >> > > > > > > > >> >>>> > >> > >> >> >> > > > > > > > >> >>>> > >> > > Let's try to get the technical >> hang-ups >> >> >> sorted >> >> >> > > > out, >> >> >> > > > > > > > though. >> >> >> > > > > > > > >> I >> >> >> > > > > > > > >> >>>> really >> >> >> > > > > > > > >> >>>> > >> > think >> >> >> > > > > > > > >> >>>> > >> > > there is some benefit to live >> discussion >> >> vs >> >> >> > > > > writing. I >> >> >> > > > > > > am >> >> >> > > > > > > > >> >>>> hopeful that >> >> >> > > > > > > > >> >>>> > >> if >> >> >> > > > > > > > >> >>>> > >> > > we post instructions and give >> ourselves a >> >> >> few >> >> >> > > > > attempts >> >> >> > > > > > > we >> >> >> > > > > > > > >> can >> >> >> > > > > > > > >> >>>> get it >> >> >> > > > > > > > >> >>>> > >> > > working. >> >> >> > > > > > > > >> >>>> > >> > > >> >> >> > > > > > > > >> >>>> > >> > > Tuesday at that time would work for >> >> >> me...any >> >> >> > > > > > objections? >> >> >> > > > > > > > >> >>>> > >> > > >> >> >> > > > > > > > >> >>>> > >> > > -Jay >> >> >> > > > > > > > >> >>>> > >> > > >> >> >> > > > > > > > >> >>>> > >> > > On Tue, Feb 24, 2015 at 8:18 AM, Joe >> >> Stein >> >> >> < >> >> >> > > > > > > > >> joe.st...@stealth.ly >> >> >> > > > > > > > >> >>>> > >> >> >> > > > > > > > >> >>>> > >> wrote: >> >> >> > > > > > > > >> >>>> > >> > > >> >> >> > > > > > > > >> >>>> > >> > > > Weekly would be great maybe like >> every >> >> >> > > Tuesday ~ >> >> >> > > > > 1pm >> >> >> > > > > > > ET >> >> >> > > > > > > > / >> >> >> > > > > > > > >> 10am >> >> >> > > > > > > > >> >>>> PT >> >> >> > > > > > > > >> >>>> > >> ???? >> >> >> > > > > > > > >> >>>> > >> > > > >> >> >> > > > > > > > >> >>>> > >> > > > I don't mind google hangout but >> there >> >> is >> >> >> > > always >> >> >> > > > > some >> >> >> > > > > > > > >> issue or >> >> >> > > > > > > > >> >>>> > >> whatever >> >> >> > > > > > > > >> >>>> > >> > so >> >> >> > > > > > > > >> >>>> > >> > > > we know the apache irc channel >> works. >> >> We >> >> >> can >> >> >> > > > start >> >> >> > > > > > > there >> >> >> > > > > > > > >> and >> >> >> > > > > > > > >> >>>> see how >> >> >> > > > > > > > >> >>>> > >> it >> >> >> > > > > > > > >> >>>> > >> > > > goes? We can pull transcripts too >> and >> >> >> > > associate >> >> >> > > > to >> >> >> > > > > > > > >> tickets if >> >> >> > > > > > > > >> >>>> need be >> >> >> > > > > > > > >> >>>> > >> > > makes >> >> >> > > > > > > > >> >>>> > >> > > > it helpful for things. >> >> >> > > > > > > > >> >>>> > >> > > > >> >> >> > > > > > > > >> >>>> > >> > > > ~ Joestein >> >> >> > > > > > > > >> >>>> > >> > > > >> >> >> > > > > > > > >> >>>> > >> > > > On Tue, Feb 24, 2015 at 11:10 AM, >> Jay >> >> >> Kreps < >> >> >> > > > > > > > >> >>>> jay.kr...@gmail.com> >> >> >> > > > > > > > >> >>>> > >> > wrote: >> >> >> > > > > > > > >> >>>> > >> > > > >> >> >> > > > > > > > >> >>>> > >> > > > > We'd talked about doing a Google >> >> >> Hangout to >> >> >> > > > chat >> >> >> > > > > > > about >> >> >> > > > > > > > >> this. >> >> >> > > > > > > > >> >>>> What >> >> >> > > > > > > > >> >>>> > >> > about >> >> >> > > > > > > > >> >>>> > >> > > > > generalizing that a little >> >> further...I >> >> >> > > > actually >> >> >> > > > > > > think >> >> >> > > > > > > > it >> >> >> > > > > > > > >> >>>> would be >> >> >> > > > > > > > >> >>>> > >> > good >> >> >> > > > > > > > >> >>>> > >> > > > for >> >> >> > > > > > > > >> >>>> > >> > > > > everyone spending a reasonable >> chunk >> >> of >> >> >> > > their >> >> >> > > > > week >> >> >> > > > > > > on >> >> >> > > > > > > > >> Kafka >> >> >> > > > > > > > >> >>>> stuff >> >> >> > > > > > > > >> >>>> > >> to >> >> >> > > > > > > > >> >>>> > >> > > > maybe >> >> >> > > > > > > > >> >>>> > >> > > > > sync up once a week. I think we >> could >> >> >> use >> >> >> > > time >> >> >> > > > > to >> >> >> > > > > > > talk >> >> >> > > > > > > > >> >>>> through >> >> >> > > > > > > > >> >>>> > >> design >> >> >> > > > > > > > >> >>>> > >> > > > > stuff, make sure we are on top of >> >> code >> >> >> > > > reviews, >> >> >> > > > > > talk >> >> >> > > > > > > > >> through >> >> >> > > > > > > > >> >>>> any >> >> >> > > > > > > > >> >>>> > >> > tricky >> >> >> > > > > > > > >> >>>> > >> > > > > issues, etc. >> >> >> > > > > > > > >> >>>> > >> > > > > >> >> >> > > > > > > > >> >>>> > >> > > > > We can make it publicly available >> so >> >> >> that >> >> >> > > any >> >> >> > > > > one >> >> >> > > > > > > can >> >> >> > > > > > > > >> follow >> >> >> > > > > > > > >> >>>> along >> >> >> > > > > > > > >> >>>> > >> > who >> >> >> > > > > > > > >> >>>> > >> > > > > likes. >> >> >> > > > > > > > >> >>>> > >> > > > > >> >> >> > > > > > > > >> >>>> > >> > > > > Any interest in doing this? If so >> >> I'll >> >> >> try >> >> >> > > to >> >> >> > > > > set >> >> >> > > > > > it >> >> >> > > > > > > > up >> >> >> > > > > > > > >> >>>> starting >> >> >> > > > > > > > >> >>>> > >> next >> >> >> > > > > > > > >> >>>> > >> > > > week. >> >> >> > > > > > > > >> >>>> > >> > > > > >> >> >> > > > > > > > >> >>>> > >> > > > > -Jay >> >> >> > > > > > > > >> >>>> > >> > > > > >> >> >> > > > > > > > >> >>>> > >> > > > > On Tue, Feb 24, 2015 at 3:57 AM, >> >> Andrii >> >> >> > > > > Biletskyi >> >> >> > > > > > < >> >> >> > > > > > > > >> >>>> > >> > > > > andrii.bilets...@stealth.ly> >> wrote: >> >> >> > > > > > > > >> >>>> > >> > > > > >> >> >> > > > > > > > >> >>>> > >> > > > > > Hi all, >> >> >> > > > > > > > >> >>>> > >> > > > > > >> >> >> > > > > > > > >> >>>> > >> > > > > > I've updated KIP page, fixed / >> >> >> aligned >> >> >> > > > > document >> >> >> > > > > > > > >> structure. >> >> >> > > > > > > > >> >>>> Also I >> >> >> > > > > > > > >> >>>> > >> > > added >> >> >> > > > > > > > >> >>>> > >> > > > > > some >> >> >> > > > > > > > >> >>>> > >> > > > > > very initial proposal for >> >> >> AdminClient so >> >> >> > > we >> >> >> > > > > have >> >> >> > > > > > > > >> something >> >> >> > > > > > > > >> >>>> to >> >> >> > > > > > > > >> >>>> > >> start >> >> >> > > > > > > > >> >>>> > >> > > > from >> >> >> > > > > > > > >> >>>> > >> > > > > > while >> >> >> > > > > > > > >> >>>> > >> > > > > > discussing the KIP. >> >> >> > > > > > > > >> >>>> > >> > > > > > >> >> >> > > > > > > > >> >>>> > >> > > > > > >> >> >> > > > > > > > >> >>>> > >> > > > > > >> >> >> > > > > > > > >> >>>> > >> > > > > >> >> >> > > > > > > > >> >>>> > >> > > > >> >> >> > > > > > > > >> >>>> > >> > > >> >> >> > > > > > > > >> >>>> > >> > >> >> >> > > > > > > > >> >>>> > >> >> >> >> > > > > > > > >> >>>> >> >> >> > > > > > > > >> >> >> >> > > > > > > > >> >> >> > > > > > > >> >> >> > > > > > >> >> >> > > > > >> >> >> > > > >> >> >> > > >> >> >> >> >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations >> >> >> > > > > > > > >> >>>> > >> > > > > > >> >> >> > > > > > > > >> >>>> > >> > > > > > Thanks, >> >> >> > > > > > > > >> >>>> > >> > > > > > Andrii Biletskyi >> >> >> > > > > > > > >> >>>> > >> > > > > > >> >> >> > > > > > > > >> >>>> > >> > > > > > On Wed, Feb 18, 2015 at 9:01 PM, >> >> >> Andrii >> >> >> > > > > > Biletskyi >> >> >> > > > > > > < >> >> >> > > > > > > > >> >>>> > >> > > > > > andrii.bilets...@stealth.ly> >> >> wrote: >> >> >> > > > > > > > >> >>>> > >> > > > > > >> >> >> > > > > > > > >> >>>> > >> > > > > > > Jay, >> >> >> > > > > > > > >> >>>> > >> > > > > > > >> >> >> > > > > > > > >> >>>> > >> > > > > > > Re error messages: you are >> right, >> >> >> in >> >> >> > > most >> >> >> > > > > > cases >> >> >> > > > > > > > >> client >> >> >> > > > > > > > >> >>>> will >> >> >> > > > > > > > >> >>>> > >> have >> >> >> > > > > > > > >> >>>> > >> > > > enough >> >> >> > > > > > > > >> >>>> > >> > > > > > > context to show descriptive >> error >> >> >> > > message. >> >> >> > > > > My >> >> >> > > > > > > > >> concern is >> >> >> > > > > > > > >> >>>> that >> >> >> > > > > > > > >> >>>> > >> we >> >> >> > > > > > > > >> >>>> > >> > > will >> >> >> > > > > > > > >> >>>> > >> > > > > > have >> >> >> > > > > > > > >> >>>> > >> > > > > > > to >> >> >> > > > > > > > >> >>>> > >> > > > > > > add lots of new error codes >> for >> >> >> each >> >> >> > > > > possible >> >> >> > > > > > > > >> error. Of >> >> >> > > > > > > > >> >>>> course, >> >> >> > > > > > > > >> >>>> > >> > we >> >> >> > > > > > > > >> >>>> > >> > > > > could >> >> >> > > > > > > > >> >>>> > >> > > > > > > reuse >> >> >> > > > > > > > >> >>>> > >> > > > > > > some of existing like >> >> >> > > > > > > UknownTopicOrPartitionCode, >> >> >> > > > > > > > >> but we >> >> >> > > > > > > > >> >>>> will >> >> >> > > > > > > > >> >>>> > >> > also >> >> >> > > > > > > > >> >>>> > >> > > > need >> >> >> > > > > > > > >> >>>> > >> > > > > > to >> >> >> > > > > > > > >> >>>> > >> > > > > > > add smth like: >> >> >> TopicAlreadyExistsCode, >> >> >> > > > > > > > >> >>>> TopicConfigInvalid (both >> >> >> > > > > > > > >> >>>> > >> > for >> >> >> > > > > > > > >> >>>> > >> > > > > topic >> >> >> > > > > > > > >> >>>> > >> > > > > > > name and config, and probably >> >> user >> >> >> would >> >> >> > > > > like >> >> >> > > > > > to >> >> >> > > > > > > > >> know >> >> >> > > > > > > > >> >>>> what >> >> >> > > > > > > > >> >>>> > >> > exactly >> >> >> > > > > > > > >> >>>> > >> > > > > > > is wrong in his config), >> >> >> > > > > > > InvalidReplicaAssignment, >> >> >> > > > > > > > >> >>>> > >> InternalError >> >> >> > > > > > > > >> >>>> > >> > > > (e.g. >> >> >> > > > > > > > >> >>>> > >> > > > > > > zookeeper failure) etc. >> >> >> > > > > > > > >> >>>> > >> > > > > > > And this is only for >> >> TopicCommand, >> >> >> we >> >> >> > > will >> >> >> > > > > > also >> >> >> > > > > > > > >> need to >> >> >> > > > > > > > >> >>>> add >> >> >> > > > > > > > >> >>>> > >> > similar >> >> >> > > > > > > > >> >>>> > >> > > > > stuff >> >> >> > > > > > > > >> >>>> > >> > > > > > > for >> >> >> > > > > > > > >> >>>> > >> > > > > > > ReassignPartitions, >> >> >> PreferredReplica. So >> >> >> > > > > we'll >> >> >> > > > > > > end >> >> >> > > > > > > > >> up >> >> >> > > > > > > > >> >>>> with a >> >> >> > > > > > > > >> >>>> > >> > large >> >> >> > > > > > > > >> >>>> > >> > > > list >> >> >> > > > > > > > >> >>>> > >> > > > > > of >> >> >> > > > > > > > >> >>>> > >> > > > > > > error codes, used only in >> Admin >> >> >> > > protocol. >> >> >> > > > > > > > >> >>>> > >> > > > > > > Having said that, I agree my >> >> >> proposal is >> >> >> > > > not >> >> >> > > > > > > > >> consistent >> >> >> > > > > > > > >> >>>> with >> >> >> > > > > > > > >> >>>> > >> > other >> >> >> > > > > > > > >> >>>> > >> > > > > cases. >> >> >> > > > > > > > >> >>>> > >> > > > > > > Maybe we can find better >> solution >> >> >> or >> >> >> > > > > something >> >> >> > > > > > > > >> >>>> in-between. >> >> >> > > > > > > > >> >>>> > >> > > > > > > >> >> >> > > > > > > > >> >>>> > >> > > > > > > Re Hangout chat: I think it >> is a >> >> >> great >> >> >> > > > idea. >> >> >> > > > > > > This >> >> >> > > > > > > > >> way we >> >> >> > > > > > > > >> >>>> can >> >> >> > > > > > > > >> >>>> > >> move >> >> >> > > > > > > > >> >>>> > >> > > on >> >> >> > > > > > > > >> >>>> > >> > > > > > > faster. >> >> >> > > > > > > > >> >>>> > >> > > > > > > Let's agree somehow on >> date/time >> >> so >> >> >> > > people >> >> >> > > > > can >> >> >> > > > > > > > join. >> >> >> > > > > > > > >> >>>> Will work >> >> >> > > > > > > > >> >>>> > >> > for >> >> >> > > > > > > > >> >>>> > >> > > me >> >> >> > > > > > > > >> >>>> > >> > > > > > this >> >> >> > > > > > > > >> >>>> > >> > > > > > > and >> >> >> > > > > > > > >> >>>> > >> > > > > > > next week almost anytime if >> >> agreed >> >> >> in >> >> >> > > > > advance. >> >> >> > > > > > > > >> >>>> > >> > > > > > > >> >> >> > > > > > > > >> >>>> > >> > > > > > > Thanks, >> >> >> > > > > > > > >> >>>> > >> > > > > > > Andrii >> >> >> > > > > > > > >> >>>> > >> > > > > > > >> >> >> > > > > > > > >> >>>> > >> > > > > > > On Wed, Feb 18, 2015 at 7:09 >> PM, >> >> >> Jay >> >> >> > > > Kreps < >> >> >> > > > > > > > >> >>>> > >> jay.kr...@gmail.com> >> >> >> > > > > > > > >> >>>> > >> > > > > wrote: >> >> >> > > > > > > > >> >>>> > >> > > > > > > >> >> >> > > > > > > > >> >>>> > >> > > > > > >> Hey Andrii, >> >> >> > > > > > > > >> >>>> > >> > > > > > >> >> >> >> > > > > > > > >> >>>> > >> > > > > > >> Generally we can do good >> error >> >> >> handling >> >> >> > > > > > without >> >> >> > > > > > > > >> needing >> >> >> > > > > > > > >> >>>> custom >> >> >> > > > > > > > >> >>>> > >> > > > > > server-side >> >> >> > > > > > > > >> >>>> > >> > > > > > >> messages. I.e. generally the >> >> >> client has >> >> >> > > > the >> >> >> > > > > > > > >> context to >> >> >> > > > > > > > >> >>>> know >> >> >> > > > > > > > >> >>>> > >> that >> >> >> > > > > > > > >> >>>> > >> > > if >> >> >> > > > > > > > >> >>>> > >> > > > it >> >> >> > > > > > > > >> >>>> > >> > > > > > got >> >> >> > > > > > > > >> >>>> > >> > > > > > >> an error that the topic >> doesn't >> >> >> exist >> >> >> > > to >> >> >> > > > > say >> >> >> > > > > > > > >> "Topic X >> >> >> > > > > > > > >> >>>> doesn't >> >> >> > > > > > > > >> >>>> > >> > > exist" >> >> >> > > > > > > > >> >>>> > >> > > > > > >> rather >> >> >> > > > > > > > >> >>>> > >> > > > > > >> than "error code 14" (or >> >> >> whatever). >> >> >> > > Maybe >> >> >> > > > > > there >> >> >> > > > > > > > are >> >> >> > > > > > > > >> >>>> specific >> >> >> > > > > > > > >> >>>> > >> > cases >> >> >> > > > > > > > >> >>>> > >> > > > > where >> >> >> > > > > > > > >> >>>> > >> > > > > > >> this is hard? If we want to >> add >> >> >> > > > server-side >> >> >> > > > > > > error >> >> >> > > > > > > > >> >>>> messages we >> >> >> > > > > > > > >> >>>> > >> > > really >> >> >> > > > > > > > >> >>>> > >> > > > > do >> >> >> > > > > > > > >> >>>> > >> > > > > > >> need to do this in a >> consistent >> >> >> way >> >> >> > > > across >> >> >> > > > > > the >> >> >> > > > > > > > >> protocol. >> >> >> > > > > > > > >> >>>> > >> > > > > > >> >> >> >> > > > > > > > >> >>>> > >> > > > > > >> I still have a bunch of open >> >> >> questions >> >> >> > > > here >> >> >> > > > > > > from >> >> >> > > > > > > > my >> >> >> > > > > > > > >> >>>> previous >> >> >> > > > > > > > >> >>>> > >> > > list. I >> >> >> > > > > > > > >> >>>> > >> > > > > > will >> >> >> > > > > > > > >> >>>> > >> > > > > > >> be out for the next few days >> for >> >> >> Strata >> >> >> > > > > > though. >> >> >> > > > > > > > >> Maybe >> >> >> > > > > > > > >> >>>> we could >> >> >> > > > > > > > >> >>>> > >> > do >> >> >> > > > > > > > >> >>>> > >> > > a >> >> >> > > > > > > > >> >>>> > >> > > > > > Google >> >> >> > > > > > > > >> >>>> > >> > > > > > >> Hangout chat on any open >> issues >> >> >> some >> >> >> > > time >> >> >> > > > > > > towards >> >> >> > > > > > > > >> the >> >> >> > > > > > > > >> >>>> end of >> >> >> > > > > > > > >> >>>> > >> > next >> >> >> > > > > > > > >> >>>> > >> > > > week >> >> >> > > > > > > > >> >>>> > >> > > > > > for >> >> >> > > > > > > > >> >>>> > >> > > > > > >> anyone interested in this >> >> ticket? >> >> >> I >> >> >> > > have >> >> >> > > > a >> >> >> > > > > > > > feeling >> >> >> > > > > > > > >> that >> >> >> > > > > > > > >> >>>> might >> >> >> > > > > > > > >> >>>> > >> > > > progress >> >> >> > > > > > > > >> >>>> > >> > > > > > >> things a little faster than >> >> >> email--I >> >> >> > > > think >> >> >> > > > > we >> >> >> > > > > > > > >> could talk >> >> >> > > > > > > > >> >>>> > >> through >> >> >> > > > > > > > >> >>>> > >> > > > those >> >> >> > > > > > > > >> >>>> > >> > > > > > >> issues I brought up fairly >> >> >> quickly... >> >> >> > > > > > > > >> >>>> > >> > > > > > >> >> >> >> > > > > > > > >> >>>> > >> > > > > > >> -Jay >> >> >> > > > > > > > >> >>>> > >> > > > > > >> >> >> >> > > > > > > > >> >>>> > >> > > > > > >> On Wed, Feb 18, 2015 at 7:27 >> AM, >> >> >> Andrii >> >> >> > > > > > > > Biletskyi < >> >> >> > > > > > > > >> >>>> > >> > > > > > >> andrii.bilets...@stealth.ly> >> >> >> wrote: >> >> >> > > > > > > > >> >>>> > >> > > > > > >> >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > Hi all, >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > I'm trying to address some >> of >> >> >> the >> >> >> > > > issues >> >> >> > > > > > > which >> >> >> > > > > > > > >> were >> >> >> > > > > > > > >> >>>> > >> mentioned >> >> >> > > > > > > > >> >>>> > >> > > > > earlier >> >> >> > > > > > > > >> >>>> > >> > > > > > >> about >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > Admin RQ/RP format. One of >> >> >> those was >> >> >> > > > > about >> >> >> > > > > > > > >> batching >> >> >> > > > > > > > >> >>>> > >> > operations. >> >> >> > > > > > > > >> >>>> > >> > > > What >> >> >> > > > > > > > >> >>>> > >> > > > > > if >> >> >> > > > > > > > >> >>>> > >> > > > > > >> we >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > follow TopicCommand >> approach >> >> >> and let >> >> >> > > > > people >> >> >> > > > > > > > >> specify >> >> >> > > > > > > > >> >>>> > >> topic-name >> >> >> > > > > > > > >> >>>> > >> > > by >> >> >> > > > > > > > >> >>>> > >> > > > > > >> regexp - >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > would that cover most of >> the >> >> use >> >> >> > > cases? >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > Secondly, is what >> information >> >> >> should >> >> >> > > we >> >> >> > > > > > > > generally >> >> >> > > > > > > > >> >>>> provide in >> >> >> > > > > > > > >> >>>> > >> > > Admin >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > responses. >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > I realize that Admin >> commands >> >> >> don't >> >> >> > > > imply >> >> >> > > > > > > they >> >> >> > > > > > > > >> will >> >> >> > > > > > > > >> >>>> be used >> >> >> > > > > > > > >> >>>> > >> > only >> >> >> > > > > > > > >> >>>> > >> > > > in >> >> >> > > > > > > > >> >>>> > >> > > > > > CLI >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > but, >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > it seems to me, CLI is a >> very >> >> >> > > important >> >> >> > > > > > > client >> >> >> > > > > > > > >> of this >> >> >> > > > > > > > >> >>>> > >> > feature. >> >> >> > > > > > > > >> >>>> > >> > > In >> >> >> > > > > > > > >> >>>> > >> > > > > > this >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > case, >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > seems logical, we would >> like >> >> to >> >> >> > > provide >> >> >> > > > > > users >> >> >> > > > > > > > >> with >> >> >> > > > > > > > >> >>>> rich >> >> >> > > > > > > > >> >>>> > >> > > experience >> >> >> > > > > > > > >> >>>> > >> > > > > in >> >> >> > > > > > > > >> >>>> > >> > > > > > >> terms >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > of >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > getting results / errors of >> >> the >> >> >> > > > executed >> >> >> > > > > > > > >> commands. >> >> >> > > > > > > > >> >>>> Usually >> >> >> > > > > > > > >> >>>> > >> we >> >> >> > > > > > > > >> >>>> > >> > > > supply >> >> >> > > > > > > > >> >>>> > >> > > > > > >> with >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > responses only errorCode, >> >> which >> >> >> looks >> >> >> > > > > very >> >> >> > > > > > > > >> limiting, >> >> >> > > > > > > > >> >>>> in case >> >> >> > > > > > > > >> >>>> > >> > of >> >> >> > > > > > > > >> >>>> > >> > > > CLI >> >> >> > > > > > > > >> >>>> > >> > > > > we >> >> >> > > > > > > > >> >>>> > >> > > > > > >> may >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > want to print human >> readable >> >> >> error >> >> >> > > > > > > description. >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > So, taking into account >> >> >> previous item >> >> >> > > > > about >> >> >> > > > > > > > >> batching, >> >> >> > > > > > > > >> >>>> what >> >> >> > > > > > > > >> >>>> > >> do >> >> >> > > > > > > > >> >>>> > >> > > you >> >> >> > > > > > > > >> >>>> > >> > > > > > think >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > about >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > having smth like: >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > ('create' doesn't support >> >> >> regexp) >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > CreateTopicRequest => >> >> TopicName >> >> >> > > > > Partitions >> >> >> > > > > > > > >> Replicas >> >> >> > > > > > > > >> >>>> > >> > > > > ReplicaAssignment >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > [Config] >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > CreateTopicResponse => >> >> ErrorCode >> >> >> > > > > > > > ErrorDescription >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > ErrorCode => int16 >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > ErrorDescription => >> string >> >> >> (empty >> >> >> > > if >> >> >> > > > > > > > >> successful) >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > >> >> >> > > > > > > > >> >>>> > >> > > > > > >> > AlterTopicRequest -> >> >> >> TopicNameRegexp >> >> >> > > > >> >> > >> >> > ... >> >> > >> >> > [Message clipped] >> >> >>
