Yes that is what I was alluding to when I said that if we finally do request rerouting in Kafka then the field would add little to no value. I wasn't sure if we agreed that we _will_ do rerouting or whether we agreed to evaluate it (KAFKA-1912). Andrii can you update the KIP with this?
Thanks, Joel On Wed, Mar 18, 2015 at 02:55:00PM -0700, Jun Rao wrote: > Joel, Andril, > > I think we agreed that those admin requests can be issued to any broker. > Because of that, there doesn't seem to be a strong need to know the > controller. So, perhaps we can proceed by not making any change to the > format of TMR right now. When we start using create topic request in the > producer, we will need a new version of TMR that doesn't trigger auto topic > creation. But that can be done later. > > As a first cut implementation, I think the broker can just write to ZK > directly for > createToipic/alterTopic/reassignPartitions/preferredLeaderElection > requests, instead of forwarding them to the controller. This will simplify > the implementation on the broker side. > > Thanks, > > Jun > > > On Wed, Mar 18, 2015 at 11:58 AM, Joel Koshy <jjkosh...@gmail.com> wrote: > > > (Thanks Andrii for the summary) > > > > For (1) yes we will circle back on that shortly after syncing up in > > person. I think it is close to getting committed although development > > for KAFKA-1927 can probably begin without it. > > > > There is one more item we covered at the hangout. i.e., whether we > > want to add the coordinator to the topic metadata response or provide > > a clearer ClusterMetadataRequest. > > > > There are two reasons I think we should try and avoid adding the > > field: > > - It is irrelevant to topic metadata > > - If we finally do request rerouting in Kafka then the field would add > > little to no value. (It still helps to have a separate > > ClusterMetadataRequest to query for cluster-wide information such as > > 'which broker is the controller?' as Joe mentioned.) > > > > I think it would be cleaner to have an explicit ClusterMetadataRequest > > that you can send to any broker in order to obtain the controller (and > > in the future possibly other cluster-wide information). I think the > > main argument against doing this and instead adding it to the topic > > metadata response was convenience - i.e., you don't have to discover > > the controller in advance. However, I don't see much actual > > benefit/convenience in this and in fact think it is a non-issue. Let > > me know if I'm overlooking something here. > > > > As an example, say we need to initiate partition reassignment by > > issuing the new ReassignPartitionsRequest to the controller (assume we > > already have the desired manual partition assignment). If we are to > > augment topic metadata response then the flow be something like this : > > > > - Issue topic metadata request to any broker (and discover the > > controller > > - Connect to controller if required (i.e., if the broker above != > > controller) > > - Issue the partition reassignment request to the controller. > > > > With an explicit cluster metadata request it would be: > > - Issue cluster metadata request to any broker > > - Connect to controller if required (i.e., if the broker above != > > controller) > > - Issue the partition reassignment request > > > > So it seems to add little practical value and bloats topic metadata > > response with an irrelevant detail. > > > > The other angle to this is the following - is it a matter of naming? > > Should we just rename topic metadata request/response to just > > MetadataRequest/Response and add cluster metadata to it? By that same > > token should we also allow querying for the consumer coordinator (and > > in future transaction coordinator) as well? This leads to a bloated > > request which isn't very appealing and altogether confusing. > > > > Thanks, > > > > Joel > > > > On Wed, Mar 18, 2015 at 09:34:12AM -0700, Jun Rao wrote: > > > Andri, > > > > > > Thanks for the summary. > > > > > > 1. I just realized that in order to start working on KAFKA-1927, we will > > > need to merge the changes to OffsetCommitRequest (from 0.8.2) to trunk. > > > This is planned to be done as part of KAFKA-1634. So, we will need > > Guozhang > > > and Joel's help to wrap this up. > > > > > > 2. Thinking about this a bit more, if the semantic of those "write" > > > requests is async (i.e., after the client gets a response, it just means > > > that the operation is initiated, but not necessarily completed), we don't > > > really need to forward the requests to the controller. Instead, the > > > receiving broker can just write the operation to ZK as the admin command > > > line tool previously does. This will simplify the implementation. > > > > > > 8. There is another implementation detail for describe topic. Ideally, we > > > want to read the topic config from the broker cache, instead of > > ZooKeeper. > > > Currently, every broker reads the topic-level config for all topics. > > > However, it ignores those for topics not hosted on itself. So, we may > > need > > > to change TopicConfigManager a bit so that it caches the configs for all > > > topics. > > > > > > Thanks, > > > > > > Jun > > > > > > > > > On Tue, Mar 17, 2015 at 1:13 PM, Andrii Biletskyi < > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > Guys, > > > > > > > > Thanks for a great discussion! > > > > Here are the actions points: > > > > > > > > 1. Q: Get rid of all scala requests objects, use java protocol > > definitions. > > > > A: Gwen kindly took that (KAFKA-1927). It's important to speed up > > > > review procedure > > > > there since this ticket blocks other important changes. > > > > > > > > 2. Q: Generic re-reroute facility vs client maintaining cluster state. > > > > A: Jay has added pseudo code to KAFKA-1912 - need to consider > > whether > > > > this will be > > > > easy to implement as a server-side feature (comments are > > > > welcomed!). > > > > > > > > 3. Q: Controller field in wire protocol. > > > > A: This might be useful for clients, add this to > > TopicMetadataResponse > > > > (already in KIP). > > > > > > > > 4. Q: Decoupling topic creation from TMR. > > > > A: I will add proposed by Jun solution (using clientId for that) > > to the > > > > KIP. > > > > > > > > 5. Q: Bumping new versions of TMR vs grabbing all protocol changes in > > one > > > > version. > > > > A: It was decided to try to gather all changes to protocol (before > > > > release). > > > > In case of TMR it worth checking: KAFKA-2020 and KIP-13 > > (quotas) > > > > > > > > 6. Q: JSON lib is needed to deserialize user's input in CLI tool. > > > > A: Use jackson for that, /tools project is a separate jar so > > shouldn't > > > > be a big deal. > > > > > > > > 7. Q: VerifyReassingPartitions vs generic status check command. > > > > A: For long-running requests like reassign partitions *progress* > > check > > > > request is useful, > > > > it makes sense to introduce it. > > > > > > > > Please add, correct me if I missed something. > > > > > > > > Thanks, > > > > Andrii Biletskyi > > > > > > > > On Tue, Mar 17, 2015 at 6:20 PM, Andrii Biletskyi < > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > Joel, > > > > > > > > > > You are right, I removed ClusterMetadata because we have partially > > > > > what we need in TopicMetadata. Also, as Jay pointed out earlier, we > > > > > would like to have "orthogonal" API, but at the same time we need > > > > > to be backward compatible. > > > > > > > > > > But I like your idea and even have some other arguments for this > > option: > > > > > There is also DescribeTopicRequest which was proposed in this KIP, > > > > > it returns topic configs, partitions, replication factor plus > > partition > > > > > ISR, ASR, > > > > > leader replica. The later part is really already there in > > > > > TopicMetadataRequest. > > > > > So again we'll have to add stuff to TMR, not to duplicate some info > > in > > > > > newly added requests. However, this way we'll end up with "monster" > > > > > request which returns cluster metadata, topic replication and config > > info > > > > > plus partition replication data. Seems logical to split TMR to > > > > > - ClusterMetadata (brokers + controller, maybe smth else) > > > > > - TopicMetadata (topic info + partition details) > > > > > But since current TMR is involved in lots of places (including > > network > > > > > client, > > > > > as I understand) this might be very serious change and it probably > > makes > > > > > sense to stick with current approach. > > > > > > > > > > Thanks, > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > On Tue, Mar 17, 2015 at 5:29 PM, Joel Koshy <jjkosh...@gmail.com> > > wrote: > > > > > > > > > >> I may be missing some context but hopefully this will also be > > covered > > > > >> today: I thought the earlier proposal where there was an explicit > > > > >> ClusterMetadata request was clearer and explicit. During the course > > of > > > > >> this thread I think the conclusion was that the main need was for > > > > >> controller information and that can be rolled into the topic > > metadata > > > > >> response but that seems a bit irrelevant to topic metadata. FWIW I > > > > >> think the full broker-list is also irrelevant to topic metadata, but > > > > >> it is already there and in use. I think there is still room for an > > > > >> explicit ClusterMetadata request since there may be other > > > > >> cluster-level information that we may want to add over time (and > > that > > > > >> have nothing to do with topic metadata). > > > > >> > > > > >> On Tue, Mar 17, 2015 at 02:45:30PM +0200, Andrii Biletskyi wrote: > > > > >> > Jun, > > > > >> > > > > > >> > 101. Okay, if you say that such use case is important. I also > > think > > > > >> > using clientId for these purposes is fine - if we already have > > this > > > > >> field > > > > >> > as part of all Wire protocol messages, why not use that. > > > > >> > I will update KIP-4 page if nobody has other ideas (which may > > come up > > > > >> > during the call today). > > > > >> > > > > > >> > 102.1 Agree, I'll update the KIP accordingly. I think we can add > > new, > > > > >> > fine-grained error codes if some error code received in specific > > case > > > > >> > won't give enough context to return a descriptive error message > > for > > > > >> user. > > > > >> > > > > > >> > Look forward to discussing all outstanding issues in detail today > > > > during > > > > >> > the call. > > > > >> > > > > > >> > Thanks, > > > > >> > Andrii Biletskyi > > > > >> > > > > > >> > > > > > >> > > > > > >> > On Mon, Mar 16, 2015 at 10:59 PM, Jun Rao <j...@confluent.io> > > wrote: > > > > >> > > > > > >> > > 101. There may be a use case where you only want the topics to > > be > > > > >> created > > > > >> > > manually by admins. Currently, you can do that by disabling auto > > > > topic > > > > >> > > creation and issue topic creation from the TopicCommand. If we > > > > >> disable auto > > > > >> > > topic creation completely on the broker and don't have a way to > > > > >> distinguish > > > > >> > > between topic creation requests from the regular clients and the > > > > >> admin, we > > > > >> > > can't support manual topic creation any more. I was thinking > > that > > > > >> another > > > > >> > > way of distinguishing the clients making the topic creation > > requests > > > > >> is > > > > >> > > using clientId. For example, the admin tool can set it to > > something > > > > >> like > > > > >> > > admin and the broker can treat that clientId specially. > > > > >> > > > > > > >> > > Also, there is a related discussion in KAFKA-2020. Currently, > > we do > > > > >> the > > > > >> > > following in TopicMetadataResponse: > > > > >> > > > > > > >> > > 1. If leader is not available, we set the partition level error > > code > > > > >> to > > > > >> > > LeaderNotAvailable. > > > > >> > > 2. If a non-leader replica is not available, we take that > > replica > > > > out > > > > >> of > > > > >> > > the assigned replica list and isr in the response. As an > > indication > > > > >> for > > > > >> > > doing that, we set the partition level error code to > > > > >> ReplicaNotAvailable. > > > > >> > > > > > > >> > > This has a few problems. First, ReplicaNotAvailable probably > > > > >> shouldn't be > > > > >> > > an error, at least for the normal producer/consumer clients that > > > > just > > > > >> want > > > > >> > > to find out the leader. Second, it can happen that both the > > leader > > > > and > > > > >> > > another replica are not available at the same time. There is no > > > > error > > > > >> code > > > > >> > > to indicate both. Third, even if a replica is not available, > > it's > > > > >> still > > > > >> > > useful to return its replica id since some clients (e.g. admin > > tool) > > > > >> may > > > > >> > > still make use of it. > > > > >> > > > > > > >> > > One way to address this issue is to always return the replica > > id for > > > > >> > > leader, assigned replicas, and isr regardless of whether the > > > > >> corresponding > > > > >> > > broker is live or not. Since we also return the list of live > > > > brokers, > > > > >> the > > > > >> > > client can figure out whether a leader or a replica is live or > > not > > > > >> and act > > > > >> > > accordingly. This way, we don't need to set the partition level > > > > error > > > > >> code > > > > >> > > when the leader or a replica is not available. This doesn't > > change > > > > >> the wire > > > > >> > > protocol, but does change the semantics. Since we are evolving > > the > > > > >> protocol > > > > >> > > of TopicMetadataRequest here, we can potentially piggyback the > > > > change. > > > > >> > > > > > > >> > > 102.1 For those types of errors due to invalid input, shouldn't > > we > > > > >> just > > > > >> > > guard it at parameter validation time and throw > > > > >> InvalidArgumentException > > > > >> > > without even sending the request to the broker? > > > > >> > > > > > > >> > > Thanks, > > > > >> > > > > > > >> > > Jun > > > > >> > > > > > > >> > > > > > > >> > > On Mon, Mar 16, 2015 at 10:37 AM, Andrii Biletskyi < > > > > >> > > andrii.bilets...@stealth.ly> wrote: > > > > >> > > > > > > >> > > > Jun, > > > > >> > > > > > > > >> > > > Answering your questions: > > > > >> > > > > > > > >> > > > 101. If I understand you correctly, you are saying future > > producer > > > > >> > > versions > > > > >> > > > (which > > > > >> > > > will be ported to TMR_V1) won't be able to automatically > > create > > > > >> topic (if > > > > >> > > > we > > > > >> > > > unconditionally remove topic creation from there). But we > > need to > > > > >> this > > > > >> > > > preserve logic. > > > > >> > > > Ok, about your proposal: I'm not a big fan too, when it comes > > to > > > > >> > > > differentiating > > > > >> > > > clients directly in protocol schema. And also I'm not sure I > > > > >> understand > > > > >> > > at > > > > >> > > > all why > > > > >> > > > auto.create.topics.enable is a server side configuration. Can > > we > > > > >> > > deprecate > > > > >> > > > this setting > > > > >> > > > in future versions, add this setting to producer and based on > > that > > > > >> upon > > > > >> > > > receiving > > > > >> > > > UnknownTopic create topic explicitly by a separate producer > > call > > > > via > > > > >> > > > adminClient? > > > > >> > > > > > > > >> > > > 102.1. Hm, yes. It's because we want to support batching and > > at > > > > the > > > > >> same > > > > >> > > > time we > > > > >> > > > want to give descriptive error messages for clients. Since > > > > >> AdminClient > > > > >> > > > holds the context > > > > >> > > > to construct such messages (e.g. AdminClient layer can know > > that > > > > >> > > > InvalidArgumentsCode > > > > >> > > > means two cases: either invalid number - e.g. -1; or > > > > >> replication-factor > > > > >> > > was > > > > >> > > > provided while > > > > >> > > > partitions argument wasn't) - I wrapped responses in > > Exceptions. > > > > >> But I'm > > > > >> > > > open to any > > > > >> > > > other ideas, this was just initial version. > > > > >> > > > 102.2. Yes, I agree. I'll change that to probably some other > > dto. > > > > >> > > > > > > > >> > > > Thanks, > > > > >> > > > Andrii Biletskyi > > > > >> > > > > > > > >> > > > On Fri, Mar 13, 2015 at 7:16 PM, Jun Rao <j...@confluent.io> > > > > wrote: > > > > >> > > > > > > > >> > > > > Andrii, > > > > >> > > > > > > > > >> > > > > 101. That's what I was thinking too, but it may not be that > > > > >> simple. In > > > > >> > > > > TopicMetadataRequest_V1, > > > > >> > > > > we can let it not trigger auto topic creation. Then, in the > > > > >> producer > > > > >> > > > side, > > > > >> > > > > if it gets an UnknownTopicException, it can explicitly > > issue a > > > > >> > > > > createTopicRequest for auto topic creation. On the consumer > > > > side, > > > > >> it > > > > >> > > will > > > > >> > > > > never issue createTopicRequest. This works when auto topic > > > > >> creation is > > > > >> > > > > enabled on the broker side. However, I am not sure how > > things > > > > >> will work > > > > >> > > > > when auto topic creation is disabled on the broker side. In > > this > > > > >> case, > > > > >> > > we > > > > >> > > > > want to have a way to manually create a topic, potentially > > > > through > > > > >> > > admin > > > > >> > > > > commands. However, then we need a way to distinguish > > > > >> createTopicRequest > > > > >> > > > > issued from the producer clients and the admin tools. May > > be we > > > > >> can > > > > >> > > add a > > > > >> > > > > new field in createTopicRequest and set it differently in > > the > > > > >> producer > > > > >> > > > > client and the admin client. However, I am not sure if > > that's > > > > the > > > > >> best > > > > >> > > > > approach. > > > > >> > > > > > > > > >> > > > > 2. Yes, refactoring existing requests is a non-trivial > > amount of > > > > >> work. > > > > >> > > I > > > > >> > > > > posted some comments in KAFKA-1927. We will probably have > > to fix > > > > >> > > > KAFKA-1927 > > > > >> > > > > first, before adding the new logic in KAFKA-1694. > > Otherwise, the > > > > >> > > changes > > > > >> > > > > will be too big. > > > > >> > > > > > > > > >> > > > > 102. About the AdminClient: > > > > >> > > > > 102.1. It's a bit weird that we return exception in the > > api. It > > > > >> seems > > > > >> > > > that > > > > >> > > > > we should either return error code or throw an exception > > when > > > > >> getting > > > > >> > > the > > > > >> > > > > response state. > > > > >> > > > > 102.2. We probably shouldn't explicitly use the request > > object > > > > in > > > > >> the > > > > >> > > > api. > > > > >> > > > > Not every request evolution requires an api change. > > > > >> > > > > > > > > >> > > > > Thanks, > > > > >> > > > > > > > > >> > > > > Jun > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > On Fri, Mar 13, 2015 at 4:08 AM, Andrii Biletskyi < > > > > >> > > > > andrii.bilets...@stealth.ly> wrote: > > > > >> > > > > > > > > >> > > > > > Jun, > > > > >> > > > > > > > > > >> > > > > > Thanks for you comments. Answers inline: > > > > >> > > > > > > > > > >> > > > > > 100. There are a few fields such as ReplicaAssignment, > > > > >> > > > > > > ReassignPartitionRequest, > > > > >> > > > > > > and PartitionsSerialized that are represented as a > > string, > > > > but > > > > >> > > > contain > > > > >> > > > > > > composite structures in json. Could we flatten them out > > > > >> directly in > > > > >> > > > the > > > > >> > > > > > > protocol definition as arrays/records? > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > Yes, now with Admin Client this looks a bit weird. My > > initial > > > > >> > > > motivation > > > > >> > > > > > was: > > > > >> > > > > > ReassignPartitionCommand accepts input in json, we want to > > > > >> remain > > > > >> > > > tools' > > > > >> > > > > > interfaces unchanged, where possible. > > > > >> > > > > > If we port it to deserialized format, in CLI (/tools > > project) > > > > >> we will > > > > >> > > > > have > > > > >> > > > > > to add some > > > > >> > > > > > json library since /tools is written in java and we'll > > need to > > > > >> > > > > deserialize > > > > >> > > > > > json file > > > > >> > > > > > provided by a user. Can we quickly agree on what this > > library > > > > >> should > > > > >> > > be > > > > >> > > > > > (Jackson, GSON, whatever)? > > > > >> > > > > > > > > > >> > > > > > 101. Does TopicMetadataRequest v1 still trigger auto topic > > > > >> creation? > > > > >> > > > This > > > > >> > > > > > > will be a bit weird now that we have a separate topic > > > > >> creation api. > > > > >> > > > > Have > > > > >> > > > > > > you thought about how the new createTopicRequest and > > > > >> > > > > TopicMetadataRequest > > > > >> > > > > > > v1 will be used in the producer/consumer client, in > > addition > > > > >> to > > > > >> > > admin > > > > >> > > > > > > tools? For example, ideally, we don't want > > > > >> TopicMetadataRequest > > > > >> > > from > > > > >> > > > > the > > > > >> > > > > > > consumer to trigger auto topic creation. > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > I agree, this strange logic should be fixed. I'm not > > confident > > > > >> in > > > > >> > > this > > > > >> > > > > > Kafka part so > > > > >> > > > > > correct me if I'm wrong, but it doesn't look like a hard > > thing > > > > >> to > > > > >> > > do, I > > > > >> > > > > > think we can > > > > >> > > > > > leverage AdminClient for that in Producer and > > unconditionally > > > > >> remove > > > > >> > > > > topic > > > > >> > > > > > creation from the TopicMetadataRequest_V1. > > > > >> > > > > > > > > > >> > > > > > 2. I think Jay meant getting rid of scala classes > > > > >> > > > > > > like HeartbeatRequestAndHeader and > > > > >> HeartbeatResponseAndHeader. We > > > > >> > > did > > > > >> > > > > > that > > > > >> > > > > > > as a stop-gap thing when adding the new requests for the > > > > >> consumers. > > > > >> > > > > > > However, the long term plan is to get rid of all those > > and > > > > >> just > > > > >> > > reuse > > > > >> > > > > the > > > > >> > > > > > > java request/response in the client. Since this KIP > > proposes > > > > >> to > > > > >> > > add a > > > > >> > > > > > > significant number of new requests, perhaps we should > > bite > > > > the > > > > >> > > bullet > > > > >> > > > > to > > > > >> > > > > > > clean up the existing scala requests first before > > adding new > > > > >> ones? > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > Yes, looks like I misunderstood the point of > > > > >> ...RequestAndHeader. > > > > >> > > > Okay, I > > > > >> > > > > > will > > > > >> > > > > > rework that. The only thing is that I don't see any > > example > > > > how > > > > >> it > > > > >> > > was > > > > >> > > > > done > > > > >> > > > > > for at > > > > >> > > > > > least one existing protocol message. Thus, as I > > understand, I > > > > >> have to > > > > >> > > > > think > > > > >> > > > > > how we > > > > >> > > > > > are going to do it. > > > > >> > > > > > Re porting all existing RQ/RP in this patch. Sounds > > > > reasonable, > > > > >> but > > > > >> > > if > > > > >> > > > > it's > > > > >> > > > > > an *obligatory* > > > > >> > > > > > requirement to have Admin KIP done, I'm afraid this can > > be a > > > > >> serious > > > > >> > > > > > blocker for us. > > > > >> > > > > > There are 13 protocol messages and all that would require > > not > > > > >> only > > > > >> > > unit > > > > >> > > > > > tests but quite > > > > >> > > > > > intensive manual testing, no? I'm afraid I'm not the > > right guy > > > > >> to > > > > >> > > cover > > > > >> > > > > > pretty much all > > > > >> > > > > > Kafka core internals :). Let me know your thoughts on this > > > > >> item. Btw > > > > >> > > > > there > > > > >> > > > > > is a ticket to > > > > >> > > > > > follow-up this issue ( > > > > >> > > https://issues.apache.org/jira/browse/KAFKA-2006 > > > > >> > > > ). > > > > >> > > > > > > > > > >> > > > > > Thanks, > > > > >> > > > > > Andrii Biletskyi > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > On Fri, Mar 13, 2015 at 6:40 AM, Jun Rao < > > j...@confluent.io> > > > > >> wrote: > > > > >> > > > > > > > > > >> > > > > > > Andrii, > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > A few more comments. > > > > >> > > > > > > > > > > >> > > > > > > 100. There are a few fields such as ReplicaAssignment, > > > > >> > > > > > > ReassignPartitionRequest, > > > > >> > > > > > > and PartitionsSerialized that are represented as a > > string, > > > > but > > > > >> > > > contain > > > > >> > > > > > > composite structures in json. Could we flatten them out > > > > >> directly in > > > > >> > > > the > > > > >> > > > > > > protocol definition as arrays/records? > > > > >> > > > > > > > > > > >> > > > > > > 101. Does TopicMetadataRequest v1 still trigger auto > > topic > > > > >> > > creation? > > > > >> > > > > This > > > > >> > > > > > > will be a bit weird now that we have a separate topic > > > > >> creation api. > > > > >> > > > > Have > > > > >> > > > > > > you thought about how the new createTopicRequest and > > > > >> > > > > TopicMetadataRequest > > > > >> > > > > > > v1 will be used in the producer/consumer client, in > > addition > > > > >> to > > > > >> > > admin > > > > >> > > > > > > tools? For example, ideally, we don't want > > > > >> TopicMetadataRequest > > > > >> > > from > > > > >> > > > > the > > > > >> > > > > > > consumer to trigger auto topic creation. > > > > >> > > > > > > > > > > >> > > > > > > 2. I think Jay meant getting rid of scala classes > > > > >> > > > > > > like HeartbeatRequestAndHeader and > > > > >> HeartbeatResponseAndHeader. We > > > > >> > > did > > > > >> > > > > > that > > > > >> > > > > > > as a stop-gap thing when adding the new requests for the > > > > >> consumers. > > > > >> > > > > > > However, the long term plan is to get rid of all those > > and > > > > >> just > > > > >> > > reuse > > > > >> > > > > the > > > > >> > > > > > > java request/response in the client. Since this KIP > > proposes > > > > >> to > > > > >> > > add a > > > > >> > > > > > > significant number of new requests, perhaps we should > > bite > > > > the > > > > >> > > bullet > > > > >> > > > > to > > > > >> > > > > > > clean up the existing scala requests first before > > adding new > > > > >> ones? > > > > >> > > > > > > > > > > >> > > > > > > Thanks, > > > > >> > > > > > > > > > > >> > > > > > > Jun > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > On Thu, Mar 12, 2015 at 3:37 PM, Andrii Biletskyi < > > > > >> > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > >> > > > > > > > > > > >> > > > > > > > Hi, > > > > >> > > > > > > > > > > > >> > > > > > > > As said above - I list again all comments from this > > thread > > > > >> so we > > > > >> > > > > > > > can see what's left and finalize all pending issues. > > > > >> > > > > > > > > > > > >> > > > > > > > Comments from Jay: > > > > >> > > > > > > > 1. This is much needed functionality, but there are a > > lot > > > > >> of the > > > > >> > > so > > > > >> > > > > > let's > > > > >> > > > > > > > really think these protocols through. We really want > > to > > > > end > > > > >> up > > > > >> > > > with a > > > > >> > > > > > set > > > > >> > > > > > > > of well thought-out, orthoganol apis. For this reason > > I > > > > >> think it > > > > >> > > is > > > > >> > > > > > > really > > > > >> > > > > > > > important to think through the end state even if that > > > > >> includes > > > > >> > > APIs > > > > >> > > > > we > > > > >> > > > > > > > won't implement in the first phase. > > > > >> > > > > > > > > > > > >> > > > > > > > A: Definitely behind this. Would appreciate if there > > are > > > > >> concrete > > > > >> > > > > > > comments > > > > >> > > > > > > > how this can be improved. > > > > >> > > > > > > > > > > > >> > > > > > > > 2. Let's please please please wait until we have > > switched > > > > >> the > > > > >> > > > server > > > > >> > > > > > over > > > > >> > > > > > > > to the new java protocol definitions. If we add upteen > > > > more > > > > >> ad > > > > >> > > hoc > > > > >> > > > > > scala > > > > >> > > > > > > > objects that is just generating more work for the > > > > >> conversion we > > > > >> > > > know > > > > >> > > > > we > > > > >> > > > > > > > have to do. > > > > >> > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - removed scala protocol > > > > >> classes. > > > > >> > > > > > > > > > > > >> > > > > > > > 3. This proposal introduces a new type of optional > > > > >> parameter. > > > > >> > > This > > > > >> > > > is > > > > >> > > > > > > > inconsistent with everything else in the protocol > > where we > > > > >> use -1 > > > > >> > > > or > > > > >> > > > > > some > > > > >> > > > > > > > other marker value. You could argue either way but > > let's > > > > >> stick > > > > >> > > with > > > > >> > > > > > that > > > > >> > > > > > > > for consistency. For clients that implemented the > > protocol > > > > >> in a > > > > >> > > > > better > > > > >> > > > > > > way > > > > >> > > > > > > > than our scala code these basic primitives are hard to > > > > >> change. > > > > >> > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - removed MaybeOf type > > and > > > > >> changed > > > > >> > > > > > protocol > > > > >> > > > > > > > accordingly. > > > > >> > > > > > > > > > > > >> > > > > > > > 4. ClusterMetadata: This seems to duplicate > > > > >> TopicMetadataRequest > > > > >> > > > > which > > > > >> > > > > > > has > > > > >> > > > > > > > brokers, topics, and partitions. I think we should > > rename > > > > >> that > > > > >> > > > > request > > > > >> > > > > > > > ClusterMetadataRequest (or just MetadataRequest) and > > > > >> include the > > > > >> > > id > > > > >> > > > > of > > > > >> > > > > > > the > > > > >> > > > > > > > controller. Or are there other things we could add > > here? > > > > >> > > > > > > > > > > > >> > > > > > > > A: I agree. Updated the KIP. Let's extends > > TopicMetadata > > > > to > > > > >> > > > version 2 > > > > >> > > > > > and > > > > >> > > > > > > > include controller. > > > > >> > > > > > > > > > > > >> > > > > > > > 5. We have a tendency to try to make a lot of requests > > > > that > > > > >> can > > > > >> > > > only > > > > >> > > > > go > > > > >> > > > > > > to > > > > >> > > > > > > > particular nodes. This adds a lot of burden for client > > > > >> > > > > implementations > > > > >> > > > > > > (it > > > > >> > > > > > > > sounds easy but each discovery can fail in many parts > > so > > > > it > > > > >> ends > > > > >> > > up > > > > >> > > > > > > being a > > > > >> > > > > > > > full state machine to do right). I think we should > > > > consider > > > > >> > > making > > > > >> > > > > > admin > > > > >> > > > > > > > commands and ideally as many of the other apis as > > possible > > > > >> > > > available > > > > >> > > > > on > > > > >> > > > > > > all > > > > >> > > > > > > > brokers and just redirect to the controller on the > > broker > > > > >> side. > > > > >> > > > > Perhaps > > > > >> > > > > > > > there would be a general way to encapsulate this > > > > re-routing > > > > >> > > > behavior. > > > > >> > > > > > > > > > > > >> > > > > > > > A: It's a very interesting idea, but seems there are > > some > > > > >> > > concerns > > > > >> > > > > > about > > > > >> > > > > > > > this > > > > >> > > > > > > > feature (like performance considerations, how this > > will > > > > >> > > complicate > > > > >> > > > > > server > > > > >> > > > > > > > etc). > > > > >> > > > > > > > I believe this shouldn't be a blocker. If this > > feature is > > > > >> > > > implemented > > > > >> > > > > > at > > > > >> > > > > > > > some > > > > >> > > > > > > > point it won't affect Admin changes - at least no > > changes > > > > to > > > > >> > > public > > > > >> > > > > API > > > > >> > > > > > > > will be required. > > > > >> > > > > > > > > > > > >> > > > > > > > 6. We should probably normalize the key value pairs > > used > > > > for > > > > >> > > > configs > > > > >> > > > > > > rather > > > > >> > > > > > > > than embedding a new formatting. So two strings rather > > > > than > > > > >> one > > > > >> > > > with > > > > >> > > > > an > > > > >> > > > > > > > internal equals sign. > > > > >> > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - normalized configs and > > > > >> changed > > > > >> > > > > protocol > > > > >> > > > > > > > accordingly. > > > > >> > > > > > > > > > > > >> > > > > > > > 7. Is the postcondition of these APIs that the > > command has > > > > >> begun > > > > >> > > or > > > > >> > > > > > that > > > > >> > > > > > > > the command has been completed? It is a lot more > > usable if > > > > >> the > > > > >> > > > > command > > > > >> > > > > > > has > > > > >> > > > > > > > been completed so you know that if you create a topic > > and > > > > >> then > > > > >> > > > > publish > > > > >> > > > > > to > > > > >> > > > > > > > it you won't get an exception about there being no > > such > > > > >> topic. > > > > >> > > > > > > > > > > > >> > > > > > > > A: For long running requests (like reassign > > partitions) - > > > > >> the > > > > >> > > post > > > > >> > > > > > > > condition is > > > > >> > > > > > > > command has begun - so we don't block the client. In > > case > > > > >> of your > > > > >> > > > > > > example - > > > > >> > > > > > > > topic commands, this will be refactored and topic > > commands > > > > >> will > > > > >> > > be > > > > >> > > > > > > executed > > > > >> > > > > > > > immediately, since the Controller will serve Admin > > > > requests > > > > >> > > > > > > > (follow-up ticket KAFKA-1777). > > > > >> > > > > > > > > > > > >> > > > > > > > 8. Describe topic and list topics duplicate a lot of > > stuff > > > > >> in the > > > > >> > > > > > > metadata > > > > >> > > > > > > > request. Is there a reason to give back topics marked > > for > > > > >> > > > deletion? I > > > > >> > > > > > > feel > > > > >> > > > > > > > like if we just make the post-condition of the delete > > > > >> command be > > > > >> > > > that > > > > >> > > > > > the > > > > >> > > > > > > > topic is deleted that will get rid of the need for > > this > > > > >> right? > > > > >> > > And > > > > >> > > > it > > > > >> > > > > > > will > > > > >> > > > > > > > be much more intuitive. > > > > >> > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - removed topics marked > > for > > > > >> deletion > > > > >> > > > in > > > > >> > > > > > > > ListTopicsRequest. > > > > >> > > > > > > > > > > > >> > > > > > > > 9. Should we consider batching these requests? We have > > > > >> generally > > > > >> > > > > tried > > > > >> > > > > > to > > > > >> > > > > > > > allow multiple operations to be batched. My suspicion > > is > > > > >> that > > > > >> > > > without > > > > >> > > > > > > this > > > > >> > > > > > > > we will get a lot of code that does something like > > > > >> > > > > > > > for(topic: adminClient.listTopics()) > > > > >> > > > > > > > adminClient.describeTopic(topic) > > > > >> > > > > > > > this code will work great when you test on 5 topics > > but > > > > not > > > > >> do as > > > > >> > > > > well > > > > >> > > > > > if > > > > >> > > > > > > > you have 50k. > > > > >> > > > > > > > > > > > >> > > > > > > > A: Updated the KIP - please check "Topic Admin Schema" > > > > >> section. > > > > >> > > > > > > > > > > > >> > > > > > > > 10. I think we should also discuss how we want to > > expose a > > > > >> > > > > programmatic > > > > >> > > > > > > JVM > > > > >> > > > > > > > client api for these operations. Currently people > > rely on > > > > >> > > > AdminUtils > > > > >> > > > > > > which > > > > >> > > > > > > > is totally sketchy. I think we probably need another > > > > client > > > > >> under > > > > >> > > > > > > clients/ > > > > >> > > > > > > > that exposes administrative functionality. We will > > need > > > > >> this just > > > > >> > > > to > > > > >> > > > > > > > properly test the new apis, I suspect. We should > > figure > > > > out > > > > >> that > > > > >> > > > API. > > > > >> > > > > > > > > > > > >> > > > > > > > A: Updated the KIP - please check "Admin Client" > > section > > > > >> with an > > > > >> > > > > > initial > > > > >> > > > > > > > API proposal. > > > > >> > > > > > > > > > > > >> > > > > > > > 11. The other information that would be really useful > > to > > > > get > > > > >> > > would > > > > >> > > > be > > > > >> > > > > > > > information about partitions--how much data is in the > > > > >> partition, > > > > >> > > > what > > > > >> > > > > > are > > > > >> > > > > > > > the segment offsets, what is the log-end offset (i.e. > > last > > > > >> > > offset), > > > > >> > > > > > what > > > > >> > > > > > > is > > > > >> > > > > > > > the compaction point, etc. I think that done right > > this > > > > >> would be > > > > >> > > > the > > > > >> > > > > > > > successor to the very awkward OffsetRequest we have > > today. > > > > >> > > > > > > > > > > > >> > > > > > > > A: I removed ConsumerGroupOffsetsRequest in the latest > > > > >> patch. I > > > > >> > > > > believe > > > > >> > > > > > > > this should > > > > >> > > > > > > > be resolved in a separate KIP / jira ticket. > > > > >> > > > > > > > > > > > >> > > > > > > > 12. Generally we can do good error handling without > > > > needing > > > > >> > > custom > > > > >> > > > > > > > server-side > > > > >> > > > > > > > messages. I.e. generally the client has the context to > > > > know > > > > >> that > > > > >> > > if > > > > >> > > > > it > > > > >> > > > > > > got > > > > >> > > > > > > > an error that the topic doesn't exist to say "Topic X > > > > >> doesn't > > > > >> > > > exist" > > > > >> > > > > > > rather > > > > >> > > > > > > > than "error code 14" (or whatever). Maybe there are > > > > specific > > > > >> > > cases > > > > >> > > > > > where > > > > >> > > > > > > > this is hard? If we want to add server-side error > > messages > > > > >> we > > > > >> > > > really > > > > >> > > > > do > > > > >> > > > > > > > need to do this in a consistent way across the > > protocol. > > > > >> > > > > > > > > > > > >> > > > > > > > A: Updated the KIP - please check "Protocol Errors" > > > > >> section. I > > > > >> > > > added > > > > >> > > > > > the > > > > >> > > > > > > > comprehensive, fine-grained list of error codes. > > > > >> > > > > > > > > > > > >> > > > > > > > Comments from Guozhang: > > > > >> > > > > > > > 13. Describe topic request: it would be great to go > > beyond > > > > >> just > > > > >> > > > > > batching > > > > >> > > > > > > on > > > > >> > > > > > > > topic name regex for this request. For example, a very > > > > >> common use > > > > >> > > > > case > > > > >> > > > > > of > > > > >> > > > > > > > the topic command is to list all topics whose config > > A's > > > > >> value is > > > > >> > > > B. > > > > >> > > > > > With > > > > >> > > > > > > > topic name regex then we have to first retrieve > > __all__ > > > > >> topics's > > > > >> > > > > > > > description info and then filter at the client end, > > which > > > > >> will > > > > >> > > be a > > > > >> > > > > > huge > > > > >> > > > > > > > burden on ZK. > > > > >> > > > > > > > AND > > > > >> > > > > > > > 14. Config K-Vs in create topic: this is related to > > the > > > > >> previous > > > > >> > > > > point; > > > > >> > > > > > > > maybe we can add another metadata K-V or just a > > metadata > > > > >> string > > > > >> > > > along > > > > >> > > > > > > side > > > > >> > > > > > > > with config K-V in create topic like we did for offset > > > > >> commit > > > > >> > > > > request. > > > > >> > > > > > > This > > > > >> > > > > > > > field can be quite useful in storing information like > > > > >> "owner" of > > > > >> > > > the > > > > >> > > > > > > topic > > > > >> > > > > > > > who issue the create command, etc, which is quite > > > > important > > > > >> for a > > > > >> > > > > > > > multi-tenant setting. Then in the describe topic > > request > > > > we > > > > >> can > > > > >> > > > also > > > > >> > > > > > > batch > > > > >> > > > > > > > on regex of the metadata field. > > > > >> > > > > > > > > > > > >> > > > > > > > A: As discussed it is very interesting but can be > > > > >> implemented > > > > >> > > later > > > > >> > > > > > after > > > > >> > > > > > > > we have some basic functionality there. > > > > >> > > > > > > > > > > > >> > > > > > > > 15. Today all the admin operations are async in the > > sense > > > > >> that > > > > >> > > > > command > > > > >> > > > > > > will > > > > >> > > > > > > > return once it is written in ZK, and that is why we > > need > > > > >> extra > > > > >> > > > > > > verification > > > > >> > > > > > > > like testUtil.waitForTopicCreated() / verify partition > > > > >> > > reassignment > > > > >> > > > > > > > request, etc. With admin requests we could add a flag > > to > > > > >> enable / > > > > >> > > > > > disable > > > > >> > > > > > > > synchronous requests; when it is turned on, the > > response > > > > >> will not > > > > >> > > > > > return > > > > >> > > > > > > > until the request has been completed. And for async > > > > >> requests we > > > > >> > > can > > > > >> > > > > > add a > > > > >> > > > > > > > "token" field in the response, and then only need a > > > > general > > > > >> > > "admin > > > > >> > > > > > > > verification request" with the given token to check > > if the > > > > >> async > > > > >> > > > > > request > > > > >> > > > > > > > has been completed. > > > > >> > > > > > > > > > > > >> > > > > > > > A: I see your point. My idea was to provide specific > > > > >> > > > Verify...Request > > > > >> > > > > > per > > > > >> > > > > > > > each > > > > >> > > > > > > > long running request, where needed. We can do it the > > way > > > > you > > > > >> > > > suggest. > > > > >> > > > > > The > > > > >> > > > > > > > only > > > > >> > > > > > > > concern is that introducing a token we again will make > > > > >> schema > > > > >> > > > > > "dynamic". > > > > >> > > > > > > We > > > > >> > > > > > > > wanted > > > > >> > > > > > > > to do similar thing introducing single AdminRequest > > for > > > > all > > > > >> topic > > > > >> > > > > > > commands > > > > >> > > > > > > > but rejected > > > > >> > > > > > > > this idea because we wanted to have schema defined. So > > > > this > > > > >> is > > > > >> > > > more a > > > > >> > > > > > > > choice between: > > > > >> > > > > > > > a) have fixed schema but introduce each time new > > > > >> Verify...Request > > > > >> > > > for > > > > >> > > > > > > > long-running requests > > > > >> > > > > > > > b) use one request for verification but generalize it > > with > > > > >> token > > > > >> > > > > > > > I'm fine with whatever decision community come to. > > Just > > > > let > > > > >> me > > > > >> > > know > > > > >> > > > > > your > > > > >> > > > > > > > thoughts. > > > > >> > > > > > > > > > > > >> > > > > > > > Comment from Gwen: > > > > >> > > > > > > > 16. Specifically for ownership, I think the plan is > > to add > > > > >> ACL > > > > >> > > (it > > > > >> > > > > > sounds > > > > >> > > > > > > > like you are describing ACL) via an external system > > > > (Argus, > > > > >> > > > Sentry). > > > > >> > > > > > > > I remember KIP-11 described this, but I can't find > > the KIP > > > > >> any > > > > >> > > > > longer. > > > > >> > > > > > > > > > > > >> > > > > > > > A: Okay, no problem. Not sure though how we are going > > to > > > > >> handle > > > > >> > > it. > > > > >> > > > > > Wait > > > > >> > > > > > > > which KIP > > > > >> > > > > > > > will be committed first and include changes to > > > > >> TopicMetadata from > > > > >> > > > the > > > > >> > > > > > > later > > > > >> > > > > > > > one? > > > > >> > > > > > > > Anyway, I added this note to "Open Questions" section > > so > > > > we > > > > >> don't > > > > >> > > > > miss > > > > >> > > > > > > this > > > > >> > > > > > > > piece. > > > > >> > > > > > > > > > > > >> > > > > > > > Thanks, > > > > >> > > > > > > > Andrii Biletskyi > > > > >> > > > > > > > > > > > >> > > > > > > > On Fri, Mar 13, 2015 at 12:34 AM, Andrii Biletskyi < > > > > >> > > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > >> > > > > > > > > > > > >> > > > > > > > > Hi all, > > > > >> > > > > > > > > > > > > >> > > > > > > > > Today I uploaded the patch that covers some of the > > > > >> discussed > > > > >> > > and > > > > >> > > > > > agreed > > > > >> > > > > > > > > items: > > > > >> > > > > > > > > - removed MaybeOf optional type > > > > >> > > > > > > > > - switched to java protocol definitions > > > > >> > > > > > > > > - simplified messages (normalized configs, removed > > topic > > > > >> marked > > > > >> > > > for > > > > >> > > > > > > > > deletion) > > > > >> > > > > > > > > > > > > >> > > > > > > > > I also updated the KIP-4 with respective changes and > > > > >> wrote down > > > > >> > > > my > > > > >> > > > > > > > > proposal for > > > > >> > > > > > > > > pending items: > > > > >> > > > > > > > > - Batch Admin Operations -> updated Wire Protocol > > schema > > > > >> > > proposal > > > > >> > > > > > > > > - Remove ClusterMetadata -> changed to extend > > > > >> > > > TopicMetadataRequest > > > > >> > > > > > > > > - Admin Client -> updated my initial proposal to > > reflect > > > > >> > > batching > > > > >> > > > > > > > > - Error codes -> proposed fine-grained error code > > > > instead > > > > >> of > > > > >> > > > > > > > > AdminRequestFailed > > > > >> > > > > > > > > > > > > >> > > > > > > > > I will also send a separate email to cover all > > comments > > > > >> from > > > > >> > > this > > > > >> > > > > > > thread. > > > > >> > > > > > > > > > > > > >> > > > > > > > > Thanks, > > > > >> > > > > > > > > Andrii Biletskyi > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > On Thu, Mar 12, 2015 at 9:26 PM, Gwen Shapira < > > > > >> > > > > gshap...@cloudera.com > > > > >> > > > > > > > > > > >> > > > > > > > > wrote: > > > > >> > > > > > > > > > > > > >> > > > > > > > >> Found KIP-11 ( > > > > >> > > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface > > > > >> > > > > > > > >> ) > > > > >> > > > > > > > >> It actually specifies changes to the Metadata > > protocol, > > > > >> so > > > > >> > > > making > > > > >> > > > > > sure > > > > >> > > > > > > > >> both KIPs are consistent in this regard will be > > good. > > > > >> > > > > > > > >> > > > > >> > > > > > > > >> On Thu, Mar 12, 2015 at 12:21 PM, Gwen Shapira < > > > > >> > > > > > gshap...@cloudera.com > > > > >> > > > > > > > > > > > >> > > > > > > > >> wrote: > > > > >> > > > > > > > >> > Specifically for ownership, I think the plan is > > to > > > > add > > > > >> ACL > > > > >> > > (it > > > > >> > > > > > > sounds > > > > >> > > > > > > > >> > like you are describing ACL) via an external > > system > > > > >> (Argus, > > > > >> > > > > > Sentry). > > > > >> > > > > > > > >> > I remember KIP-11 described this, but I can't > > find > > > > the > > > > >> KIP > > > > >> > > any > > > > >> > > > > > > longer. > > > > >> > > > > > > > >> > > > > > >> > > > > > > > >> > Regardless, I think KIP-4 focuses on getting > > > > >> information > > > > >> > > that > > > > >> > > > > > > already > > > > >> > > > > > > > >> > exists from Kafka brokers, not on adding > > information > > > > >> that > > > > >> > > > > perhaps > > > > >> > > > > > > > >> > should exist but doesn't yet? > > > > >> > > > > > > > >> > > > > > >> > > > > > > > >> > Gwen > > > > >> > > > > > > > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > >> > On Thu, Mar 12, 2015 at 6:37 AM, Guozhang Wang < > > > > >> > > > > > wangg...@gmail.com> > > > > >> > > > > > > > >> wrote: > > > > >> > > > > > > > >> >> Folks, > > > > >> > > > > > > > >> >> > > > > >> > > > > > > > >> >> Just want to elaborate a bit more on the > > > > create-topic > > > > >> > > > metadata > > > > >> > > > > > and > > > > >> > > > > > > > >> batching > > > > >> > > > > > > > >> >> describe-topic based on config / metadata in my > > > > >> previous > > > > >> > > > email > > > > >> > > > > as > > > > >> > > > > > > we > > > > >> > > > > > > > >> work > > > > >> > > > > > > > >> >> on KAFKA-1694. The main motivation is to have > > some > > > > >> sort of > > > > >> > > > > topic > > > > >> > > > > > > > >> management > > > > >> > > > > > > > >> >> mechanisms, which I think is quite important in > > a > > > > >> > > > multi-tenant > > > > >> > > > > / > > > > >> > > > > > > > cloud > > > > >> > > > > > > > >> >> architecture: today anyone can create topics in > > a > > > > >> shared > > > > >> > > > Kafka > > > > >> > > > > > > > >> cluster, but > > > > >> > > > > > > > >> >> there is no concept or "ownership" of topics > > that > > > > are > > > > >> > > created > > > > >> > > > > by > > > > >> > > > > > > > >> different > > > > >> > > > > > > > >> >> users. For example, at LinkedIn we basically > > > > >> distinguish > > > > >> > > > topic > > > > >> > > > > > > owners > > > > >> > > > > > > > >> via > > > > >> > > > > > > > >> >> some casual topic name prefix, which is a bit > > > > awkward > > > > >> and > > > > >> > > > does > > > > >> > > > > > not > > > > >> > > > > > > > fly > > > > >> > > > > > > > >> as > > > > >> > > > > > > > >> >> we scale our customers. It would be great to use > > > > >> > > > > describe-topics > > > > >> > > > > > > such > > > > >> > > > > > > > >> as: > > > > >> > > > > > > > >> >> > > > > >> > > > > > > > >> >> Describe all topics that is created by me. > > > > >> > > > > > > > >> >> > > > > >> > > > > > > > >> >> Describe all topics whose retention time is > > > > overriden > > > > >> to X. > > > > >> > > > > > > > >> >> > > > > >> > > > > > > > >> >> Describe all topics whose writable group include > > > > user > > > > >> Y > > > > >> > > (this > > > > >> > > > > is > > > > >> > > > > > > > >> related to > > > > >> > > > > > > > >> >> authorization), etc.. > > > > >> > > > > > > > >> >> > > > > >> > > > > > > > >> >> One possible way to achieve this is to add a > > > > metadata > > > > >> file > > > > >> > > in > > > > >> > > > > the > > > > >> > > > > > > > >> >> create-topic request, whose value will also be > > > > >> written ZK > > > > >> > > as > > > > >> > > > we > > > > >> > > > > > > > create > > > > >> > > > > > > > >> the > > > > >> > > > > > > > >> >> topic; then describe-topics can choose to batch > > > > topics > > > > >> > > based > > > > >> > > > on > > > > >> > > > > > 1) > > > > >> > > > > > > > name > > > > >> > > > > > > > >> >> regex, 2) config K-V matching, 3) metadata > > regex, > > > > etc. > > > > >> > > > > > > > >> >> > > > > >> > > > > > > > >> >> Thoughts? > > > > >> > > > > > > > >> >> > > > > >> > > > > > > > >> >> Guozhang > > > > >> > > > > > > > >> >> > > > > >> > > > > > > > >> >> On Thu, Mar 5, 2015 at 4:37 PM, Guozhang Wang < > > > > >> > > > > > wangg...@gmail.com> > > > > >> > > > > > > > >> wrote: > > > > >> > > > > > > > >> >> > > > > >> > > > > > > > >> >>> Thanks for the updated wiki. A few comments > > below: > > > > >> > > > > > > > >> >>> > > > > >> > > > > > > > >> >>> 1. Error description in response: I think if > > some > > > > >> > > errorCode > > > > >> > > > > > could > > > > >> > > > > > > > >> indicate > > > > >> > > > > > > > >> >>> several different error cases then we should > > really > > > > >> change > > > > >> > > > it > > > > >> > > > > to > > > > >> > > > > > > > >> multiple > > > > >> > > > > > > > >> >>> codes. In general the errorCode itself would be > > > > >> precise > > > > >> > > and > > > > >> > > > > > > > >> sufficient for > > > > >> > > > > > > > >> >>> describing the server side errors. > > > > >> > > > > > > > >> >>> > > > > >> > > > > > > > >> >>> 2. Describe topic request: it would be great > > to go > > > > >> beyond > > > > >> > > > just > > > > >> > > > > > > > >> batching on > > > > >> > > > > > > > >> >>> topic name regex for this request. For > > example, a > > > > >> very > > > > >> > > > common > > > > >> > > > > > use > > > > >> > > > > > > > >> case of > > > > >> > > > > > > > >> >>> the topic command is to list all topics whose > > > > config > > > > >> A's > > > > >> > > > value > > > > >> > > > > > is > > > > >> > > > > > > B. > > > > >> > > > > > > > >> With > > > > >> > > > > > > > >> >>> topic name regex then we have to first retrieve > > > > >> __all__ > > > > >> > > > > topics's > > > > >> > > > > > > > >> >>> description info and then filter at the client > > end, > > > > >> which > > > > >> > > > will > > > > >> > > > > > be > > > > >> > > > > > > a > > > > >> > > > > > > > >> huge > > > > >> > > > > > > > >> >>> burden on ZK. > > > > >> > > > > > > > >> >>> > > > > >> > > > > > > > >> >>> 3. Config K-Vs in create topic: this is > > related to > > > > >> the > > > > >> > > > > previous > > > > >> > > > > > > > point; > > > > >> > > > > > > > >> >>> maybe we can add another metadata K-V or just a > > > > >> metadata > > > > >> > > > > string > > > > >> > > > > > > > along > > > > >> > > > > > > > >> side > > > > >> > > > > > > > >> >>> with config K-V in create topic like we did for > > > > >> offset > > > > >> > > > commit > > > > >> > > > > > > > >> request. This > > > > >> > > > > > > > >> >>> field can be quite useful in storing > > information > > > > like > > > > >> > > > "owner" > > > > >> > > > > of > > > > >> > > > > > > the > > > > >> > > > > > > > >> topic > > > > >> > > > > > > > >> >>> who issue the create command, etc, which is > > quite > > > > >> > > important > > > > >> > > > > for > > > > >> > > > > > a > > > > >> > > > > > > > >> >>> multi-tenant setting. Then in the describe > > topic > > > > >> request > > > > >> > > we > > > > >> > > > > can > > > > >> > > > > > > also > > > > >> > > > > > > > >> batch > > > > >> > > > > > > > >> >>> on regex of the metadata field. > > > > >> > > > > > > > >> >>> > > > > >> > > > > > > > >> >>> 4. Today all the admin operations are async in > > the > > > > >> sense > > > > >> > > > that > > > > >> > > > > > > > command > > > > >> > > > > > > > >> will > > > > >> > > > > > > > >> >>> return once it is written in ZK, and that is > > why we > > > > >> need > > > > >> > > > extra > > > > >> > > > > > > > >> verification > > > > >> > > > > > > > >> >>> like testUtil.waitForTopicCreated() / verify > > > > >> partition > > > > >> > > > > > > reassignment > > > > >> > > > > > > > >> >>> request, etc. With admin requests we could add > > a > > > > >> flag to > > > > >> > > > > enable > > > > >> > > > > > / > > > > >> > > > > > > > >> disable > > > > >> > > > > > > > >> >>> synchronous requests; when it is turned on, the > > > > >> response > > > > >> > > > will > > > > >> > > > > > not > > > > >> > > > > > > > >> return > > > > >> > > > > > > > >> >>> until the request has been completed. And for > > async > > > > >> > > requests > > > > >> > > > > we > > > > >> > > > > > > can > > > > >> > > > > > > > >> add a > > > > >> > > > > > > > >> >>> "token" field in the response, and then only > > need a > > > > >> > > general > > > > >> > > > > > "admin > > > > >> > > > > > > > >> >>> verification request" with the given token to > > check > > > > >> if the > > > > >> > > > > async > > > > >> > > > > > > > >> request > > > > >> > > > > > > > >> >>> has been completed. > > > > >> > > > > > > > >> >>> > > > > >> > > > > > > > >> >>> 5. +1 for extending Metadata request to include > > > > >> > > controller / > > > > >> > > > > > > > >> coordinator > > > > >> > > > > > > > >> >>> information, and then we can remove the > > > > >> ConsumerMetadata / > > > > >> > > > > > > > >> ClusterMetadata > > > > >> > > > > > > > >> >>> requests. > > > > >> > > > > > > > >> >>> > > > > >> > > > > > > > >> >>> Guozhang > > > > >> > > > > > > > >> >>> > > > > >> > > > > > > > >> >>> On Tue, Mar 3, 2015 at 10:23 AM, Joel Koshy < > > > > >> > > > > > jjkosh...@gmail.com> > > > > >> > > > > > > > >> wrote: > > > > >> > > > > > > > >> >>> > > > > >> > > > > > > > >> >>>> Thanks for sending that out Joe - I don't > > think I > > > > >> will be > > > > >> > > > > able > > > > >> > > > > > to > > > > >> > > > > > > > >> make > > > > >> > > > > > > > >> >>>> it today, so if notes can be sent out > > afterward > > > > that > > > > >> > > would > > > > >> > > > be > > > > >> > > > > > > > great. > > > > >> > > > > > > > >> >>>> > > > > >> > > > > > > > >> >>>> On Mon, Mar 02, 2015 at 09:16:13AM -0800, Gwen > > > > >> Shapira > > > > >> > > > wrote: > > > > >> > > > > > > > >> >>>> > Thanks for sending this out Joe. Looking > > forward > > > > >> to > > > > >> > > > > chatting > > > > >> > > > > > > with > > > > >> > > > > > > > >> >>>> everyone :) > > > > >> > > > > > > > >> >>>> > > > > > >> > > > > > > > >> >>>> > On Mon, Mar 2, 2015 at 6:46 AM, Joe Stein < > > > > >> > > > > > > joe.st...@stealth.ly> > > > > >> > > > > > > > >> wrote: > > > > >> > > > > > > > >> >>>> > > Hey, I just sent out a google hangout > > invite > > > > to > > > > >> all > > > > >> > > > pmc, > > > > >> > > > > > > > >> committers > > > > >> > > > > > > > >> >>>> and > > > > >> > > > > > > > >> >>>> > > everyone I found working on a KIP. If I > > missed > > > > >> anyone > > > > >> > > > in > > > > >> > > > > > the > > > > >> > > > > > > > >> invite > > > > >> > > > > > > > >> >>>> please > > > > >> > > > > > > > >> >>>> > > let me know and can update it, np. > > > > >> > > > > > > > >> >>>> > > > > > > >> > > > > > > > >> >>>> > > We should do this every Tuesday @ 2pm > > Eastern > > > > >> Time. > > > > >> > > > Maybe > > > > >> > > > > > we > > > > >> > > > > > > > can > > > > >> > > > > > > > >> get > > > > >> > > > > > > > >> >>>> INFRA > > > > >> > > > > > > > >> >>>> > > help to make a google account so we can > > manage > > > > >> > > better? > > > > >> > > > > > > > >> >>>> > > > > > > >> > > > > > > > >> >>>> > > To discuss > > > > >> > > > > > > > >> >>>> > > > > > > >> > > > > > > > >> >>>> > > > > >> > > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > > >> > > > > > > > >> >>>> > > in progress and related JIRA that are > > > > >> interdependent > > > > >> > > > and > > > > >> > > > > > > common > > > > >> > > > > > > > >> work. > > > > >> > > > > > > > >> >>>> > > > > > > >> > > > > > > > >> >>>> > > ~ Joe Stein > > > > >> > > > > > > > >> >>>> > > > > > > >> > > > > > > > >> >>>> > > On Tue, Feb 24, 2015 at 2:59 PM, Jay > > Kreps < > > > > >> > > > > > > > jay.kr...@gmail.com> > > > > >> > > > > > > > >> >>>> wrote: > > > > >> > > > > > > > >> >>>> > > > > > > >> > > > > > > > >> >>>> > >> Let's stay on Google hangouts that will > > also > > > > >> record > > > > >> > > > and > > > > >> > > > > > make > > > > >> > > > > > > > the > > > > >> > > > > > > > >> >>>> sessions > > > > >> > > > > > > > >> >>>> > >> available on youtube. > > > > >> > > > > > > > >> >>>> > >> > > > > >> > > > > > > > >> >>>> > >> -Jay > > > > >> > > > > > > > >> >>>> > >> > > > > >> > > > > > > > >> >>>> > >> On Tue, Feb 24, 2015 at 11:49 AM, Jeff > > > > Holoman > > > > >> < > > > > >> > > > > > > > >> >>>> jholo...@cloudera.com> > > > > >> > > > > > > > >> >>>> > >> wrote: > > > > >> > > > > > > > >> >>>> > >> > > > > >> > > > > > > > >> >>>> > >> > Jay / Joe > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > >> >>>> > >> > We're happy to send out a Webex for > > this > > > > >> purpose. > > > > >> > > We > > > > >> > > > > > could > > > > >> > > > > > > > >> record > > > > >> > > > > > > > >> >>>> the > > > > >> > > > > > > > >> >>>> > >> > sessions if there is interest and > > publish > > > > >> them > > > > >> > > out. > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > >> >>>> > >> > Thanks > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > >> >>>> > >> > Jeff > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > >> >>>> > >> > On Tue, Feb 24, 2015 at 11:28 AM, Jay > > > > Kreps < > > > > >> > > > > > > > >> jay.kr...@gmail.com> > > > > >> > > > > > > > >> >>>> wrote: > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > >> >>>> > >> > > Let's try to get the technical > > hang-ups > > > > >> sorted > > > > >> > > > out, > > > > >> > > > > > > > though. > > > > >> > > > > > > > >> I > > > > >> > > > > > > > >> >>>> really > > > > >> > > > > > > > >> >>>> > >> > think > > > > >> > > > > > > > >> >>>> > >> > > there is some benefit to live > > discussion > > > > vs > > > > >> > > > > writing. I > > > > >> > > > > > > am > > > > >> > > > > > > > >> >>>> hopeful that > > > > >> > > > > > > > >> >>>> > >> if > > > > >> > > > > > > > >> >>>> > >> > > we post instructions and give > > ourselves a > > > > >> few > > > > >> > > > > attempts > > > > >> > > > > > > we > > > > >> > > > > > > > >> can > > > > >> > > > > > > > >> >>>> get it > > > > >> > > > > > > > >> >>>> > >> > > working. > > > > >> > > > > > > > >> >>>> > >> > > > > > > >> > > > > > > > >> >>>> > >> > > Tuesday at that time would work for > > > > >> me...any > > > > >> > > > > > objections? > > > > >> > > > > > > > >> >>>> > >> > > > > > > >> > > > > > > > >> >>>> > >> > > -Jay > > > > >> > > > > > > > >> >>>> > >> > > > > > > >> > > > > > > > >> >>>> > >> > > On Tue, Feb 24, 2015 at 8:18 AM, Joe > > > > Stein > > > > >> < > > > > >> > > > > > > > >> joe.st...@stealth.ly > > > > >> > > > > > > > >> >>>> > > > > > >> > > > > > > > >> >>>> > >> wrote: > > > > >> > > > > > > > >> >>>> > >> > > > > > > >> > > > > > > > >> >>>> > >> > > > Weekly would be great maybe like > > every > > > > >> > > Tuesday ~ > > > > >> > > > > 1pm > > > > >> > > > > > > ET > > > > >> > > > > > > > / > > > > >> > > > > > > > >> 10am > > > > >> > > > > > > > >> >>>> PT > > > > >> > > > > > > > >> >>>> > >> ???? > > > > >> > > > > > > > >> >>>> > >> > > > > > > > >> > > > > > > > >> >>>> > >> > > > I don't mind google hangout but > > there > > > > is > > > > >> > > always > > > > >> > > > > some > > > > >> > > > > > > > >> issue or > > > > >> > > > > > > > >> >>>> > >> whatever > > > > >> > > > > > > > >> >>>> > >> > so > > > > >> > > > > > > > >> >>>> > >> > > > we know the apache irc channel > > works. > > > > We > > > > >> can > > > > >> > > > start > > > > >> > > > > > > there > > > > >> > > > > > > > >> and > > > > >> > > > > > > > >> >>>> see how > > > > >> > > > > > > > >> >>>> > >> it > > > > >> > > > > > > > >> >>>> > >> > > > goes? We can pull transcripts too > > and > > > > >> > > associate > > > > >> > > > to > > > > >> > > > > > > > >> tickets if > > > > >> > > > > > > > >> >>>> need be > > > > >> > > > > > > > >> >>>> > >> > > makes > > > > >> > > > > > > > >> >>>> > >> > > > it helpful for things. > > > > >> > > > > > > > >> >>>> > >> > > > > > > > >> > > > > > > > >> >>>> > >> > > > ~ Joestein > > > > >> > > > > > > > >> >>>> > >> > > > > > > > >> > > > > > > > >> >>>> > >> > > > On Tue, Feb 24, 2015 at 11:10 AM, > > Jay > > > > >> Kreps < > > > > >> > > > > > > > >> >>>> jay.kr...@gmail.com> > > > > >> > > > > > > > >> >>>> > >> > wrote: > > > > >> > > > > > > > >> >>>> > >> > > > > > > > >> > > > > > > > >> >>>> > >> > > > > We'd talked about doing a Google > > > > >> Hangout to > > > > >> > > > chat > > > > >> > > > > > > about > > > > >> > > > > > > > >> this. > > > > >> > > > > > > > >> >>>> What > > > > >> > > > > > > > >> >>>> > >> > about > > > > >> > > > > > > > >> >>>> > >> > > > > generalizing that a little > > > > further...I > > > > >> > > > actually > > > > >> > > > > > > think > > > > >> > > > > > > > it > > > > >> > > > > > > > >> >>>> would be > > > > >> > > > > > > > >> >>>> > >> > good > > > > >> > > > > > > > >> >>>> > >> > > > for > > > > >> > > > > > > > >> >>>> > >> > > > > everyone spending a reasonable > > chunk > > > > of > > > > >> > > their > > > > >> > > > > week > > > > >> > > > > > > on > > > > >> > > > > > > > >> Kafka > > > > >> > > > > > > > >> >>>> stuff > > > > >> > > > > > > > >> >>>> > >> to > > > > >> > > > > > > > >> >>>> > >> > > > maybe > > > > >> > > > > > > > >> >>>> > >> > > > > sync up once a week. I think we > > could > > > > >> use > > > > >> > > time > > > > >> > > > > to > > > > >> > > > > > > talk > > > > >> > > > > > > > >> >>>> through > > > > >> > > > > > > > >> >>>> > >> design > > > > >> > > > > > > > >> >>>> > >> > > > > stuff, make sure we are on top of > > > > code > > > > >> > > > reviews, > > > > >> > > > > > talk > > > > >> > > > > > > > >> through > > > > >> > > > > > > > >> >>>> any > > > > >> > > > > > > > >> >>>> > >> > tricky > > > > >> > > > > > > > >> >>>> > >> > > > > issues, etc. > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > We can make it publicly > > available so > > > > >> that > > > > >> > > any > > > > >> > > > > one > > > > >> > > > > > > can > > > > >> > > > > > > > >> follow > > > > >> > > > > > > > >> >>>> along > > > > >> > > > > > > > >> >>>> > >> > who > > > > >> > > > > > > > >> >>>> > >> > > > > likes. > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > Any interest in doing this? If so > > > > I'll > > > > >> try > > > > >> > > to > > > > >> > > > > set > > > > >> > > > > > it > > > > >> > > > > > > > up > > > > >> > > > > > > > >> >>>> starting > > > > >> > > > > > > > >> >>>> > >> next > > > > >> > > > > > > > >> >>>> > >> > > > week. > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > -Jay > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > On Tue, Feb 24, 2015 at 3:57 AM, > > > > Andrii > > > > >> > > > > Biletskyi > > > > >> > > > > > < > > > > >> > > > > > > > >> >>>> > >> > > > > andrii.bilets...@stealth.ly> > > wrote: > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > Hi all, > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > I've updated KIP page, fixed / > > > > >> aligned > > > > >> > > > > document > > > > >> > > > > > > > >> structure. > > > > >> > > > > > > > >> >>>> Also I > > > > >> > > > > > > > >> >>>> > >> > > added > > > > >> > > > > > > > >> >>>> > >> > > > > > some > > > > >> > > > > > > > >> >>>> > >> > > > > > very initial proposal for > > > > >> AdminClient so > > > > >> > > we > > > > >> > > > > have > > > > >> > > > > > > > >> something > > > > >> > > > > > > > >> >>>> to > > > > >> > > > > > > > >> >>>> > >> start > > > > >> > > > > > > > >> >>>> > >> > > > from > > > > >> > > > > > > > >> >>>> > >> > > > > > while > > > > >> > > > > > > > >> >>>> > >> > > > > > discussing the KIP. > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > >> >>>> > >> > > > > >> > > > > > > > >> >>>> > > > > >> > > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > Thanks, > > > > >> > > > > > > > >> >>>> > >> > > > > > Andrii Biletskyi > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > On Wed, Feb 18, 2015 at 9:01 > > PM, > > > > >> Andrii > > > > >> > > > > > Biletskyi > > > > >> > > > > > > < > > > > >> > > > > > > > >> >>>> > >> > > > > > andrii.bilets...@stealth.ly> > > > > wrote: > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Jay, > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Re error messages: you are > > right, > > > > >> in > > > > >> > > most > > > > >> > > > > > cases > > > > >> > > > > > > > >> client > > > > >> > > > > > > > >> >>>> will > > > > >> > > > > > > > >> >>>> > >> have > > > > >> > > > > > > > >> >>>> > >> > > > enough > > > > >> > > > > > > > >> >>>> > >> > > > > > > context to show descriptive > > error > > > > >> > > message. > > > > >> > > > > My > > > > >> > > > > > > > >> concern is > > > > >> > > > > > > > >> >>>> that > > > > >> > > > > > > > >> >>>> > >> we > > > > >> > > > > > > > >> >>>> > >> > > will > > > > >> > > > > > > > >> >>>> > >> > > > > > have > > > > >> > > > > > > > >> >>>> > >> > > > > > > to > > > > >> > > > > > > > >> >>>> > >> > > > > > > add lots of new error codes > > for > > > > >> each > > > > >> > > > > possible > > > > >> > > > > > > > >> error. Of > > > > >> > > > > > > > >> >>>> course, > > > > >> > > > > > > > >> >>>> > >> > we > > > > >> > > > > > > > >> >>>> > >> > > > > could > > > > >> > > > > > > > >> >>>> > >> > > > > > > reuse > > > > >> > > > > > > > >> >>>> > >> > > > > > > some of existing like > > > > >> > > > > > > UknownTopicOrPartitionCode, > > > > >> > > > > > > > >> but we > > > > >> > > > > > > > >> >>>> will > > > > >> > > > > > > > >> >>>> > >> > also > > > > >> > > > > > > > >> >>>> > >> > > > need > > > > >> > > > > > > > >> >>>> > >> > > > > > to > > > > >> > > > > > > > >> >>>> > >> > > > > > > add smth like: > > > > >> TopicAlreadyExistsCode, > > > > >> > > > > > > > >> >>>> TopicConfigInvalid (both > > > > >> > > > > > > > >> >>>> > >> > for > > > > >> > > > > > > > >> >>>> > >> > > > > topic > > > > >> > > > > > > > >> >>>> > >> > > > > > > name and config, and probably > > > > user > > > > >> would > > > > >> > > > > like > > > > >> > > > > > to > > > > >> > > > > > > > >> know > > > > >> > > > > > > > >> >>>> what > > > > >> > > > > > > > >> >>>> > >> > exactly > > > > >> > > > > > > > >> >>>> > >> > > > > > > is wrong in his config), > > > > >> > > > > > > InvalidReplicaAssignment, > > > > >> > > > > > > > >> >>>> > >> InternalError > > > > >> > > > > > > > >> >>>> > >> > > > (e.g. > > > > >> > > > > > > > >> >>>> > >> > > > > > > zookeeper failure) etc. > > > > >> > > > > > > > >> >>>> > >> > > > > > > And this is only for > > > > TopicCommand, > > > > >> we > > > > >> > > will > > > > >> > > > > > also > > > > >> > > > > > > > >> need to > > > > >> > > > > > > > >> >>>> add > > > > >> > > > > > > > >> >>>> > >> > similar > > > > >> > > > > > > > >> >>>> > >> > > > > stuff > > > > >> > > > > > > > >> >>>> > >> > > > > > > for > > > > >> > > > > > > > >> >>>> > >> > > > > > > ReassignPartitions, > > > > >> PreferredReplica. So > > > > >> > > > > we'll > > > > >> > > > > > > end > > > > >> > > > > > > > >> up > > > > >> > > > > > > > >> >>>> with a > > > > >> > > > > > > > >> >>>> > >> > large > > > > >> > > > > > > > >> >>>> > >> > > > list > > > > >> > > > > > > > >> >>>> > >> > > > > > of > > > > >> > > > > > > > >> >>>> > >> > > > > > > error codes, used only in > > Admin > > > > >> > > protocol. > > > > >> > > > > > > > >> >>>> > >> > > > > > > Having said that, I agree my > > > > >> proposal is > > > > >> > > > not > > > > >> > > > > > > > >> consistent > > > > >> > > > > > > > >> >>>> with > > > > >> > > > > > > > >> >>>> > >> > other > > > > >> > > > > > > > >> >>>> > >> > > > > cases. > > > > >> > > > > > > > >> >>>> > >> > > > > > > Maybe we can find better > > solution > > > > >> or > > > > >> > > > > something > > > > >> > > > > > > > >> >>>> in-between. > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Re Hangout chat: I think it > > is a > > > > >> great > > > > >> > > > idea. > > > > >> > > > > > > This > > > > >> > > > > > > > >> way we > > > > >> > > > > > > > >> >>>> can > > > > >> > > > > > > > >> >>>> > >> move > > > > >> > > > > > > > >> >>>> > >> > > on > > > > >> > > > > > > > >> >>>> > >> > > > > > > faster. > > > > >> > > > > > > > >> >>>> > >> > > > > > > Let's agree somehow on > > date/time > > > > so > > > > >> > > people > > > > >> > > > > can > > > > >> > > > > > > > join. > > > > >> > > > > > > > >> >>>> Will work > > > > >> > > > > > > > >> >>>> > >> > for > > > > >> > > > > > > > >> >>>> > >> > > me > > > > >> > > > > > > > >> >>>> > >> > > > > > this > > > > >> > > > > > > > >> >>>> > >> > > > > > > and > > > > >> > > > > > > > >> >>>> > >> > > > > > > next week almost anytime if > > > > agreed > > > > >> in > > > > >> > > > > advance. > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Thanks, > > > > >> > > > > > > > >> >>>> > >> > > > > > > Andrii > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > On Wed, Feb 18, 2015 at 7:09 > > PM, > > > > >> Jay > > > > >> > > > Kreps < > > > > >> > > > > > > > >> >>>> > >> jay.kr...@gmail.com> > > > > >> > > > > > > > >> >>>> > >> > > > > wrote: > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> Hey Andrii, > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > >> > > > > > > > >> >>>> > >> > > > > > >> Generally we can do good > > error > > > > >> handling > > > > >> > > > > > without > > > > >> > > > > > > > >> needing > > > > >> > > > > > > > >> >>>> custom > > > > >> > > > > > > > >> >>>> > >> > > > > > server-side > > > > >> > > > > > > > >> >>>> > >> > > > > > >> messages. I.e. generally the > > > > >> client has > > > > >> > > > the > > > > >> > > > > > > > >> context to > > > > >> > > > > > > > >> >>>> know > > > > >> > > > > > > > >> >>>> > >> that > > > > >> > > > > > > > >> >>>> > >> > > if > > > > >> > > > > > > > >> >>>> > >> > > > it > > > > >> > > > > > > > >> >>>> > >> > > > > > got > > > > >> > > > > > > > >> >>>> > >> > > > > > >> an error that the topic > > doesn't > > > > >> exist > > > > >> > > to > > > > >> > > > > say > > > > >> > > > > > > > >> "Topic X > > > > >> > > > > > > > >> >>>> doesn't > > > > >> > > > > > > > >> >>>> > >> > > exist" > > > > >> > > > > > > > >> >>>> > >> > > > > > >> rather > > > > >> > > > > > > > >> >>>> > >> > > > > > >> than "error code 14" (or > > > > >> whatever). > > > > >> > > Maybe > > > > >> > > > > > there > > > > >> > > > > > > > are > > > > >> > > > > > > > >> >>>> specific > > > > >> > > > > > > > >> >>>> > >> > cases > > > > >> > > > > > > > >> >>>> > >> > > > > where > > > > >> > > > > > > > >> >>>> > >> > > > > > >> this is hard? If we want to > > add > > > > >> > > > server-side > > > > >> > > > > > > error > > > > >> > > > > > > > >> >>>> messages we > > > > >> > > > > > > > >> >>>> > >> > > really > > > > >> > > > > > > > >> >>>> > >> > > > > do > > > > >> > > > > > > > >> >>>> > >> > > > > > >> need to do this in a > > consistent > > > > >> way > > > > >> > > > across > > > > >> > > > > > the > > > > >> > > > > > > > >> protocol. > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > >> > > > > > > > >> >>>> > >> > > > > > >> I still have a bunch of open > > > > >> questions > > > > >> > > > here > > > > >> > > > > > > from > > > > >> > > > > > > > my > > > > >> > > > > > > > >> >>>> previous > > > > >> > > > > > > > >> >>>> > >> > > list. I > > > > >> > > > > > > > >> >>>> > >> > > > > > will > > > > >> > > > > > > > >> >>>> > >> > > > > > >> be out for the next few > > days for > > > > >> Strata > > > > >> > > > > > though. > > > > >> > > > > > > > >> Maybe > > > > >> > > > > > > > >> >>>> we could > > > > >> > > > > > > > >> >>>> > >> > do > > > > >> > > > > > > > >> >>>> > >> > > a > > > > >> > > > > > > > >> >>>> > >> > > > > > Google > > > > >> > > > > > > > >> >>>> > >> > > > > > >> Hangout chat on any open > > issues > > > > >> some > > > > >> > > time > > > > >> > > > > > > towards > > > > >> > > > > > > > >> the > > > > >> > > > > > > > >> >>>> end of > > > > >> > > > > > > > >> >>>> > >> > next > > > > >> > > > > > > > >> >>>> > >> > > > week > > > > >> > > > > > > > >> >>>> > >> > > > > > for > > > > >> > > > > > > > >> >>>> > >> > > > > > >> anyone interested in this > > > > ticket? > > > > >> I > > > > >> > > have > > > > >> > > > a > > > > >> > > > > > > > feeling > > > > >> > > > > > > > >> that > > > > >> > > > > > > > >> >>>> might > > > > >> > > > > > > > >> >>>> > >> > > > progress > > > > >> > > > > > > > >> >>>> > >> > > > > > >> things a little faster than > > > > >> email--I > > > > >> > > > think > > > > >> > > > > we > > > > >> > > > > > > > >> could talk > > > > >> > > > > > > > >> >>>> > >> through > > > > >> > > > > > > > >> >>>> > >> > > > those > > > > >> > > > > > > > >> >>>> > >> > > > > > >> issues I brought up fairly > > > > >> quickly... > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > >> > > > > > > > >> >>>> > >> > > > > > >> -Jay > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > >> > > > > > > > >> >>>> > >> > > > > > >> On Wed, Feb 18, 2015 at > > 7:27 AM, > > > > >> Andrii > > > > >> > > > > > > > Biletskyi < > > > > >> > > > > > > > >> >>>> > >> > > > > > >> andrii.bilets...@stealth.ly > > > > > > > >> wrote: > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > Hi all, > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > I'm trying to address > > some of > > > > >> the > > > > >> > > > issues > > > > >> > > > > > > which > > > > >> > > > > > > > >> were > > > > >> > > > > > > > >> >>>> > >> mentioned > > > > >> > > > > > > > >> >>>> > >> > > > > earlier > > > > >> > > > > > > > >> >>>> > >> > > > > > >> about > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > Admin RQ/RP format. One of > > > > >> those was > > > > >> > > > > about > > > > >> > > > > > > > >> batching > > > > >> > > > > > > > >> >>>> > >> > operations. > > > > >> > > > > > > > >> >>>> > >> > > > What > > > > >> > > > > > > > >> >>>> > >> > > > > > if > > > > >> > > > > > > > >> >>>> > >> > > > > > >> we > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > follow TopicCommand > > approach > > > > >> and let > > > > >> > > > > people > > > > >> > > > > > > > >> specify > > > > >> > > > > > > > >> >>>> > >> topic-name > > > > >> > > > > > > > >> >>>> > >> > > by > > > > >> > > > > > > > >> >>>> > >> > > > > > >> regexp - > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > would that cover most of > > the > > > > use > > > > >> > > cases? > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > Secondly, is what > > information > > > > >> should > > > > >> > > we > > > > >> > > > > > > > generally > > > > >> > > > > > > > >> >>>> provide in > > > > >> > > > > > > > >> >>>> > >> > > Admin > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > responses. > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > I realize that Admin > > commands > > > > >> don't > > > > >> > > > imply > > > > >> > > > > > > they > > > > >> > > > > > > > >> will > > > > >> > > > > > > > >> >>>> be used > > > > >> > > > > > > > >> >>>> > >> > only > > > > >> > > > > > > > >> >>>> > >> > > > in > > > > >> > > > > > > > >> >>>> > >> > > > > > CLI > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > but, > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > it seems to me, CLI is a > > very > > > > >> > > important > > > > >> > > > > > > client > > > > >> > > > > > > > >> of this > > > > >> > > > > > > > >> >>>> > >> > feature. > > > > >> > > > > > > > >> >>>> > >> > > In > > > > >> > > > > > > > >> >>>> > >> > > > > > this > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > case, > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > seems logical, we would > > like > > > > to > > > > >> > > provide > > > > >> > > > > > users > > > > >> > > > > > > > >> with > > > > >> > > > > > > > >> >>>> rich > > > > >> > > > > > > > >> >>>> > >> > > experience > > > > >> > > > > > > > >> >>>> > >> > > > > in > > > > >> > > > > > > > >> >>>> > >> > > > > > >> terms > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > of > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > getting results / errors > > of > > > > the > > > > >> > > > executed > > > > >> > > > > > > > >> commands. > > > > >> > > > > > > > >> >>>> Usually > > > > >> > > > > > > > >> >>>> > >> we > > > > >> > > > > > > > >> >>>> > >> > > > supply > > > > >> > > > > > > > >> >>>> > >> > > > > > >> with > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > responses only errorCode, > > > > which > > > > >> looks > > > > >> > > > > very > > > > >> > > > > > > > >> limiting, > > > > >> > > > > > > > >> >>>> in case > > > > >> > > > > > > > >> >>>> > >> > of > > > > >> > > > > > > > >> >>>> > >> > > > CLI > > > > >> > > > > > > > >> >>>> > >> > > > > we > > > > >> > > > > > > > >> >>>> > >> > > > > > >> may > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > want to print human > > readable > > > > >> error > > > > >> > > > > > > description. > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > So, taking into account > > > > >> previous item > > > > >> > > > > about > > > > >> > > > > > > > >> batching, > > > > >> > > > > > > > >> >>>> what > > > > >> > > > > > > > >> >>>> > >> do > > > > >> > > > > > > > >> >>>> > >> > > you > > > > >> > > > > > > > >> >>>> > >> > > > > > think > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > about > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > having smth like: > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > ('create' doesn't support > > > > >> regexp) > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > CreateTopicRequest => > > > > TopicName > > > > >> > > > > Partitions > > > > >> > > > > > > > >> Replicas > > > > >> > > > > > > > >> >>>> > >> > > > > ReplicaAssignment > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > [Config] > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > CreateTopicResponse => > > > > ErrorCode > > > > >> > > > > > > > ErrorDescription > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > ErrorCode => int16 > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > ErrorDescription => > > string > > > > >> (empty > > > > >> > > if > > > > >> > > > > > > > >> successful) > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > AlterTopicRequest -> > > > > >> TopicNameRegexp > > > > >> > > > > > > > > > > > > > ... > > > > > > > > > > [Message clipped] > > > > > > > >
