I think while loop is fine for supporting blocking, just that we need to add back off to avoid bombarding brokers with DescribeTopic requests.
Also I have linked KAFKA-1125 <https://issues.apache.org/jira/browse/KAFKA-1125> to your proposal, and when KAFKA-1694 is done this ticket can also be closed. Guozhang On Fri, Mar 20, 2015 at 9:41 AM, Andrii Biletskyi < andrii.bilets...@stealth.ly> wrote: > Great. > I want to elaborate this a bit more, to see we are on the same page > concerning the client code. > > So with all topic commands being async a client (AdminClient in our > case or any other other client people would like to implement) to support > a blocking operation (which seems to be a natural use-case e.g. for topic > creation): would have to do: > 1. issue CreateTopicRequest > 2. if successful, in a "while" loop send DescribeTopicRequest and > break the loop once all topics are returned in response (or upon timeout). > 3. if unsuccessful throw exception > Would it be okay? > > Thanks, > Andrii Biletskyi > > > On Fri, Mar 20, 2015 at 6:21 PM, Jun Rao <j...@confluent.io> wrote: > > > Andrii, > > > > I think you are right. It seems that only ReassignPartitions needs a > > separate verification request. > > > > Thanks, > > > > Jun > > > > On Thu, Mar 19, 2015 at 9:22 AM, Andrii Biletskyi < > > andrii.bilets...@stealth.ly> wrote: > > > > > Guys, > > > I like this idea too. Let's stick with that. I'll update KIP > accordingly. > > > > > > I was also thinking we can avoid adding dedicated status check > > > requests for topic commands. - We have everything in DescribeTopic > > > for that! E.g.: > > > User issued CreateTopic - to check the status client sends > DescribeTopic > > > and checks whether is something returned for that topic. The same for > > > alteration, deletion. > > > Btw, PreferredReplica status can be also checked with > > DescribeTopicRequest > > > (head of assigned replicas list == leader). > > > For ReassignPartitions as discussed we'll need to have a separate > > Verify... > > > request. > > > > > > Thanks, > > > Andrii Biletskyi > > > > > > > > > On Thu, Mar 19, 2015 at 6:03 PM, Guozhang Wang <wangg...@gmail.com> > > wrote: > > > > > > > +1 on broker writing to ZK for async handling. I was thinking that in > > the > > > > end state the admin requests would be eventually sent to controller > > > either > > > > through re-routing or clients discovering them, instead of letting > > > > controller listen on ZK admin path. But thinking about it a second > > time, > > > I > > > > think it is actually simpler to let controller manage > > > > incoming queued-up admin requests through ZK. > > > > > > > > Guozhang > > > > > > > > > > > > On Wed, Mar 18, 2015 at 4:16 PM, Joel Koshy <jjkosh...@gmail.com> > > wrote: > > > > > > > > > +1 as well. I think it helps to keep the rerouting approach > > orthogonal > > > > > to this KIP. > > > > > > > > > > On Wed, Mar 18, 2015 at 03:40:48PM -0700, Jay Kreps wrote: > > > > > > I'm +1 on Jun's suggestion as long as it can work for all the > > > requests. > > > > > > > > > > > > On Wed, Mar 18, 2015 at 3:35 PM, Jun Rao <j...@confluent.io> > wrote: > > > > > > > > > > > > > Andrii, > > > > > > > > > > > > > > I think we agreed on the following. > > > > > > > > > > > > > > (a) Admin requests can be sent to and handled by any broker. > > > > > > > (b) Admin requests are processed asynchronously, at least for > > now. > > > > > That is, > > > > > > > when the client gets a response, it just means that the request > > is > > > > > > > initiated, but not necessarily completed. Then, it's up to the > > > client > > > > > to > > > > > > > issue another request to check the status for completion. > > > > > > > > > > > > > > To support (a), we were thinking of doing request forwarding to > > the > > > > > > > controller (utilizing KAFKA-1912). I am making an alternative > > > > proposal. > > > > > > > Basically, the broker can just write to ZooKeeper to inform the > > > > > controller > > > > > > > about the request. For example, to handle > partitionReassignment, > > > the > > > > > broker > > > > > > > will just write the requested partitions to > > > > /admin/reassign_partitions > > > > > > > (like what AdminUtils currently does) and then send a response > to > > > the > > > > > > > client. This shouldn't take long and the implementation will be > > > > simpler > > > > > > > than forwarding the requests to the controller through RPC. > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 3:03 PM, Andrii Biletskyi < > > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > > > Jun, > > > > > > > > > > > > > > > > I might be wrong but didn't we agree we will let any broker > > from > > > > the > > > > > > > > cluster handle *long-running* admin requests (at this time > > > > > > > preferredReplica > > > > > > > > and > > > > > > > > reassignPartitions), via zk admin path. Thus CreateTopics etc > > > > should > > > > > be > > > > > > > > sent > > > > > > > > only to the controller. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 11:55 PM, Jun Rao <j...@confluent.io> > > > > wrote: > > > > > > > > > > > > > > > > > Joel, Andril, > > > > > > > > > > > > > > > > > > I think we agreed that those admin requests can be issued > to > > > any > > > > > > > broker. > > > > > > > > > Because of that, there doesn't seem to be a strong need to > > know > > > > the > > > > > > > > > controller. So, perhaps we can proceed by not making any > > change > > > > to > > > > > the > > > > > > > > > format of TMR right now. When we start using create topic > > > request > > > > > in > > > > > > > the > > > > > > > > > producer, we will need a new version of TMR that doesn't > > > trigger > > > > > auto > > > > > > > > topic > > > > > > > > > creation. But that can be done later. > > > > > > > > > > > > > > > > > > As a first cut implementation, I think the broker can just > > > write > > > > > to ZK > > > > > > > > > directly for > > > > > > > > > > > > > createToipic/alterTopic/reassignPartitions/preferredLeaderElection > > > > > > > > > requests, instead of forwarding them to the controller. > This > > > will > > > > > > > > simplify > > > > > > > > > the implementation on the broker side. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 11:58 AM, Joel Koshy < > > > > jjkosh...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > (Thanks Andrii for the summary) > > > > > > > > > > > > > > > > > > > > For (1) yes we will circle back on that shortly after > > syncing > > > > up > > > > > in > > > > > > > > > > person. I think it is close to getting committed although > > > > > development > > > > > > > > > > for KAFKA-1927 can probably begin without it. > > > > > > > > > > > > > > > > > > > > There is one more item we covered at the hangout. i.e., > > > whether > > > > > we > > > > > > > > > > want to add the coordinator to the topic metadata > response > > or > > > > > provide > > > > > > > > > > a clearer ClusterMetadataRequest. > > > > > > > > > > > > > > > > > > > > There are two reasons I think we should try and avoid > > adding > > > > the > > > > > > > > > > field: > > > > > > > > > > - It is irrelevant to topic metadata > > > > > > > > > > - If we finally do request rerouting in Kafka then the > > field > > > > > would > > > > > > > add > > > > > > > > > > little to no value. (It still helps to have a separate > > > > > > > > > > ClusterMetadataRequest to query for cluster-wide > > > information > > > > > such > > > > > > > as > > > > > > > > > > 'which broker is the controller?' as Joe mentioned.) > > > > > > > > > > > > > > > > > > > > I think it would be cleaner to have an explicit > > > > > > > ClusterMetadataRequest > > > > > > > > > > that you can send to any broker in order to obtain the > > > > controller > > > > > > > (and > > > > > > > > > > in the future possibly other cluster-wide information). I > > > think > > > > > the > > > > > > > > > > main argument against doing this and instead adding it to > > the > > > > > topic > > > > > > > > > > metadata response was convenience - i.e., you don't have > to > > > > > discover > > > > > > > > > > the controller in advance. However, I don't see much > actual > > > > > > > > > > benefit/convenience in this and in fact think it is a > > > > non-issue. > > > > > Let > > > > > > > > > > me know if I'm overlooking something here. > > > > > > > > > > > > > > > > > > > > As an example, say we need to initiate partition > > reassignment > > > > by > > > > > > > > > > issuing the new ReassignPartitionsRequest to the > controller > > > > > (assume > > > > > > > we > > > > > > > > > > already have the desired manual partition assignment). > If > > we > > > > > are to > > > > > > > > > > augment topic metadata response then the flow be > something > > > like > > > > > this > > > > > > > : > > > > > > > > > > > > > > > > > > > > - Issue topic metadata request to any broker (and > discover > > > the > > > > > > > > > > controller > > > > > > > > > > - Connect to controller if required (i.e., if the broker > > > above > > > > != > > > > > > > > > > controller) > > > > > > > > > > - Issue the partition reassignment request to the > > controller. > > > > > > > > > > > > > > > > > > > > With an explicit cluster metadata request it would be: > > > > > > > > > > - Issue cluster metadata request to any broker > > > > > > > > > > - Connect to controller if required (i.e., if the broker > > > above > > > > != > > > > > > > > > > controller) > > > > > > > > > > - Issue the partition reassignment request > > > > > > > > > > > > > > > > > > > > So it seems to add little practical value and bloats > topic > > > > > metadata > > > > > > > > > > response with an irrelevant detail. > > > > > > > > > > > > > > > > > > > > The other angle to this is the following - is it a matter > > of > > > > > naming? > > > > > > > > > > Should we just rename topic metadata request/response to > > just > > > > > > > > > > MetadataRequest/Response and add cluster metadata to it? > By > > > > that > > > > > same > > > > > > > > > > token should we also allow querying for the consumer > > > > coordinator > > > > > (and > > > > > > > > > > in future transaction coordinator) as well? This leads > to a > > > > > bloated > > > > > > > > > > request which isn't very appealing and altogether > > confusing. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > Joel > > > > > > > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 09:34:12AM -0700, Jun Rao wrote: > > > > > > > > > > > Andri, > > > > > > > > > > > > > > > > > > > > > > Thanks for the summary. > > > > > > > > > > > > > > > > > > > > > > 1. I just realized that in order to start working on > > > > > KAFKA-1927, we > > > > > > > > > will > > > > > > > > > > > need to merge the changes to OffsetCommitRequest (from > > > 0.8.2) > > > > > to > > > > > > > > trunk. > > > > > > > > > > > This is planned to be done as part of KAFKA-1634. So, > we > > > will > > > > > need > > > > > > > > > > Guozhang > > > > > > > > > > > and Joel's help to wrap this up. > > > > > > > > > > > > > > > > > > > > > > 2. Thinking about this a bit more, if the semantic of > > those > > > > > "write" > > > > > > > > > > > requests is async (i.e., after the client gets a > > response, > > > it > > > > > just > > > > > > > > > means > > > > > > > > > > > that the operation is initiated, but not necessarily > > > > > completed), we > > > > > > > > > don't > > > > > > > > > > > really need to forward the requests to the controller. > > > > > Instead, the > > > > > > > > > > > receiving broker can just write the operation to ZK as > > the > > > > > admin > > > > > > > > > command > > > > > > > > > > > line tool previously does. This will simplify the > > > > > implementation. > > > > > > > > > > > > > > > > > > > > > > 8. There is another implementation detail for describe > > > topic. > > > > > > > > Ideally, > > > > > > > > > we > > > > > > > > > > > want to read the topic config from the broker cache, > > > instead > > > > of > > > > > > > > > > ZooKeeper. > > > > > > > > > > > Currently, every broker reads the topic-level config > for > > > all > > > > > > > topics. > > > > > > > > > > > However, it ignores those for topics not hosted on > > itself. > > > > So, > > > > > we > > > > > > > may > > > > > > > > > > need > > > > > > > > > > > to change TopicConfigManager a bit so that it caches > the > > > > > configs > > > > > > > for > > > > > > > > > all > > > > > > > > > > > topics. > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2015 at 1:13 PM, Andrii Biletskyi < > > > > > > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > > > > > > > > > > > Guys, > > > > > > > > > > > > > > > > > > > > > > > > Thanks for a great discussion! > > > > > > > > > > > > Here are the actions points: > > > > > > > > > > > > > > > > > > > > > > > > 1. Q: Get rid of all scala requests objects, use java > > > > > protocol > > > > > > > > > > definitions. > > > > > > > > > > > > A: Gwen kindly took that (KAFKA-1927). It's > > important > > > > to > > > > > > > speed > > > > > > > > up > > > > > > > > > > > > review procedure > > > > > > > > > > > > there since this ticket blocks other > important > > > > > changes. > > > > > > > > > > > > > > > > > > > > > > > > 2. Q: Generic re-reroute facility vs client > maintaining > > > > > cluster > > > > > > > > > state. > > > > > > > > > > > > A: Jay has added pseudo code to KAFKA-1912 - need > > to > > > > > consider > > > > > > > > > > whether > > > > > > > > > > > > this will be > > > > > > > > > > > > easy to implement as a server-side feature > > > > (comments > > > > > are > > > > > > > > > > > > welcomed!). > > > > > > > > > > > > > > > > > > > > > > > > 3. Q: Controller field in wire protocol. > > > > > > > > > > > > A: This might be useful for clients, add this to > > > > > > > > > > TopicMetadataResponse > > > > > > > > > > > > (already in KIP). > > > > > > > > > > > > > > > > > > > > > > > > 4. Q: Decoupling topic creation from TMR. > > > > > > > > > > > > A: I will add proposed by Jun solution (using > > > clientId > > > > > for > > > > > > > > that) > > > > > > > > > > to the > > > > > > > > > > > > KIP. > > > > > > > > > > > > > > > > > > > > > > > > 5. Q: Bumping new versions of TMR vs grabbing all > > > protocol > > > > > > > changes > > > > > > > > in > > > > > > > > > > one > > > > > > > > > > > > version. > > > > > > > > > > > > A: It was decided to try to gather all changes to > > > > > protocol > > > > > > > > > (before > > > > > > > > > > > > release). > > > > > > > > > > > > In case of TMR it worth checking: KAFKA-2020 > > and > > > > > KIP-13 > > > > > > > > > > (quotas) > > > > > > > > > > > > > > > > > > > > > > > > 6. Q: JSON lib is needed to deserialize user's input > in > > > CLI > > > > > tool. > > > > > > > > > > > > A: Use jackson for that, /tools project is a > > separate > > > > > jar so > > > > > > > > > > shouldn't > > > > > > > > > > > > be a big deal. > > > > > > > > > > > > > > > > > > > > > > > > 7. Q: VerifyReassingPartitions vs generic status > check > > > > > command. > > > > > > > > > > > > A: For long-running requests like reassign > > > partitions > > > > > > > > *progress* > > > > > > > > > > check > > > > > > > > > > > > request is useful, > > > > > > > > > > > > it makes sense to introduce it. > > > > > > > > > > > > > > > > > > > > > > > > Please add, correct me if I missed something. > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2015 at 6:20 PM, Andrii Biletskyi < > > > > > > > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Joel, > > > > > > > > > > > > > > > > > > > > > > > > > > You are right, I removed ClusterMetadata because we > > > have > > > > > > > > partially > > > > > > > > > > > > > what we need in TopicMetadata. Also, as Jay pointed > > out > > > > > > > earlier, > > > > > > > > we > > > > > > > > > > > > > would like to have "orthogonal" API, but at the > same > > > time > > > > > we > > > > > > > need > > > > > > > > > > > > > to be backward compatible. > > > > > > > > > > > > > > > > > > > > > > > > > > But I like your idea and even have some other > > arguments > > > > for > > > > > > > this > > > > > > > > > > option: > > > > > > > > > > > > > There is also DescribeTopicRequest which was > proposed > > > in > > > > > this > > > > > > > > KIP, > > > > > > > > > > > > > it returns topic configs, partitions, replication > > > factor > > > > > plus > > > > > > > > > > partition > > > > > > > > > > > > > ISR, ASR, > > > > > > > > > > > > > leader replica. The later part is really already > > there > > > in > > > > > > > > > > > > > TopicMetadataRequest. > > > > > > > > > > > > > So again we'll have to add stuff to TMR, not to > > > duplicate > > > > > some > > > > > > > > info > > > > > > > > > > in > > > > > > > > > > > > > newly added requests. However, this way we'll end > up > > > with > > > > > > > > "monster" > > > > > > > > > > > > > request which returns cluster metadata, topic > > > replication > > > > > and > > > > > > > > > config > > > > > > > > > > info > > > > > > > > > > > > > plus partition replication data. Seems logical to > > split > > > > > TMR to > > > > > > > > > > > > > - ClusterMetadata (brokers + controller, maybe smth > > > else) > > > > > > > > > > > > > - TopicMetadata (topic info + partition details) > > > > > > > > > > > > > But since current TMR is involved in lots of places > > > > > (including > > > > > > > > > > network > > > > > > > > > > > > > client, > > > > > > > > > > > > > as I understand) this might be very serious change > > and > > > it > > > > > > > > probably > > > > > > > > > > makes > > > > > > > > > > > > > sense to stick with current approach. > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2015 at 5:29 PM, Joel Koshy < > > > > > > > jjkosh...@gmail.com > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > >> I may be missing some context but hopefully this > > will > > > > > also be > > > > > > > > > > covered > > > > > > > > > > > > >> today: I thought the earlier proposal where there > > was > > > an > > > > > > > > explicit > > > > > > > > > > > > >> ClusterMetadata request was clearer and explicit. > > > During > > > > > the > > > > > > > > > course > > > > > > > > > > of > > > > > > > > > > > > >> this thread I think the conclusion was that the > main > > > > need > > > > > was > > > > > > > > for > > > > > > > > > > > > >> controller information and that can be rolled into > > the > > > > > topic > > > > > > > > > > metadata > > > > > > > > > > > > >> response but that seems a bit irrelevant to topic > > > > > metadata. > > > > > > > > FWIW I > > > > > > > > > > > > >> think the full broker-list is also irrelevant to > > topic > > > > > > > metadata, > > > > > > > > > but > > > > > > > > > > > > >> it is already there and in use. I think there is > > still > > > > > room > > > > > > > for > > > > > > > > an > > > > > > > > > > > > >> explicit ClusterMetadata request since there may > be > > > > other > > > > > > > > > > > > >> cluster-level information that we may want to add > > over > > > > > time > > > > > > > (and > > > > > > > > > > that > > > > > > > > > > > > >> have nothing to do with topic metadata). > > > > > > > > > > > > >> > > > > > > > > > > > > >> On Tue, Mar 17, 2015 at 02:45:30PM +0200, Andrii > > > > Biletskyi > > > > > > > > wrote: > > > > > > > > > > > > >> > Jun, > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > 101. Okay, if you say that such use case is > > > > important. I > > > > > > > also > > > > > > > > > > think > > > > > > > > > > > > >> > using clientId for these purposes is fine - if > we > > > > > already > > > > > > > have > > > > > > > > > > this > > > > > > > > > > > > >> field > > > > > > > > > > > > >> > as part of all Wire protocol messages, why not > use > > > > that. > > > > > > > > > > > > >> > I will update KIP-4 page if nobody has other > ideas > > > > > (which > > > > > > > may > > > > > > > > > > come up > > > > > > > > > > > > >> > during the call today). > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > 102.1 Agree, I'll update the KIP accordingly. I > > > think > > > > > we can > > > > > > > > add > > > > > > > > > > new, > > > > > > > > > > > > >> > fine-grained error codes if some error code > > received > > > > in > > > > > > > > specific > > > > > > > > > > case > > > > > > > > > > > > >> > won't give enough context to return a > descriptive > > > > error > > > > > > > > message > > > > > > > > > > for > > > > > > > > > > > > >> user. > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > Look forward to discussing all outstanding > issues > > in > > > > > detail > > > > > > > > > today > > > > > > > > > > > > during > > > > > > > > > > > > >> > the call. > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > Thanks, > > > > > > > > > > > > >> > Andrii Biletskyi > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > On Mon, Mar 16, 2015 at 10:59 PM, Jun Rao < > > > > > j...@confluent.io > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > 101. There may be a use case where you only > want > > > the > > > > > > > topics > > > > > > > > to > > > > > > > > > > be > > > > > > > > > > > > >> created > > > > > > > > > > > > >> > > manually by admins. Currently, you can do that > > by > > > > > > > disabling > > > > > > > > > auto > > > > > > > > > > > > topic > > > > > > > > > > > > >> > > creation and issue topic creation from the > > > > > TopicCommand. > > > > > > > If > > > > > > > > we > > > > > > > > > > > > >> disable auto > > > > > > > > > > > > >> > > topic creation completely on the broker and > > don't > > > > > have a > > > > > > > way > > > > > > > > > to > > > > > > > > > > > > >> distinguish > > > > > > > > > > > > >> > > between topic creation requests from the > regular > > > > > clients > > > > > > > and > > > > > > > > > the > > > > > > > > > > > > >> admin, we > > > > > > > > > > > > >> > > can't support manual topic creation any more. > I > > > was > > > > > > > thinking > > > > > > > > > > that > > > > > > > > > > > > >> another > > > > > > > > > > > > >> > > way of distinguishing the clients making the > > topic > > > > > > > creation > > > > > > > > > > requests > > > > > > > > > > > > >> is > > > > > > > > > > > > >> > > using clientId. For example, the admin tool > can > > > set > > > > > it to > > > > > > > > > > something > > > > > > > > > > > > >> like > > > > > > > > > > > > >> > > admin and the broker can treat that clientId > > > > > specially. > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > Also, there is a related discussion in > > KAFKA-2020. > > > > > > > > Currently, > > > > > > > > > > we do > > > > > > > > > > > > >> the > > > > > > > > > > > > >> > > following in TopicMetadataResponse: > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > 1. If leader is not available, we set the > > > partition > > > > > level > > > > > > > > > error > > > > > > > > > > code > > > > > > > > > > > > >> to > > > > > > > > > > > > >> > > LeaderNotAvailable. > > > > > > > > > > > > >> > > 2. If a non-leader replica is not available, > we > > > take > > > > > that > > > > > > > > > > replica > > > > > > > > > > > > out > > > > > > > > > > > > >> of > > > > > > > > > > > > >> > > the assigned replica list and isr in the > > response. > > > > As > > > > > an > > > > > > > > > > indication > > > > > > > > > > > > >> for > > > > > > > > > > > > >> > > doing that, we set the partition level error > > code > > > to > > > > > > > > > > > > >> ReplicaNotAvailable. > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > This has a few problems. First, > > > ReplicaNotAvailable > > > > > > > probably > > > > > > > > > > > > >> shouldn't be > > > > > > > > > > > > >> > > an error, at least for the normal > > > producer/consumer > > > > > > > clients > > > > > > > > > that > > > > > > > > > > > > just > > > > > > > > > > > > >> want > > > > > > > > > > > > >> > > to find out the leader. Second, it can happen > > that > > > > > both > > > > > > > the > > > > > > > > > > leader > > > > > > > > > > > > and > > > > > > > > > > > > >> > > another replica are not available at the same > > > time. > > > > > There > > > > > > > is > > > > > > > > > no > > > > > > > > > > > > error > > > > > > > > > > > > >> code > > > > > > > > > > > > >> > > to indicate both. Third, even if a replica is > > not > > > > > > > available, > > > > > > > > > > it's > > > > > > > > > > > > >> still > > > > > > > > > > > > >> > > useful to return its replica id since some > > clients > > > > > (e.g. > > > > > > > > admin > > > > > > > > > > tool) > > > > > > > > > > > > >> may > > > > > > > > > > > > >> > > still make use of it. > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > One way to address this issue is to always > > return > > > > the > > > > > > > > replica > > > > > > > > > > id for > > > > > > > > > > > > >> > > leader, assigned replicas, and isr regardless > of > > > > > whether > > > > > > > the > > > > > > > > > > > > >> corresponding > > > > > > > > > > > > >> > > broker is live or not. Since we also return > the > > > list > > > > > of > > > > > > > live > > > > > > > > > > > > brokers, > > > > > > > > > > > > >> the > > > > > > > > > > > > >> > > client can figure out whether a leader or a > > > replica > > > > is > > > > > > > live > > > > > > > > or > > > > > > > > > > not > > > > > > > > > > > > >> and act > > > > > > > > > > > > >> > > accordingly. This way, we don't need to set > the > > > > > partition > > > > > > > > > level > > > > > > > > > > > > error > > > > > > > > > > > > >> code > > > > > > > > > > > > >> > > when the leader or a replica is not available. > > > This > > > > > > > doesn't > > > > > > > > > > change > > > > > > > > > > > > >> the wire > > > > > > > > > > > > >> > > protocol, but does change the semantics. Since > > we > > > > are > > > > > > > > evolving > > > > > > > > > > the > > > > > > > > > > > > >> protocol > > > > > > > > > > > > >> > > of TopicMetadataRequest here, we can > potentially > > > > > piggyback > > > > > > > > the > > > > > > > > > > > > change. > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > 102.1 For those types of errors due to invalid > > > > input, > > > > > > > > > shouldn't > > > > > > > > > > we > > > > > > > > > > > > >> just > > > > > > > > > > > > >> > > guard it at parameter validation time and > throw > > > > > > > > > > > > >> InvalidArgumentException > > > > > > > > > > > > >> > > without even sending the request to the > broker? > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > Thanks, > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > Jun > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > On Mon, Mar 16, 2015 at 10:37 AM, Andrii > > > Biletskyi < > > > > > > > > > > > > >> > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > Jun, > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > Answering your questions: > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > 101. If I understand you correctly, you are > > > saying > > > > > > > future > > > > > > > > > > producer > > > > > > > > > > > > >> > > versions > > > > > > > > > > > > >> > > > (which > > > > > > > > > > > > >> > > > will be ported to TMR_V1) won't be able to > > > > > automatically > > > > > > > > > > create > > > > > > > > > > > > >> topic (if > > > > > > > > > > > > >> > > > we > > > > > > > > > > > > >> > > > unconditionally remove topic creation from > > > there). > > > > > But > > > > > > > we > > > > > > > > > > need to > > > > > > > > > > > > >> this > > > > > > > > > > > > >> > > > preserve logic. > > > > > > > > > > > > >> > > > Ok, about your proposal: I'm not a big fan > > too, > > > > > when it > > > > > > > > > comes > > > > > > > > > > to > > > > > > > > > > > > >> > > > differentiating > > > > > > > > > > > > >> > > > clients directly in protocol schema. And > also > > > I'm > > > > > not > > > > > > > > sure I > > > > > > > > > > > > >> understand > > > > > > > > > > > > >> > > at > > > > > > > > > > > > >> > > > all why > > > > > > > > > > > > >> > > > auto.create.topics.enable is a server side > > > > > > > configuration. > > > > > > > > > Can > > > > > > > > > > we > > > > > > > > > > > > >> > > deprecate > > > > > > > > > > > > >> > > > this setting > > > > > > > > > > > > >> > > > in future versions, add this setting to > > producer > > > > and > > > > > > > based > > > > > > > > > on > > > > > > > > > > that > > > > > > > > > > > > >> upon > > > > > > > > > > > > >> > > > receiving > > > > > > > > > > > > >> > > > UnknownTopic create topic explicitly by a > > > separate > > > > > > > > producer > > > > > > > > > > call > > > > > > > > > > > > via > > > > > > > > > > > > >> > > > adminClient? > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > 102.1. Hm, yes. It's because we want to > > support > > > > > batching > > > > > > > > and > > > > > > > > > > at > > > > > > > > > > > > the > > > > > > > > > > > > >> same > > > > > > > > > > > > >> > > > time we > > > > > > > > > > > > >> > > > want to give descriptive error messages for > > > > clients. > > > > > > > Since > > > > > > > > > > > > >> AdminClient > > > > > > > > > > > > >> > > > holds the context > > > > > > > > > > > > >> > > > to construct such messages (e.g. AdminClient > > > layer > > > > > can > > > > > > > > know > > > > > > > > > > that > > > > > > > > > > > > >> > > > InvalidArgumentsCode > > > > > > > > > > > > >> > > > means two cases: either invalid number - > e.g. > > > -1; > > > > or > > > > > > > > > > > > >> replication-factor > > > > > > > > > > > > >> > > was > > > > > > > > > > > > >> > > > provided while > > > > > > > > > > > > >> > > > partitions argument wasn't) - I wrapped > > > responses > > > > in > > > > > > > > > > Exceptions. > > > > > > > > > > > > >> But I'm > > > > > > > > > > > > >> > > > open to any > > > > > > > > > > > > >> > > > other ideas, this was just initial version. > > > > > > > > > > > > >> > > > 102.2. Yes, I agree. I'll change that to > > > probably > > > > > some > > > > > > > > other > > > > > > > > > > dto. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > Thanks, > > > > > > > > > > > > >> > > > Andrii Biletskyi > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > On Fri, Mar 13, 2015 at 7:16 PM, Jun Rao < > > > > > > > > j...@confluent.io> > > > > > > > > > > > > wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > Andrii, > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > 101. That's what I was thinking too, but > it > > > may > > > > > not be > > > > > > > > > that > > > > > > > > > > > > >> simple. In > > > > > > > > > > > > >> > > > > TopicMetadataRequest_V1, > > > > > > > > > > > > >> > > > > we can let it not trigger auto topic > > creation. > > > > > Then, > > > > > > > in > > > > > > > > > the > > > > > > > > > > > > >> producer > > > > > > > > > > > > >> > > > side, > > > > > > > > > > > > >> > > > > if it gets an UnknownTopicException, it > can > > > > > explicitly > > > > > > > > > > issue a > > > > > > > > > > > > >> > > > > createTopicRequest for auto topic > creation. > > On > > > > the > > > > > > > > > consumer > > > > > > > > > > > > side, > > > > > > > > > > > > >> it > > > > > > > > > > > > >> > > will > > > > > > > > > > > > >> > > > > never issue createTopicRequest. This works > > > when > > > > > auto > > > > > > > > topic > > > > > > > > > > > > >> creation is > > > > > > > > > > > > >> > > > > enabled on the broker side. However, I am > > not > > > > > sure how > > > > > > > > > > things > > > > > > > > > > > > >> will work > > > > > > > > > > > > >> > > > > when auto topic creation is disabled on > the > > > > broker > > > > > > > side. > > > > > > > > > In > > > > > > > > > > this > > > > > > > > > > > > >> case, > > > > > > > > > > > > >> > > we > > > > > > > > > > > > >> > > > > want to have a way to manually create a > > topic, > > > > > > > > potentially > > > > > > > > > > > > through > > > > > > > > > > > > >> > > admin > > > > > > > > > > > > >> > > > > commands. However, then we need a way to > > > > > distinguish > > > > > > > > > > > > >> createTopicRequest > > > > > > > > > > > > >> > > > > issued from the producer clients and the > > admin > > > > > tools. > > > > > > > > May > > > > > > > > > > be we > > > > > > > > > > > > >> can > > > > > > > > > > > > >> > > add a > > > > > > > > > > > > >> > > > > new field in createTopicRequest and set it > > > > > differently > > > > > > > > in > > > > > > > > > > the > > > > > > > > > > > > >> producer > > > > > > > > > > > > >> > > > > client and the admin client. However, I am > > not > > > > > sure if > > > > > > > > > > that's > > > > > > > > > > > > the > > > > > > > > > > > > >> best > > > > > > > > > > > > >> > > > > approach. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > 2. Yes, refactoring existing requests is a > > > > > non-trivial > > > > > > > > > > amount of > > > > > > > > > > > > >> work. > > > > > > > > > > > > >> > > I > > > > > > > > > > > > >> > > > > posted some comments in KAFKA-1927. We > will > > > > > probably > > > > > > > > have > > > > > > > > > > to fix > > > > > > > > > > > > >> > > > KAFKA-1927 > > > > > > > > > > > > >> > > > > first, before adding the new logic in > > > > KAFKA-1694. > > > > > > > > > > Otherwise, the > > > > > > > > > > > > >> > > changes > > > > > > > > > > > > >> > > > > will be too big. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > 102. About the AdminClient: > > > > > > > > > > > > >> > > > > 102.1. It's a bit weird that we return > > > exception > > > > > in > > > > > > > the > > > > > > > > > > api. It > > > > > > > > > > > > >> seems > > > > > > > > > > > > >> > > > that > > > > > > > > > > > > >> > > > > we should either return error code or > throw > > an > > > > > > > exception > > > > > > > > > > when > > > > > > > > > > > > >> getting > > > > > > > > > > > > >> > > the > > > > > > > > > > > > >> > > > > response state. > > > > > > > > > > > > >> > > > > 102.2. We probably shouldn't explicitly > use > > > the > > > > > > > request > > > > > > > > > > object > > > > > > > > > > > > in > > > > > > > > > > > > >> the > > > > > > > > > > > > >> > > > api. > > > > > > > > > > > > >> > > > > Not every request evolution requires an > api > > > > > change. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > Thanks, > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > Jun > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > On Fri, Mar 13, 2015 at 4:08 AM, Andrii > > > > Biletskyi > > > > > < > > > > > > > > > > > > >> > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > Jun, > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > Thanks for you comments. Answers inline: > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > 100. There are a few fields such as > > > > > > > ReplicaAssignment, > > > > > > > > > > > > >> > > > > > > ReassignPartitionRequest, > > > > > > > > > > > > >> > > > > > > and PartitionsSerialized that are > > > > represented > > > > > as a > > > > > > > > > > string, > > > > > > > > > > > > but > > > > > > > > > > > > >> > > > contain > > > > > > > > > > > > >> > > > > > > composite structures in json. Could we > > > > flatten > > > > > > > them > > > > > > > > > out > > > > > > > > > > > > >> directly in > > > > > > > > > > > > >> > > > the > > > > > > > > > > > > >> > > > > > > protocol definition as arrays/records? > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > Yes, now with Admin Client this looks a > > bit > > > > > weird. > > > > > > > My > > > > > > > > > > initial > > > > > > > > > > > > >> > > > motivation > > > > > > > > > > > > >> > > > > > was: > > > > > > > > > > > > >> > > > > > ReassignPartitionCommand accepts input > in > > > > json, > > > > > we > > > > > > > > want > > > > > > > > > to > > > > > > > > > > > > >> remain > > > > > > > > > > > > >> > > > tools' > > > > > > > > > > > > >> > > > > > interfaces unchanged, where possible. > > > > > > > > > > > > >> > > > > > If we port it to deserialized format, in > > CLI > > > > > (/tools > > > > > > > > > > project) > > > > > > > > > > > > >> we will > > > > > > > > > > > > >> > > > > have > > > > > > > > > > > > >> > > > > > to add some > > > > > > > > > > > > >> > > > > > json library since /tools is written in > > java > > > > and > > > > > > > we'll > > > > > > > > > > need to > > > > > > > > > > > > >> > > > > deserialize > > > > > > > > > > > > >> > > > > > json file > > > > > > > > > > > > >> > > > > > provided by a user. Can we quickly agree > > on > > > > what > > > > > > > this > > > > > > > > > > library > > > > > > > > > > > > >> should > > > > > > > > > > > > >> > > be > > > > > > > > > > > > >> > > > > > (Jackson, GSON, whatever)? > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > 101. Does TopicMetadataRequest v1 still > > > > trigger > > > > > auto > > > > > > > > > topic > > > > > > > > > > > > >> creation? > > > > > > > > > > > > >> > > > This > > > > > > > > > > > > >> > > > > > > will be a bit weird now that we have a > > > > > separate > > > > > > > > topic > > > > > > > > > > > > >> creation api. > > > > > > > > > > > > >> > > > > Have > > > > > > > > > > > > >> > > > > > > you thought about how the new > > > > > createTopicRequest > > > > > > > and > > > > > > > > > > > > >> > > > > TopicMetadataRequest > > > > > > > > > > > > >> > > > > > > v1 will be used in the > producer/consumer > > > > > client, > > > > > > > in > > > > > > > > > > addition > > > > > > > > > > > > >> to > > > > > > > > > > > > >> > > admin > > > > > > > > > > > > >> > > > > > > tools? For example, ideally, we don't > > want > > > > > > > > > > > > >> TopicMetadataRequest > > > > > > > > > > > > >> > > from > > > > > > > > > > > > >> > > > > the > > > > > > > > > > > > >> > > > > > > consumer to trigger auto topic > creation. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > I agree, this strange logic should be > > fixed. > > > > > I'm not > > > > > > > > > > confident > > > > > > > > > > > > >> in > > > > > > > > > > > > >> > > this > > > > > > > > > > > > >> > > > > > Kafka part so > > > > > > > > > > > > >> > > > > > correct me if I'm wrong, but it doesn't > > look > > > > > like a > > > > > > > > hard > > > > > > > > > > thing > > > > > > > > > > > > >> to > > > > > > > > > > > > >> > > do, I > > > > > > > > > > > > >> > > > > > think we can > > > > > > > > > > > > >> > > > > > leverage AdminClient for that in > Producer > > > and > > > > > > > > > > unconditionally > > > > > > > > > > > > >> remove > > > > > > > > > > > > >> > > > > topic > > > > > > > > > > > > >> > > > > > creation from the > TopicMetadataRequest_V1. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > 2. I think Jay meant getting rid of > scala > > > > > classes > > > > > > > > > > > > >> > > > > > > like HeartbeatRequestAndHeader and > > > > > > > > > > > > >> HeartbeatResponseAndHeader. We > > > > > > > > > > > > >> > > did > > > > > > > > > > > > >> > > > > > that > > > > > > > > > > > > >> > > > > > > as a stop-gap thing when adding the > new > > > > > requests > > > > > > > for > > > > > > > > > the > > > > > > > > > > > > >> consumers. > > > > > > > > > > > > >> > > > > > > However, the long term plan is to get > > rid > > > of > > > > > all > > > > > > > > those > > > > > > > > > > and > > > > > > > > > > > > >> just > > > > > > > > > > > > >> > > reuse > > > > > > > > > > > > >> > > > > the > > > > > > > > > > > > >> > > > > > > java request/response in the client. > > Since > > > > > this > > > > > > > KIP > > > > > > > > > > proposes > > > > > > > > > > > > >> to > > > > > > > > > > > > >> > > add a > > > > > > > > > > > > >> > > > > > > significant number of new requests, > > > perhaps > > > > we > > > > > > > > should > > > > > > > > > > bite > > > > > > > > > > > > the > > > > > > > > > > > > >> > > bullet > > > > > > > > > > > > >> > > > > to > > > > > > > > > > > > >> > > > > > > clean up the existing scala requests > > first > > > > > before > > > > > > > > > > adding new > > > > > > > > > > > > >> ones? > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > Yes, looks like I misunderstood the > point > > of > > > > > > > > > > > > >> ...RequestAndHeader. > > > > > > > > > > > > >> > > > Okay, I > > > > > > > > > > > > >> > > > > > will > > > > > > > > > > > > >> > > > > > rework that. The only thing is that I > > don't > > > > see > > > > > any > > > > > > > > > > example > > > > > > > > > > > > how > > > > > > > > > > > > >> it > > > > > > > > > > > > >> > > was > > > > > > > > > > > > >> > > > > done > > > > > > > > > > > > >> > > > > > for at > > > > > > > > > > > > >> > > > > > least one existing protocol message. > Thus, > > > as > > > > I > > > > > > > > > > understand, I > > > > > > > > > > > > >> have to > > > > > > > > > > > > >> > > > > think > > > > > > > > > > > > >> > > > > > how we > > > > > > > > > > > > >> > > > > > are going to do it. > > > > > > > > > > > > >> > > > > > Re porting all existing RQ/RP in this > > patch. > > > > > Sounds > > > > > > > > > > > > reasonable, > > > > > > > > > > > > >> but > > > > > > > > > > > > >> > > if > > > > > > > > > > > > >> > > > > it's > > > > > > > > > > > > >> > > > > > an *obligatory* > > > > > > > > > > > > >> > > > > > requirement to have Admin KIP done, I'm > > > afraid > > > > > this > > > > > > > > can > > > > > > > > > > be a > > > > > > > > > > > > >> serious > > > > > > > > > > > > >> > > > > > blocker for us. > > > > > > > > > > > > >> > > > > > There are 13 protocol messages and all > > that > > > > > would > > > > > > > > > require > > > > > > > > > > not > > > > > > > > > > > > >> only > > > > > > > > > > > > >> > > unit > > > > > > > > > > > > >> > > > > > tests but quite > > > > > > > > > > > > >> > > > > > intensive manual testing, no? I'm afraid > > I'm > > > > > not the > > > > > > > > > > right guy > > > > > > > > > > > > >> to > > > > > > > > > > > > >> > > cover > > > > > > > > > > > > >> > > > > > pretty much all > > > > > > > > > > > > >> > > > > > Kafka core internals :). Let me know > your > > > > > thoughts > > > > > > > on > > > > > > > > > this > > > > > > > > > > > > >> item. Btw > > > > > > > > > > > > >> > > > > there > > > > > > > > > > > > >> > > > > > is a ticket to > > > > > > > > > > > > >> > > > > > follow-up this issue ( > > > > > > > > > > > > >> > > > > https://issues.apache.org/jira/browse/KAFKA-2006 > > > > > > > > > > > > >> > > > ). > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > Thanks, > > > > > > > > > > > > >> > > > > > Andrii Biletskyi > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > On Fri, Mar 13, 2015 at 6:40 AM, Jun > Rao < > > > > > > > > > > j...@confluent.io> > > > > > > > > > > > > >> wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > Andrii, > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > A few more comments. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > 100. There are a few fields such as > > > > > > > > ReplicaAssignment, > > > > > > > > > > > > >> > > > > > > ReassignPartitionRequest, > > > > > > > > > > > > >> > > > > > > and PartitionsSerialized that are > > > > represented > > > > > as a > > > > > > > > > > string, > > > > > > > > > > > > but > > > > > > > > > > > > >> > > > contain > > > > > > > > > > > > >> > > > > > > composite structures in json. Could we > > > > flatten > > > > > > > them > > > > > > > > > out > > > > > > > > > > > > >> directly in > > > > > > > > > > > > >> > > > the > > > > > > > > > > > > >> > > > > > > protocol definition as arrays/records? > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > 101. Does TopicMetadataRequest v1 > still > > > > > trigger > > > > > > > auto > > > > > > > > > > topic > > > > > > > > > > > > >> > > creation? > > > > > > > > > > > > >> > > > > This > > > > > > > > > > > > >> > > > > > > will be a bit weird now that we have a > > > > > separate > > > > > > > > topic > > > > > > > > > > > > >> creation api. > > > > > > > > > > > > >> > > > > Have > > > > > > > > > > > > >> > > > > > > you thought about how the new > > > > > createTopicRequest > > > > > > > and > > > > > > > > > > > > >> > > > > TopicMetadataRequest > > > > > > > > > > > > >> > > > > > > v1 will be used in the > producer/consumer > > > > > client, > > > > > > > in > > > > > > > > > > addition > > > > > > > > > > > > >> to > > > > > > > > > > > > >> > > admin > > > > > > > > > > > > >> > > > > > > tools? For example, ideally, we don't > > want > > > > > > > > > > > > >> TopicMetadataRequest > > > > > > > > > > > > >> > > from > > > > > > > > > > > > >> > > > > the > > > > > > > > > > > > >> > > > > > > consumer to trigger auto topic > creation. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > 2. I think Jay meant getting rid of > > scala > > > > > classes > > > > > > > > > > > > >> > > > > > > like HeartbeatRequestAndHeader and > > > > > > > > > > > > >> HeartbeatResponseAndHeader. We > > > > > > > > > > > > >> > > did > > > > > > > > > > > > >> > > > > > that > > > > > > > > > > > > >> > > > > > > as a stop-gap thing when adding the > new > > > > > requests > > > > > > > for > > > > > > > > > the > > > > > > > > > > > > >> consumers. > > > > > > > > > > > > >> > > > > > > However, the long term plan is to get > > rid > > > of > > > > > all > > > > > > > > those > > > > > > > > > > and > > > > > > > > > > > > >> just > > > > > > > > > > > > >> > > reuse > > > > > > > > > > > > >> > > > > the > > > > > > > > > > > > >> > > > > > > java request/response in the client. > > Since > > > > > this > > > > > > > KIP > > > > > > > > > > proposes > > > > > > > > > > > > >> to > > > > > > > > > > > > >> > > add a > > > > > > > > > > > > >> > > > > > > significant number of new requests, > > > perhaps > > > > we > > > > > > > > should > > > > > > > > > > bite > > > > > > > > > > > > the > > > > > > > > > > > > >> > > bullet > > > > > > > > > > > > >> > > > > to > > > > > > > > > > > > >> > > > > > > clean up the existing scala requests > > first > > > > > before > > > > > > > > > > adding new > > > > > > > > > > > > >> ones? > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > Thanks, > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > Jun > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > On Thu, Mar 12, 2015 at 3:37 PM, > Andrii > > > > > Biletskyi > > > > > > > < > > > > > > > > > > > > >> > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > Hi, > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > As said above - I list again all > > > comments > > > > > from > > > > > > > > this > > > > > > > > > > thread > > > > > > > > > > > > >> so we > > > > > > > > > > > > >> > > > > > > > can see what's left and finalize all > > > > pending > > > > > > > > issues. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > Comments from Jay: > > > > > > > > > > > > >> > > > > > > > 1. This is much needed > functionality, > > > but > > > > > there > > > > > > > > are > > > > > > > > > a > > > > > > > > > > lot > > > > > > > > > > > > >> of the > > > > > > > > > > > > >> > > so > > > > > > > > > > > > >> > > > > > let's > > > > > > > > > > > > >> > > > > > > > really think these protocols > through. > > We > > > > > really > > > > > > > > want > > > > > > > > > > to > > > > > > > > > > > > end > > > > > > > > > > > > >> up > > > > > > > > > > > > >> > > > with a > > > > > > > > > > > > >> > > > > > set > > > > > > > > > > > > >> > > > > > > > of well thought-out, orthoganol > apis. > > > For > > > > > this > > > > > > > > > reason > > > > > > > > > > I > > > > > > > > > > > > >> think it > > > > > > > > > > > > >> > > is > > > > > > > > > > > > >> > > > > > > really > > > > > > > > > > > > >> > > > > > > > important to think through the end > > state > > > > > even if > > > > > > > > > that > > > > > > > > > > > > >> includes > > > > > > > > > > > > >> > > APIs > > > > > > > > > > > > >> > > > > we > > > > > > > > > > > > >> > > > > > > > won't implement in the first phase. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Definitely behind this. Would > > > > appreciate > > > > > if > > > > > > > > there > > > > > > > > > > are > > > > > > > > > > > > >> concrete > > > > > > > > > > > > >> > > > > > > comments > > > > > > > > > > > > >> > > > > > > > how this can be improved. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > 2. Let's please please please wait > > until > > > > we > > > > > have > > > > > > > > > > switched > > > > > > > > > > > > >> the > > > > > > > > > > > > >> > > > server > > > > > > > > > > > > >> > > > > > over > > > > > > > > > > > > >> > > > > > > > to the new java protocol > definitions. > > If > > > > we > > > > > add > > > > > > > > > upteen > > > > > > > > > > > > more > > > > > > > > > > > > >> ad > > > > > > > > > > > > >> > > hoc > > > > > > > > > > > > >> > > > > > scala > > > > > > > > > > > > >> > > > > > > > objects that is just generating more > > > work > > > > > for > > > > > > > the > > > > > > > > > > > > >> conversion we > > > > > > > > > > > > >> > > > know > > > > > > > > > > > > >> > > > > we > > > > > > > > > > > > >> > > > > > > > have to do. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - > removed > > > > scala > > > > > > > > > protocol > > > > > > > > > > > > >> classes. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > 3. This proposal introduces a new > type > > > of > > > > > > > optional > > > > > > > > > > > > >> parameter. > > > > > > > > > > > > >> > > This > > > > > > > > > > > > >> > > > is > > > > > > > > > > > > >> > > > > > > > inconsistent with everything else in > > the > > > > > > > protocol > > > > > > > > > > where we > > > > > > > > > > > > >> use -1 > > > > > > > > > > > > >> > > > or > > > > > > > > > > > > >> > > > > > some > > > > > > > > > > > > >> > > > > > > > other marker value. You could argue > > > either > > > > > way > > > > > > > but > > > > > > > > > > let's > > > > > > > > > > > > >> stick > > > > > > > > > > > > >> > > with > > > > > > > > > > > > >> > > > > > that > > > > > > > > > > > > >> > > > > > > > for consistency. For clients that > > > > > implemented > > > > > > > the > > > > > > > > > > protocol > > > > > > > > > > > > >> in a > > > > > > > > > > > > >> > > > > better > > > > > > > > > > > > >> > > > > > > way > > > > > > > > > > > > >> > > > > > > > than our scala code these basic > > > primitives > > > > > are > > > > > > > > hard > > > > > > > > > to > > > > > > > > > > > > >> change. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - > removed > > > > > MaybeOf > > > > > > > > type > > > > > > > > > > and > > > > > > > > > > > > >> changed > > > > > > > > > > > > >> > > > > > protocol > > > > > > > > > > > > >> > > > > > > > accordingly. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > 4. ClusterMetadata: This seems to > > > > duplicate > > > > > > > > > > > > >> TopicMetadataRequest > > > > > > > > > > > > >> > > > > which > > > > > > > > > > > > >> > > > > > > has > > > > > > > > > > > > >> > > > > > > > brokers, topics, and partitions. I > > think > > > > we > > > > > > > should > > > > > > > > > > rename > > > > > > > > > > > > >> that > > > > > > > > > > > > >> > > > > request > > > > > > > > > > > > >> > > > > > > > ClusterMetadataRequest (or just > > > > > MetadataRequest) > > > > > > > > and > > > > > > > > > > > > >> include the > > > > > > > > > > > > >> > > id > > > > > > > > > > > > >> > > > > of > > > > > > > > > > > > >> > > > > > > the > > > > > > > > > > > > >> > > > > > > > controller. Or are there other > things > > we > > > > > could > > > > > > > add > > > > > > > > > > here? > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: I agree. Updated the KIP. Let's > > > extends > > > > > > > > > > TopicMetadata > > > > > > > > > > > > to > > > > > > > > > > > > >> > > > version 2 > > > > > > > > > > > > >> > > > > > and > > > > > > > > > > > > >> > > > > > > > include controller. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > 5. We have a tendency to try to > make a > > > lot > > > > > of > > > > > > > > > requests > > > > > > > > > > > > that > > > > > > > > > > > > >> can > > > > > > > > > > > > >> > > > only > > > > > > > > > > > > >> > > > > go > > > > > > > > > > > > >> > > > > > > to > > > > > > > > > > > > >> > > > > > > > particular nodes. This adds a lot of > > > > burden > > > > > for > > > > > > > > > client > > > > > > > > > > > > >> > > > > implementations > > > > > > > > > > > > >> > > > > > > (it > > > > > > > > > > > > >> > > > > > > > sounds easy but each discovery can > > fail > > > in > > > > > many > > > > > > > > > parts > > > > > > > > > > so > > > > > > > > > > > > it > > > > > > > > > > > > >> ends > > > > > > > > > > > > >> > > up > > > > > > > > > > > > >> > > > > > > being a > > > > > > > > > > > > >> > > > > > > > full state machine to do right). I > > think > > > > we > > > > > > > should > > > > > > > > > > > > consider > > > > > > > > > > > > >> > > making > > > > > > > > > > > > >> > > > > > admin > > > > > > > > > > > > >> > > > > > > > commands and ideally as many of the > > > other > > > > > apis > > > > > > > as > > > > > > > > > > possible > > > > > > > > > > > > >> > > > available > > > > > > > > > > > > >> > > > > on > > > > > > > > > > > > >> > > > > > > all > > > > > > > > > > > > >> > > > > > > > brokers and just redirect to the > > > > controller > > > > > on > > > > > > > the > > > > > > > > > > broker > > > > > > > > > > > > >> side. > > > > > > > > > > > > >> > > > > Perhaps > > > > > > > > > > > > >> > > > > > > > there would be a general way to > > > > encapsulate > > > > > this > > > > > > > > > > > > re-routing > > > > > > > > > > > > >> > > > behavior. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: It's a very interesting idea, but > > > seems > > > > > there > > > > > > > > are > > > > > > > > > > some > > > > > > > > > > > > >> > > concerns > > > > > > > > > > > > >> > > > > > about > > > > > > > > > > > > >> > > > > > > > this > > > > > > > > > > > > >> > > > > > > > feature (like performance > > > considerations, > > > > > how > > > > > > > this > > > > > > > > > > will > > > > > > > > > > > > >> > > complicate > > > > > > > > > > > > >> > > > > > server > > > > > > > > > > > > >> > > > > > > > etc). > > > > > > > > > > > > >> > > > > > > > I believe this shouldn't be a > blocker. > > > If > > > > > this > > > > > > > > > > feature is > > > > > > > > > > > > >> > > > implemented > > > > > > > > > > > > >> > > > > > at > > > > > > > > > > > > >> > > > > > > > some > > > > > > > > > > > > >> > > > > > > > point it won't affect Admin changes > - > > at > > > > > least > > > > > > > no > > > > > > > > > > changes > > > > > > > > > > > > to > > > > > > > > > > > > >> > > public > > > > > > > > > > > > >> > > > > API > > > > > > > > > > > > >> > > > > > > > will be required. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > 6. We should probably normalize the > > key > > > > > value > > > > > > > > pairs > > > > > > > > > > used > > > > > > > > > > > > for > > > > > > > > > > > > >> > > > configs > > > > > > > > > > > > >> > > > > > > rather > > > > > > > > > > > > >> > > > > > > > than embedding a new formatting. So > > two > > > > > strings > > > > > > > > > rather > > > > > > > > > > > > than > > > > > > > > > > > > >> one > > > > > > > > > > > > >> > > > with > > > > > > > > > > > > >> > > > > an > > > > > > > > > > > > >> > > > > > > > internal equals sign. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - > > > normalized > > > > > > > configs > > > > > > > > > and > > > > > > > > > > > > >> changed > > > > > > > > > > > > >> > > > > protocol > > > > > > > > > > > > >> > > > > > > > accordingly. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > 7. Is the postcondition of these > APIs > > > that > > > > > the > > > > > > > > > > command has > > > > > > > > > > > > >> begun > > > > > > > > > > > > >> > > or > > > > > > > > > > > > >> > > > > > that > > > > > > > > > > > > >> > > > > > > > the command has been completed? It > is > > a > > > > lot > > > > > more > > > > > > > > > > usable if > > > > > > > > > > > > >> the > > > > > > > > > > > > >> > > > > command > > > > > > > > > > > > >> > > > > > > has > > > > > > > > > > > > >> > > > > > > > been completed so you know that if > you > > > > > create a > > > > > > > > > topic > > > > > > > > > > and > > > > > > > > > > > > >> then > > > > > > > > > > > > >> > > > > publish > > > > > > > > > > > > >> > > > > > to > > > > > > > > > > > > >> > > > > > > > it you won't get an exception about > > > there > > > > > being > > > > > > > no > > > > > > > > > > such > > > > > > > > > > > > >> topic. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: For long running requests (like > > > > reassign > > > > > > > > > > partitions) - > > > > > > > > > > > > >> the > > > > > > > > > > > > >> > > post > > > > > > > > > > > > >> > > > > > > > condition is > > > > > > > > > > > > >> > > > > > > > command has begun - so we don't > block > > > the > > > > > > > client. > > > > > > > > In > > > > > > > > > > case > > > > > > > > > > > > >> of your > > > > > > > > > > > > >> > > > > > > example - > > > > > > > > > > > > >> > > > > > > > topic commands, this will be > > refactored > > > > and > > > > > > > topic > > > > > > > > > > commands > > > > > > > > > > > > >> will > > > > > > > > > > > > >> > > be > > > > > > > > > > > > >> > > > > > > executed > > > > > > > > > > > > >> > > > > > > > immediately, since the Controller > will > > > > serve > > > > > > > Admin > > > > > > > > > > > > requests > > > > > > > > > > > > >> > > > > > > > (follow-up ticket KAFKA-1777). > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > 8. Describe topic and list topics > > > > duplicate > > > > > a > > > > > > > lot > > > > > > > > of > > > > > > > > > > stuff > > > > > > > > > > > > >> in the > > > > > > > > > > > > >> > > > > > > metadata > > > > > > > > > > > > >> > > > > > > > request. Is there a reason to give > > back > > > > > topics > > > > > > > > > marked > > > > > > > > > > for > > > > > > > > > > > > >> > > > deletion? I > > > > > > > > > > > > >> > > > > > > feel > > > > > > > > > > > > >> > > > > > > > like if we just make the > > post-condition > > > of > > > > > the > > > > > > > > > delete > > > > > > > > > > > > >> command be > > > > > > > > > > > > >> > > > that > > > > > > > > > > > > >> > > > > > the > > > > > > > > > > > > >> > > > > > > > topic is deleted that will get rid > of > > > the > > > > > need > > > > > > > for > > > > > > > > > > this > > > > > > > > > > > > >> right? > > > > > > > > > > > > >> > > And > > > > > > > > > > > > >> > > > it > > > > > > > > > > > > >> > > > > > > will > > > > > > > > > > > > >> > > > > > > > be much more intuitive. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - > removed > > > > > topics > > > > > > > > marked > > > > > > > > > > for > > > > > > > > > > > > >> deletion > > > > > > > > > > > > >> > > > in > > > > > > > > > > > > >> > > > > > > > ListTopicsRequest. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > 9. Should we consider batching these > > > > > requests? > > > > > > > We > > > > > > > > > have > > > > > > > > > > > > >> generally > > > > > > > > > > > > >> > > > > tried > > > > > > > > > > > > >> > > > > > to > > > > > > > > > > > > >> > > > > > > > allow multiple operations to be > > batched. > > > > My > > > > > > > > > suspicion > > > > > > > > > > is > > > > > > > > > > > > >> that > > > > > > > > > > > > >> > > > without > > > > > > > > > > > > >> > > > > > > this > > > > > > > > > > > > >> > > > > > > > we will get a lot of code that does > > > > > something > > > > > > > like > > > > > > > > > > > > >> > > > > > > > for(topic: > > adminClient.listTopics()) > > > > > > > > > > > > >> > > > > > > > > adminClient.describeTopic(topic) > > > > > > > > > > > > >> > > > > > > > this code will work great when you > > test > > > > on 5 > > > > > > > > topics > > > > > > > > > > but > > > > > > > > > > > > not > > > > > > > > > > > > >> do as > > > > > > > > > > > > >> > > > > well > > > > > > > > > > > > >> > > > > > if > > > > > > > > > > > > >> > > > > > > > you have 50k. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Updated the KIP - please check > > "Topic > > > > > Admin > > > > > > > > > Schema" > > > > > > > > > > > > >> section. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > 10. I think we should also discuss > how > > > we > > > > > want > > > > > > > to > > > > > > > > > > expose a > > > > > > > > > > > > >> > > > > programmatic > > > > > > > > > > > > >> > > > > > > JVM > > > > > > > > > > > > >> > > > > > > > client api for these operations. > > > Currently > > > > > > > people > > > > > > > > > > rely on > > > > > > > > > > > > >> > > > AdminUtils > > > > > > > > > > > > >> > > > > > > which > > > > > > > > > > > > >> > > > > > > > is totally sketchy. I think we > > probably > > > > need > > > > > > > > another > > > > > > > > > > > > client > > > > > > > > > > > > >> under > > > > > > > > > > > > >> > > > > > > clients/ > > > > > > > > > > > > >> > > > > > > > that exposes administrative > > > functionality. > > > > > We > > > > > > > will > > > > > > > > > > need > > > > > > > > > > > > >> this just > > > > > > > > > > > > >> > > > to > > > > > > > > > > > > >> > > > > > > > properly test the new apis, I > suspect. > > > We > > > > > should > > > > > > > > > > figure > > > > > > > > > > > > out > > > > > > > > > > > > >> that > > > > > > > > > > > > >> > > > API. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Updated the KIP - please check > > "Admin > > > > > Client" > > > > > > > > > > section > > > > > > > > > > > > >> with an > > > > > > > > > > > > >> > > > > > initial > > > > > > > > > > > > >> > > > > > > > API proposal. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > 11. The other information that would > > be > > > > > really > > > > > > > > > useful > > > > > > > > > > to > > > > > > > > > > > > get > > > > > > > > > > > > >> > > would > > > > > > > > > > > > >> > > > be > > > > > > > > > > > > >> > > > > > > > information about partitions--how > much > > > > data > > > > > is > > > > > > > in > > > > > > > > > the > > > > > > > > > > > > >> partition, > > > > > > > > > > > > >> > > > what > > > > > > > > > > > > >> > > > > > are > > > > > > > > > > > > >> > > > > > > > the segment offsets, what is the > > log-end > > > > > offset > > > > > > > > > (i.e. > > > > > > > > > > last > > > > > > > > > > > > >> > > offset), > > > > > > > > > > > > >> > > > > > what > > > > > > > > > > > > >> > > > > > > is > > > > > > > > > > > > >> > > > > > > > the compaction point, etc. I think > > that > > > > done > > > > > > > right > > > > > > > > > > this > > > > > > > > > > > > >> would be > > > > > > > > > > > > >> > > > the > > > > > > > > > > > > >> > > > > > > > successor to the very awkward > > > > OffsetRequest > > > > > we > > > > > > > > have > > > > > > > > > > today. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: I removed > > ConsumerGroupOffsetsRequest > > > > in > > > > > the > > > > > > > > > latest > > > > > > > > > > > > >> patch. I > > > > > > > > > > > > >> > > > > believe > > > > > > > > > > > > >> > > > > > > > this should > > > > > > > > > > > > >> > > > > > > > be resolved in a separate KIP / jira > > > > ticket. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > 12. Generally we can do good error > > > > handling > > > > > > > > without > > > > > > > > > > > > needing > > > > > > > > > > > > >> > > custom > > > > > > > > > > > > >> > > > > > > > server-side > > > > > > > > > > > > >> > > > > > > > messages. I.e. generally the client > > has > > > > the > > > > > > > > context > > > > > > > > > to > > > > > > > > > > > > know > > > > > > > > > > > > >> that > > > > > > > > > > > > >> > > if > > > > > > > > > > > > >> > > > > it > > > > > > > > > > > > >> > > > > > > got > > > > > > > > > > > > >> > > > > > > > an error that the topic doesn't > exist > > to > > > > say > > > > > > > > "Topic > > > > > > > > > X > > > > > > > > > > > > >> doesn't > > > > > > > > > > > > >> > > > exist" > > > > > > > > > > > > >> > > > > > > rather > > > > > > > > > > > > >> > > > > > > > than "error code 14" (or whatever). > > > Maybe > > > > > there > > > > > > > > are > > > > > > > > > > > > specific > > > > > > > > > > > > >> > > cases > > > > > > > > > > > > >> > > > > > where > > > > > > > > > > > > >> > > > > > > > this is hard? If we want to add > > > > server-side > > > > > > > error > > > > > > > > > > messages > > > > > > > > > > > > >> we > > > > > > > > > > > > >> > > > really > > > > > > > > > > > > >> > > > > do > > > > > > > > > > > > >> > > > > > > > need to do this in a consistent way > > > across > > > > > the > > > > > > > > > > protocol. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Updated the KIP - please check > > > > "Protocol > > > > > > > > Errors" > > > > > > > > > > > > >> section. I > > > > > > > > > > > > >> > > > added > > > > > > > > > > > > >> > > > > > the > > > > > > > > > > > > >> > > > > > > > comprehensive, fine-grained list of > > > error > > > > > codes. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > Comments from Guozhang: > > > > > > > > > > > > >> > > > > > > > 13. Describe topic request: it would > > be > > > > > great to > > > > > > > > go > > > > > > > > > > beyond > > > > > > > > > > > > >> just > > > > > > > > > > > > >> > > > > > batching > > > > > > > > > > > > >> > > > > > > on > > > > > > > > > > > > >> > > > > > > > topic name regex for this request. > For > > > > > example, > > > > > > > a > > > > > > > > > very > > > > > > > > > > > > >> common use > > > > > > > > > > > > >> > > > > case > > > > > > > > > > > > >> > > > > > of > > > > > > > > > > > > >> > > > > > > > the topic command is to list all > > topics > > > > > whose > > > > > > > > config > > > > > > > > > > A's > > > > > > > > > > > > >> value is > > > > > > > > > > > > >> > > > B. > > > > > > > > > > > > >> > > > > > With > > > > > > > > > > > > >> > > > > > > > topic name regex then we have to > first > > > > > retrieve > > > > > > > > > > __all__ > > > > > > > > > > > > >> topics's > > > > > > > > > > > > >> > > > > > > > description info and then filter at > > the > > > > > client > > > > > > > > end, > > > > > > > > > > which > > > > > > > > > > > > >> will > > > > > > > > > > > > >> > > be a > > > > > > > > > > > > >> > > > > > huge > > > > > > > > > > > > >> > > > > > > > burden on ZK. > > > > > > > > > > > > >> > > > > > > > AND > > > > > > > > > > > > >> > > > > > > > 14. Config K-Vs in create topic: > this > > is > > > > > related > > > > > > > > to > > > > > > > > > > the > > > > > > > > > > > > >> previous > > > > > > > > > > > > >> > > > > point; > > > > > > > > > > > > >> > > > > > > > maybe we can add another metadata > K-V > > or > > > > > just a > > > > > > > > > > metadata > > > > > > > > > > > > >> string > > > > > > > > > > > > >> > > > along > > > > > > > > > > > > >> > > > > > > side > > > > > > > > > > > > >> > > > > > > > with config K-V in create topic like > > we > > > > did > > > > > for > > > > > > > > > offset > > > > > > > > > > > > >> commit > > > > > > > > > > > > >> > > > > request. > > > > > > > > > > > > >> > > > > > > This > > > > > > > > > > > > >> > > > > > > > field can be quite useful in storing > > > > > information > > > > > > > > > like > > > > > > > > > > > > >> "owner" of > > > > > > > > > > > > >> > > > the > > > > > > > > > > > > >> > > > > > > topic > > > > > > > > > > > > >> > > > > > > > who issue the create command, etc, > > which > > > > is > > > > > > > quite > > > > > > > > > > > > important > > > > > > > > > > > > >> for a > > > > > > > > > > > > >> > > > > > > > multi-tenant setting. Then in the > > > describe > > > > > topic > > > > > > > > > > request > > > > > > > > > > > > we > > > > > > > > > > > > >> can > > > > > > > > > > > > >> > > > also > > > > > > > > > > > > >> > > > > > > batch > > > > > > > > > > > > >> > > > > > > > on regex of the metadata field. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: As discussed it is very > interesting > > > but > > > > > can > > > > > > > be > > > > > > > > > > > > >> implemented > > > > > > > > > > > > >> > > later > > > > > > > > > > > > >> > > > > > after > > > > > > > > > > > > >> > > > > > > > we have some basic functionality > > there. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > 15. Today all the admin operations > are > > > > > async in > > > > > > > > the > > > > > > > > > > sense > > > > > > > > > > > > >> that > > > > > > > > > > > > >> > > > > command > > > > > > > > > > > > >> > > > > > > will > > > > > > > > > > > > >> > > > > > > > return once it is written in ZK, and > > > that > > > > > is why > > > > > > > > we > > > > > > > > > > need > > > > > > > > > > > > >> extra > > > > > > > > > > > > >> > > > > > > verification > > > > > > > > > > > > >> > > > > > > > like testUtil.waitForTopicCreated() > / > > > > verify > > > > > > > > > partition > > > > > > > > > > > > >> > > reassignment > > > > > > > > > > > > >> > > > > > > > request, etc. With admin requests we > > > could > > > > > add a > > > > > > > > > flag > > > > > > > > > > to > > > > > > > > > > > > >> enable / > > > > > > > > > > > > >> > > > > > disable > > > > > > > > > > > > >> > > > > > > > synchronous requests; when it is > > turned > > > > on, > > > > > the > > > > > > > > > > response > > > > > > > > > > > > >> will not > > > > > > > > > > > > >> > > > > > return > > > > > > > > > > > > >> > > > > > > > until the request has been > completed. > > > And > > > > > for > > > > > > > > async > > > > > > > > > > > > >> requests we > > > > > > > > > > > > >> > > can > > > > > > > > > > > > >> > > > > > add a > > > > > > > > > > > > >> > > > > > > > "token" field in the response, and > > then > > > > only > > > > > > > need > > > > > > > > a > > > > > > > > > > > > general > > > > > > > > > > > > >> > > "admin > > > > > > > > > > > > >> > > > > > > > verification request" with the given > > > token > > > > > to > > > > > > > > check > > > > > > > > > > if the > > > > > > > > > > > > >> async > > > > > > > > > > > > >> > > > > > request > > > > > > > > > > > > >> > > > > > > > has been completed. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: I see your point. My idea was to > > > > provide > > > > > > > > specific > > > > > > > > > > > > >> > > > Verify...Request > > > > > > > > > > > > >> > > > > > per > > > > > > > > > > > > >> > > > > > > > each > > > > > > > > > > > > >> > > > > > > > long running request, where needed. > We > > > can > > > > > do it > > > > > > > > the > > > > > > > > > > way > > > > > > > > > > > > you > > > > > > > > > > > > >> > > > suggest. > > > > > > > > > > > > >> > > > > > The > > > > > > > > > > > > >> > > > > > > > only > > > > > > > > > > > > >> > > > > > > > concern is that introducing a token > we > > > > again > > > > > > > will > > > > > > > > > make > > > > > > > > > > > > >> schema > > > > > > > > > > > > >> > > > > > "dynamic". > > > > > > > > > > > > >> > > > > > > We > > > > > > > > > > > > >> > > > > > > > wanted > > > > > > > > > > > > >> > > > > > > > to do similar thing introducing > single > > > > > > > > AdminRequest > > > > > > > > > > for > > > > > > > > > > > > all > > > > > > > > > > > > >> topic > > > > > > > > > > > > >> > > > > > > commands > > > > > > > > > > > > >> > > > > > > > but rejected > > > > > > > > > > > > >> > > > > > > > this idea because we wanted to have > > > schema > > > > > > > > defined. > > > > > > > > > So > > > > > > > > > > > > this > > > > > > > > > > > > >> is > > > > > > > > > > > > >> > > > more a > > > > > > > > > > > > >> > > > > > > > choice between: > > > > > > > > > > > > >> > > > > > > > a) have fixed schema but introduce > > each > > > > > time new > > > > > > > > > > > > >> Verify...Request > > > > > > > > > > > > >> > > > for > > > > > > > > > > > > >> > > > > > > > long-running requests > > > > > > > > > > > > >> > > > > > > > b) use one request for verification > > but > > > > > > > generalize > > > > > > > > > it > > > > > > > > > > with > > > > > > > > > > > > >> token > > > > > > > > > > > > >> > > > > > > > I'm fine with whatever decision > > > community > > > > > come > > > > > > > to. > > > > > > > > > > Just > > > > > > > > > > > > let > > > > > > > > > > > > >> me > > > > > > > > > > > > >> > > know > > > > > > > > > > > > >> > > > > > your > > > > > > > > > > > > >> > > > > > > > thoughts. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > Comment from Gwen: > > > > > > > > > > > > >> > > > > > > > 16. Specifically for ownership, I > > think > > > > the > > > > > plan > > > > > > > > is > > > > > > > > > > to add > > > > > > > > > > > > >> ACL > > > > > > > > > > > > >> > > (it > > > > > > > > > > > > >> > > > > > sounds > > > > > > > > > > > > >> > > > > > > > like you are describing ACL) via an > > > > external > > > > > > > > system > > > > > > > > > > > > (Argus, > > > > > > > > > > > > >> > > > Sentry). > > > > > > > > > > > > >> > > > > > > > I remember KIP-11 described this, > but > > I > > > > > can't > > > > > > > find > > > > > > > > > > the KIP > > > > > > > > > > > > >> any > > > > > > > > > > > > >> > > > > longer. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Okay, no problem. Not sure though > > how > > > > we > > > > > are > > > > > > > > > going > > > > > > > > > > to > > > > > > > > > > > > >> handle > > > > > > > > > > > > >> > > it. > > > > > > > > > > > > >> > > > > > Wait > > > > > > > > > > > > >> > > > > > > > which KIP > > > > > > > > > > > > >> > > > > > > > will be committed first and include > > > > changes > > > > > to > > > > > > > > > > > > >> TopicMetadata from > > > > > > > > > > > > >> > > > the > > > > > > > > > > > > >> > > > > > > later > > > > > > > > > > > > >> > > > > > > > one? > > > > > > > > > > > > >> > > > > > > > Anyway, I added this note to "Open > > > > > Questions" > > > > > > > > > section > > > > > > > > > > so > > > > > > > > > > > > we > > > > > > > > > > > > >> don't > > > > > > > > > > > > >> > > > > miss > > > > > > > > > > > > >> > > > > > > this > > > > > > > > > > > > >> > > > > > > > piece. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > Thanks, > > > > > > > > > > > > >> > > > > > > > Andrii Biletskyi > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > On Fri, Mar 13, 2015 at 12:34 AM, > > Andrii > > > > > > > > Biletskyi < > > > > > > > > > > > > >> > > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > Hi all, > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > Today I uploaded the patch that > > covers > > > > > some of > > > > > > > > the > > > > > > > > > > > > >> discussed > > > > > > > > > > > > >> > > and > > > > > > > > > > > > >> > > > > > agreed > > > > > > > > > > > > >> > > > > > > > > items: > > > > > > > > > > > > >> > > > > > > > > - removed MaybeOf optional type > > > > > > > > > > > > >> > > > > > > > > - switched to java protocol > > > definitions > > > > > > > > > > > > >> > > > > > > > > - simplified messages (normalized > > > > configs, > > > > > > > > removed > > > > > > > > > > topic > > > > > > > > > > > > >> marked > > > > > > > > > > > > >> > > > for > > > > > > > > > > > > >> > > > > > > > > deletion) > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > I also updated the KIP-4 with > > > respective > > > > > > > changes > > > > > > > > > and > > > > > > > > > > > > >> wrote down > > > > > > > > > > > > >> > > > my > > > > > > > > > > > > >> > > > > > > > > proposal for > > > > > > > > > > > > >> > > > > > > > > pending items: > > > > > > > > > > > > >> > > > > > > > > - Batch Admin Operations -> > updated > > > Wire > > > > > > > > Protocol > > > > > > > > > > schema > > > > > > > > > > > > >> > > proposal > > > > > > > > > > > > >> > > > > > > > > - Remove ClusterMetadata -> > changed > > to > > > > > extend > > > > > > > > > > > > >> > > > TopicMetadataRequest > > > > > > > > > > > > >> > > > > > > > > - Admin Client -> updated my > initial > > > > > proposal > > > > > > > to > > > > > > > > > > reflect > > > > > > > > > > > > >> > > batching > > > > > > > > > > > > >> > > > > > > > > - Error codes -> proposed > > fine-grained > > > > > error > > > > > > > > code > > > > > > > > > > > > instead > > > > > > > > > > > > >> of > > > > > > > > > > > > >> > > > > > > > > AdminRequestFailed > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > I will also send a separate email > to > > > > > cover all > > > > > > > > > > comments > > > > > > > > > > > > >> from > > > > > > > > > > > > >> > > this > > > > > > > > > > > > >> > > > > > > thread. > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > Thanks, > > > > > > > > > > > > >> > > > > > > > > Andrii Biletskyi > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > On Thu, Mar 12, 2015 at 9:26 PM, > > Gwen > > > > > Shapira > > > > > > > < > > > > > > > > > > > > >> > > > > gshap...@cloudera.com > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > >> Found KIP-11 ( > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface > > > > > > > > > > > > >> > > > > > > > >> ) > > > > > > > > > > > > >> > > > > > > > >> It actually specifies changes to > > the > > > > > Metadata > > > > > > > > > > protocol, > > > > > > > > > > > > >> so > > > > > > > > > > > > >> > > > making > > > > > > > > > > > > >> > > > > > sure > > > > > > > > > > > > >> > > > > > > > >> both KIPs are consistent in this > > > regard > > > > > will > > > > > > > be > > > > > > > > > > good. > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > >> On Thu, Mar 12, 2015 at 12:21 PM, > > > Gwen > > > > > > > Shapira > > > > > > > > < > > > > > > > > > > > > >> > > > > > gshap...@cloudera.com > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > > > > > > >> > > > > > > > >> > Specifically for ownership, I > > think > > > > the > > > > > > > plan > > > > > > > > is > > > > > > > > > > to > > > > > > > > > > > > add > > > > > > > > > > > > >> ACL > > > > > > > > > > > > >> > > (it > > > > > > > > > > > > >> > > > > > > sounds > > > > > > > > > > > > >> > > > > > > > >> > like you are describing ACL) > via > > an > > > > > > > external > > > > > > > > > > system > > > > > > > > > > > > >> (Argus, > > > > > > > > > > > > >> > > > > > Sentry). > > > > > > > > > > > > >> > > > > > > > >> > I remember KIP-11 described > this, > > > > but I > > > > > > > can't > > > > > > > > > > find > > > > > > > > > > > > the > > > > > > > > > > > > >> KIP > > > > > > > > > > > > >> > > any > > > > > > > > > > > > >> > > > > > > longer. > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > >> > Regardless, I think KIP-4 > focuses > > > on > > > > > > > getting > > > > > > > > > > > > >> information > > > > > > > > > > > > >> > > that > > > > > > > > > > > > >> > > > > > > already > > > > > > > > > > > > >> > > > > > > > >> > exists from Kafka brokers, not > on > > > > > adding > > > > > > > > > > information > > > > > > > > > > > > >> that > > > > > > > > > > > > >> > > > > perhaps > > > > > > > > > > > > >> > > > > > > > >> > should exist but doesn't yet? > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > >> > Gwen > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > >> > On Thu, Mar 12, 2015 at 6:37 > AM, > > > > > Guozhang > > > > > > > > Wang > > > > > > > > > < > > > > > > > > > > > > >> > > > > > wangg...@gmail.com> > > > > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > > > > > > >> > > > > > > > >> >> Folks, > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > >> > > > > > > > >> >> Just want to elaborate a bit > > more > > > on > > > > > the > > > > > > > > > > > > create-topic > > > > > > > > > > > > >> > > > metadata > > > > > > > > > > > > >> > > > > > and > > > > > > > > > > > > >> > > > > > > > >> batching > > > > > > > > > > > > >> > > > > > > > >> >> describe-topic based on > config / > > > > > metadata > > > > > > > in > > > > > > > > > my > > > > > > > > > > > > >> previous > > > > > > > > > > > > >> > > > email > > > > > > > > > > > > >> > > > > as > > > > > > > > > > > > >> > > > > > > we > > > > > > > > > > > > >> > > > > > > > >> work > > > > > > > > > > > > >> > > > > > > > >> >> on KAFKA-1694. The main > > motivation > > > > is > > > > > to > > > > > > > > have > > > > > > > > > > some > > > > > > > > > > > > >> sort of > > > > > > > > > > > > >> > > > > topic > > > > > > > > > > > > >> > > > > > > > >> management > > > > > > > > > > > > >> > > > > > > > >> >> mechanisms, which I think is > > quite > > > > > > > important > > > > > > > > > in > > > > > > > > > > a > > > > > > > > > > > > >> > > > multi-tenant > > > > > > > > > > > > >> > > > > / > > > > > > > > > > > > >> > > > > > > > cloud > > > > > > > > > > > > >> > > > > > > > >> >> architecture: today anyone can > > > > create > > > > > > > topics > > > > > > > > > in > > > > > > > > > > a > > > > > > > > > > > > >> shared > > > > > > > > > > > > >> > > > Kafka > > > > > > > > > > > > >> > > > > > > > >> cluster, but > > > > > > > > > > > > >> > > > > > > > >> >> there is no concept or > > "ownership" > > > > of > > > > > > > topics > > > > > > > > > > that > > > > > > > > > > > > are > > > > > > > > > > > > >> > > created > > > > > > > > > > > > >> > > > > by > > > > > > > > > > > > >> > > > > > > > >> different > > > > > > > > > > > > >> > > > > > > > >> >> users. For example, at > LinkedIn > > we > > > > > > > basically > > > > > > > > > > > > >> distinguish > > > > > > > > > > > > >> > > > topic > > > > > > > > > > > > >> > > > > > > owners > > > > > > > > > > > > >> > > > > > > > >> via > > > > > > > > > > > > >> > > > > > > > >> >> some casual topic name prefix, > > > which > > > > > is a > > > > > > > > bit > > > > > > > > > > > > awkward > > > > > > > > > > > > >> and > > > > > > > > > > > > >> > > > does > > > > > > > > > > > > >> > > > > > not > > > > > > > > > > > > >> > > > > > > > fly > > > > > > > > > > > > >> > > > > > > > >> as > > > > > > > > > > > > >> > > > > > > > >> >> we scale our customers. It > would > > > be > > > > > great > > > > > > > to > > > > > > > > > use > > > > > > > > > > > > >> > > > > describe-topics > > > > > > > > > > > > >> > > > > > > such > > > > > > > > > > > > >> > > > > > > > >> as: > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > >> > > > > > > > >> >> Describe all topics that is > > > created > > > > > by me. > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > >> > > > > > > > >> >> Describe all topics whose > > > retention > > > > > time > > > > > > > is > > > > > > > > > > > > overriden > > > > > > > > > > > > >> to X. > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > >> > > > > > > > >> >> Describe all topics whose > > writable > > > > > group > > > > > > > > > include > > > > > > > > > > > > user > > > > > > > > > > > > >> Y > > > > > > > > > > > > >> > > (this > > > > > > > > > > > > >> > > > > is > > > > > > > > > > > > >> > > > > > > > >> related to > > > > > > > > > > > > >> > > > > > > > >> >> authorization), etc.. > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > >> > > > > > > > >> >> One possible way to achieve > this > > > is > > > > to > > > > > > > add a > > > > > > > > > > > > metadata > > > > > > > > > > > > >> file > > > > > > > > > > > > >> > > in > > > > > > > > > > > > >> > > > > the > > > > > > > > > > > > >> > > > > > > > >> >> create-topic request, whose > > value > > > > will > > > > > > > also > > > > > > > > be > > > > > > > > > > > > >> written ZK > > > > > > > > > > > > >> > > as > > > > > > > > > > > > >> > > > we > > > > > > > > > > > > >> > > > > > > > create > > > > > > > > > > > > >> > > > > > > > >> the > > > > > > > > > > > > >> > > > > > > > >> >> topic; then describe-topics > can > > > > > choose to > > > > > > > > > batch > > > > > > > > > > > > topics > > > > > > > > > > > > >> > > based > > > > > > > > > > > > >> > > > on > > > > > > > > > > > > >> > > > > > 1) > > > > > > > > > > > > >> > > > > > > > name > > > > > > > > > > > > >> > > > > > > > >> >> regex, 2) config K-V matching, > > 3) > > > > > metadata > > > > > > > > > > regex, > > > > > > > > > > > > etc. > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > >> > > > > > > > >> >> Thoughts? > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > >> > > > > > > > >> >> Guozhang > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > >> > > > > > > > >> >> On Thu, Mar 5, 2015 at 4:37 > PM, > > > > > Guozhang > > > > > > > > Wang > > > > > > > > > < > > > > > > > > > > > > >> > > > > > wangg...@gmail.com> > > > > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > >> > > > > > > > >> >>> Thanks for the updated wiki. > A > > > few > > > > > > > comments > > > > > > > > > > below: > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > >> > > > > > > > >> >>> 1. Error description in > > > response: I > > > > > think > > > > > > > > if > > > > > > > > > > some > > > > > > > > > > > > >> > > errorCode > > > > > > > > > > > > >> > > > > > could > > > > > > > > > > > > >> > > > > > > > >> indicate > > > > > > > > > > > > >> > > > > > > > >> >>> several different error cases > > > then > > > > we > > > > > > > > should > > > > > > > > > > really > > > > > > > > > > > > >> change > > > > > > > > > > > > >> > > > it > > > > > > > > > > > > >> > > > > to > > > > > > > > > > > > >> > > > > > > > >> multiple > > > > > > > > > > > > >> > > > > > > > >> >>> codes. In general the > errorCode > > > > > itself > > > > > > > > would > > > > > > > > > be > > > > > > > > > > > > >> precise > > > > > > > > > > > > >> > > and > > > > > > > > > > > > >> > > > > > > > >> sufficient for > > > > > > > > > > > > >> > > > > > > > >> >>> describing the server side > > > errors. > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > >> > > > > > > > >> >>> 2. Describe topic request: it > > > would > > > > > be > > > > > > > > great > > > > > > > > > > to go > > > > > > > > > > > > >> beyond > > > > > > > > > > > > >> > > > just > > > > > > > > > > > > >> > > > > > > > >> batching on > > > > > > > > > > > > >> > > > > > > > >> >>> topic name regex for this > > > request. > > > > > For > > > > > > > > > > example, a > > > > > > > > > > > > >> very > > > > > > > > > > > > >> > > > common > > > > > > > > > > > > >> > > > > > use > > > > > > > > > > > > >> > > > > > > > >> case of > > > > > > > > > > > > >> > > > > > > > >> >>> the topic command is to list > > all > > > > > topics > > > > > > > > whose > > > > > > > > > > > > config > > > > > > > > > > > > >> A's > > > > > > > > > > > > >> > > > value > > > > > > > > > > > > >> > > > > > is > > > > > > > > > > > > >> > > > > > > B. > > > > > > > > > > > > >> > > > > > > > >> With > > > > > > > > > > > > >> > > > > > > > >> >>> topic name regex then we have > > to > > > > > first > > > > > > > > > retrieve > > > > > > > > > > > > >> __all__ > > > > > > > > > > > > >> > > > > topics's > > > > > > > > > > > > >> > > > > > > > >> >>> description info and then > > filter > > > at > > > > > the > > > > > > > > > client > > > > > > > > > > end, > > > > > > > > > > > > >> which > > > > > > > > > > > > >> > > > will > > > > > > > > > > > > >> > > > > > be > > > > > > > > > > > > >> > > > > > > a > > > > > > > > > > > > >> > > > > > > > >> huge > > > > > > > > > > > > >> > > > > > > > >> >>> burden on ZK. > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > >> > > > > > > > >> >>> 3. Config K-Vs in create > topic: > > > > this > > > > > is > > > > > > > > > > related to > > > > > > > > > > > > >> the > > > > > > > > > > > > >> > > > > previous > > > > > > > > > > > > >> > > > > > > > point; > > > > > > > > > > > > >> > > > > > > > >> >>> maybe we can add another > > metadata > > > > > K-V or > > > > > > > > > just a > > > > > > > > > > > > >> metadata > > > > > > > > > > > > >> > > > > string > > > > > > > > > > > > >> > > > > > > > along > > > > > > > > > > > > >> > > > > > > > >> side > > > > > > > > > > > > >> > > > > > > > >> >>> with config K-V in create > topic > > > > like > > > > > we > > > > > > > did > > > > > > > > > for > > > > > > > > > > > > >> offset > > > > > > > > > > > > >> > > > commit > > > > > > > > > > > > >> > > > > > > > >> request. This > > > > > > > > > > > > >> > > > > > > > >> >>> field can be quite useful in > > > > storing > > > > > > > > > > information > > > > > > > > > > > > like > > > > > > > > > > > > >> > > > "owner" > > > > > > > > > > > > >> > > > > of > > > > > > > > > > > > >> > > > > > > the > > > > > > > > > > > > >> > > > > > > > >> topic > > > > > > > > > > > > >> > > > > > > > >> >>> who issue the create command, > > > etc, > > > > > which > > > > > > > is > > > > > > > > > > quite > > > > > > > > > > > > >> > > important > > > > > > > > > > > > >> > > > > for > > > > > > > > > > > > >> > > > > > a > > > > > > > > > > > > >> > > > > > > > >> >>> multi-tenant setting. Then in > > the > > > > > > > describe > > > > > > > > > > topic > > > > > > > > > > > > >> request > > > > > > > > > > > > >> > > we > > > > > > > > > > > > >> > > > > can > > > > > > > > > > > > >> > > > > > > also > > > > > > > > > > > > >> > > > > > > > >> batch > > > > > > > > > > > > >> > > > > > > > >> >>> on regex of the metadata > field. > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > >> > > > > > > > >> >>> 4. Today all the admin > > operations > > > > are > > > > > > > async > > > > > > > > > in > > > > > > > > > > the > > > > > > > > > > > > >> sense > > > > > > > > > > > > >> > > > that > > > > > > > > > > > > >> > > > > > > > command > > > > > > > > > > > > >> > > > > > > > >> will > > > > > > > > > > > > >> > > > > > > > >> >>> return once it is written in > > ZK, > > > > and > > > > > that > > > > > > > > is > > > > > > > > > > why we > > > > > > > > > > > > >> need > > > > > > > > > > > > >> > > > extra > > > > > > > > > > > > >> > > > > > > > >> verification > > > > > > > > > > > > >> > > > > > > > >> >>> like > > > > testUtil.waitForTopicCreated() / > > > > > > > > verify > > > > > > > > > > > > >> partition > > > > > > > > > > > > >> > > > > > > reassignment > > > > > > > > > > > > >> > > > > > > > >> >>> request, etc. With admin > > requests > > > > we > > > > > > > could > > > > > > > > > add > > > > > > > > > > a > > > > > > > > > > > > >> flag to > > > > > > > > > > > > >> > > > > enable > > > > > > > > > > > > >> > > > > > / > > > > > > > > > > > > >> > > > > > > > >> disable > > > > > > > > > > > > >> > > > > > > > >> >>> synchronous requests; when it > > is > > > > > turned > > > > > > > on, > > > > > > > > > the > > > > > > > > > > > > >> response > > > > > > > > > > > > >> > > > will > > > > > > > > > > > > >> > > > > > not > > > > > > > > > > > > >> > > > > > > > >> return > > > > > > > > > > > > >> > > > > > > > >> >>> until the request has been > > > > > completed. And > > > > > > > > for > > > > > > > > > > async > > > > > > > > > > > > >> > > requests > > > > > > > > > > > > >> > > > > we > > > > > > > > > > > > >> > > > > > > can > > > > > > > > > > > > >> > > > > > > > >> add a > > > > > > > > > > > > >> > > > > > > > >> >>> "token" field in the > response, > > > and > > > > > then > > > > > > > > only > > > > > > > > > > need a > > > > > > > > > > > > >> > > general > > > > > > > > > > > > >> > > > > > "admin > > > > > > > > > > > > >> > > > > > > > >> >>> verification request" with > the > > > > given > > > > > > > token > > > > > > > > to > > > > > > > > > > check > > > > > > > > > > > > >> if the > > > > > > > > > > > > >> > > > > async > > > > > > > > > > > > >> > > > > > > > >> request > > > > > > > > > > > > >> > > > > > > > >> >>> has been completed. > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > >> > > > > > > > >> >>> 5. +1 for extending Metadata > > > > request > > > > > to > > > > > > > > > include > > > > > > > > > > > > >> > > controller / > > > > > > > > > > > > >> > > > > > > > >> coordinator > > > > > > > > > > > > >> > > > > > > > >> >>> information, and then we can > > > remove > > > > > the > > > > > > > > > > > > >> ConsumerMetadata / > > > > > > > > > > > > >> > > > > > > > >> ClusterMetadata > > > > > > > > > > > > >> > > > > > > > >> >>> requests. > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > >> > > > > > > > >> >>> Guozhang > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > >> > > > > > > > >> >>> On Tue, Mar 3, 2015 at 10:23 > > AM, > > > > Joel > > > > > > > > Koshy < > > > > > > > > > > > > >> > > > > > jjkosh...@gmail.com> > > > > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > >> > > > > > > > >> >>>> Thanks for sending that out > > Joe > > > - > > > > I > > > > > > > don't > > > > > > > > > > think I > > > > > > > > > > > > >> will be > > > > > > > > > > > > >> > > > > able > > > > > > > > > > > > >> > > > > > to > > > > > > > > > > > > >> > > > > > > > >> make > > > > > > > > > > > > >> > > > > > > > >> >>>> it today, so if notes can be > > > sent > > > > > out > > > > > > > > > > afterward > > > > > > > > > > > > that > > > > > > > > > > > > >> > > would > > > > > > > > > > > > >> > > > be > > > > > > > > > > > > >> > > > > > > > great. > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > >> > > > > > > > >> >>>> On Mon, Mar 02, 2015 at > > > 09:16:13AM > > > > > > > -0800, > > > > > > > > > Gwen > > > > > > > > > > > > >> Shapira > > > > > > > > > > > > >> > > > wrote: > > > > > > > > > > > > >> > > > > > > > >> >>>> > Thanks for sending this > out > > > Joe. > > > > > > > Looking > > > > > > > > > > forward > > > > > > > > > > > > >> to > > > > > > > > > > > > >> > > > > chatting > > > > > > > > > > > > >> > > > > > > with > > > > > > > > > > > > >> > > > > > > > >> >>>> everyone :) > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > >> > > > > > > > >> >>>> > On Mon, Mar 2, 2015 at > 6:46 > > > AM, > > > > > Joe > > > > > > > > Stein > > > > > > > > > < > > > > > > > > > > > > >> > > > > > > joe.st...@stealth.ly> > > > > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > > > > > > >> > > > > > > > >> >>>> > > Hey, I just sent out a > > > google > > > > > > > hangout > > > > > > > > > > invite > > > > > > > > > > > > to > > > > > > > > > > > > >> all > > > > > > > > > > > > >> > > > pmc, > > > > > > > > > > > > >> > > > > > > > >> committers > > > > > > > > > > > > >> > > > > > > > >> >>>> and > > > > > > > > > > > > >> > > > > > > > >> >>>> > > everyone I found working > > on > > > a > > > > > KIP. > > > > > > > If > > > > > > > > I > > > > > > > > > > missed > > > > > > > > > > > > >> anyone > > > > > > > > > > > > >> > > > in > > > > > > > > > > > > >> > > > > > the > > > > > > > > > > > > >> > > > > > > > >> invite > > > > > > > > > > > > >> > > > > > > > >> >>>> please > > > > > > > > > > > > >> > > > > > > > >> >>>> > > let me know and can > update > > > it, > > > > > np. > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > We should do this every > > > > Tuesday > > > > > @ > > > > > > > 2pm > > > > > > > > > > Eastern > > > > > > > > > > > > >> Time. > > > > > > > > > > > > >> > > > Maybe > > > > > > > > > > > > >> > > > > > we > > > > > > > > > > > > >> > > > > > > > can > > > > > > > > > > > > >> > > > > > > > >> get > > > > > > > > > > > > >> > > > > > > > >> >>>> INFRA > > > > > > > > > > > > >> > > > > > > > >> >>>> > > help to make a google > > > account > > > > > so we > > > > > > > > can > > > > > > > > > > manage > > > > > > > > > > > > >> > > better? > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > To discuss > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > > > > > > > > > > >> > > > > > > > >> >>>> > > in progress and related > > JIRA > > > > > that > > > > > > > are > > > > > > > > > > > > >> interdependent > > > > > > > > > > > > >> > > > and > > > > > > > > > > > > >> > > > > > > common > > > > > > > > > > > > >> > > > > > > > >> work. > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > ~ Joe Stein > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > On Tue, Feb 24, 2015 at > > 2:59 > > > > > PM, Jay > > > > > > > > > > Kreps < > > > > > > > > > > > > >> > > > > > > > jay.kr...@gmail.com> > > > > > > > > > > > > >> > > > > > > > >> >>>> wrote: > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> Let's stay on Google > > > hangouts > > > > > that > > > > > > > > will > > > > > > > > > > also > > > > > > > > > > > > >> record > > > > > > > > > > > > >> > > > and > > > > > > > > > > > > >> > > > > > make > > > > > > > > > > > > >> > > > > > > > the > > > > > > > > > > > > >> > > > > > > > >> >>>> sessions > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> available on youtube. > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> -Jay > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> On Tue, Feb 24, 2015 at > > > 11:49 > > > > > AM, > > > > > > > > Jeff > > > > > > > > > > > > Holoman > > > > > > > > > > > > >> < > > > > > > > > > > > > >> > > > > > > > >> >>>> jholo...@cloudera.com> > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> wrote: > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > Jay / Joe > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > We're happy to send > > out a > > > > > Webex > > > > > > > for > > > > > > > > > > this > > > > > > > > > > > > >> purpose. > > > > > > > > > > > > >> > > We > > > > > > > > > > > > >> > > > > > could > > > > > > > > > > > > >> > > > > > > > >> record > > > > > > > > > > > > >> > > > > > > > >> >>>> the > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > sessions if there is > > > > > interest and > > > > > > > > > > publish > > > > > > > > > > > > >> them > > > > > > > > > > > > >> > > out. > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > Thanks > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > Jeff > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > On Tue, Feb 24, 2015 > at > > > > > 11:28 AM, > > > > > > > > Jay > > > > > > > > > > > > Kreps < > > > > > > > > > > > > >> > > > > > > > >> jay.kr...@gmail.com> > > > > > > > > > > > > >> > > > > > > > >> >>>> wrote: > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > Let's try to get > the > > > > > technical > > > > > > > > > > hang-ups > > > > > > > > > > > > >> sorted > > > > > > > > > > > > >> > > > out, > > > > > > > > > > > > >> > > > > > > > though. > > > > > > > > > > > > >> > > > > > > > >> I > > > > > > > > > > > > >> > > > > > > > >> >>>> really > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > think > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > there is some > benefit > > > to > > > > > live > > > > > > > > > > discussion > > > > > > > > > > > > vs > > > > > > > > > > > > >> > > > > writing. I > > > > > > > > > > > > >> > > > > > > am > > > > > > > > > > > > >> > > > > > > > >> >>>> hopeful that > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> if > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > we post > instructions > > > and > > > > > give > > > > > > > > > > ourselves a > > > > > > > > > > > > >> few > > > > > > > > > > > > >> > > > > attempts > > > > > > > > > > > > >> > > > > > > we > > > > > > > > > > > > >> > > > > > > > >> can > > > > > > > > > > > > >> > > > > > > > >> >>>> get it > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > working. > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > Tuesday at that > time > > > > would > > > > > work > > > > > > > > for > > > > > > > > > > > > >> me...any > > > > > > > > > > > > >> > > > > > objections? > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > -Jay > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > On Tue, Feb 24, > 2015 > > at > > > > > 8:18 > > > > > > > AM, > > > > > > > > > Joe > > > > > > > > > > > > Stein > > > > > > > > > > > > >> < > > > > > > > > > > > > >> > > > > > > > >> joe.st...@stealth.ly > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> wrote: > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > Weekly would be > > great > > > > > maybe > > > > > > > > like > > > > > > > > > > every > > > > > > > > > > > > >> > > Tuesday ~ > > > > > > > > > > > > >> > > > > 1pm > > > > > > > > > > > > >> > > > > > > ET > > > > > > > > > > > > >> > > > > > > > / > > > > > > > > > > > > >> > > > > > > > >> 10am > > > > > > > > > > > > >> > > > > > > > >> >>>> PT > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> ???? > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > I don't mind > google > > > > > hangout > > > > > > > but > > > > > > > > > > there > > > > > > > > > > > > is > > > > > > > > > > > > >> > > always > > > > > > > > > > > > >> > > > > some > > > > > > > > > > > > >> > > > > > > > >> issue or > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> whatever > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > so > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > we know the > apache > > > irc > > > > > > > channel > > > > > > > > > > works. > > > > > > > > > > > > We > > > > > > > > > > > > >> can > > > > > > > > > > > > >> > > > start > > > > > > > > > > > > >> > > > > > > there > > > > > > > > > > > > >> > > > > > > > >> and > > > > > > > > > > > > >> > > > > > > > >> >>>> see how > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> it > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > goes? We can pull > > > > > transcripts > > > > > > > > too > > > > > > > > > > and > > > > > > > > > > > > >> > > associate > > > > > > > > > > > > >> > > > to > > > > > > > > > > > > >> > > > > > > > >> tickets if > > > > > > > > > > > > >> > > > > > > > >> >>>> need be > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > makes > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > it helpful for > > > things. > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > ~ Joestein > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > On Tue, Feb 24, > > 2015 > > > at > > > > > 11:10 > > > > > > > > AM, > > > > > > > > > > Jay > > > > > > > > > > > > >> Kreps < > > > > > > > > > > > > >> > > > > > > > >> >>>> jay.kr...@gmail.com> > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > wrote: > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > We'd talked > about > > > > > doing a > > > > > > > > > Google > > > > > > > > > > > > >> Hangout to > > > > > > > > > > > > >> > > > chat > > > > > > > > > > > > >> > > > > > > about > > > > > > > > > > > > >> > > > > > > > >> this. > > > > > > > > > > > > >> > > > > > > > >> >>>> What > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > about > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > generalizing > > that a > > > > > little > > > > > > > > > > > > further...I > > > > > > > > > > > > >> > > > actually > > > > > > > > > > > > >> > > > > > > think > > > > > > > > > > > > >> > > > > > > > it > > > > > > > > > > > > >> > > > > > > > >> >>>> would be > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > good > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > for > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > everyone > > spending a > > > > > > > > reasonable > > > > > > > > > > chunk > > > > > > > > > > > > of > > > > > > > > > > > > >> > > their > > > > > > > > > > > > >> > > > > week > > > > > > > > > > > > >> > > > > > > on > > > > > > > > > > > > >> > > > > > > > >> Kafka > > > > > > > > > > > > >> > > > > > > > >> >>>> stuff > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> to > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > maybe > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > sync up once a > > > week. > > > > I > > > > > > > think > > > > > > > > we > > > > > > > > > > could > > > > > > > > > > > > >> use > > > > > > > > > > > > >> > > time > > > > > > > > > > > > >> > > > > to > > > > > > > > > > > > >> > > > > > > talk > > > > > > > > > > > > >> > > > > > > > >> >>>> through > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> design > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > stuff, make > sure > > we > > > > > are on > > > > > > > > top > > > > > > > > > of > > > > > > > > > > > > code > > > > > > > > > > > > >> > > > reviews, > > > > > > > > > > > > >> > > > > > talk > > > > > > > > > > > > >> > > > > > > > >> through > > > > > > > > > > > > >> > > > > > > > >> >>>> any > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > tricky > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > issues, etc. > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > We can make it > > > > publicly > > > > > > > > > > available so > > > > > > > > > > > > >> that > > > > > > > > > > > > >> > > any > > > > > > > > > > > > >> > > > > one > > > > > > > > > > > > >> > > > > > > can > > > > > > > > > > > > >> > > > > > > > >> follow > > > > > > > > > > > > >> > > > > > > > >> >>>> along > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > who > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > likes. > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > Any interest in > > > doing > > > > > this? > > > > > > > > If > > > > > > > > > so > > > > > > > > > > > > I'll > > > > > > > > > > > > >> try > > > > > > > > > > > > >> > > to > > > > > > > > > > > > >> > > > > set > > > > > > > > > > > > >> > > > > > it > > > > > > > > > > > > >> > > > > > > > up > > > > > > > > > > > > >> > > > > > > > >> >>>> starting > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> next > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > week. > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > -Jay > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > On Tue, Feb 24, > > > 2015 > > > > at > > > > > > > 3:57 > > > > > > > > > AM, > > > > > > > > > > > > Andrii > > > > > > > > > > > > >> > > > > Biletskyi > > > > > > > > > > > > >> > > > > > < > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > andrii.bilets...@stealth.ly> > > > > > > > > > > wrote: > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > Hi all, > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > I've updated > > KIP > > > > > page, > > > > > > > > fixed > > > > > > > > > / > > > > > > > > > > > > >> aligned > > > > > > > > > > > > >> > > > > document > > > > > > > > > > > > >> > > > > > > > >> structure. > > > > > > > > > > > > >> > > > > > > > >> >>>> Also I > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > added > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > some > > > > > ... > > > > [Message clipped] > -- -- Guozhang
