Hm, actually the ticket you linked, Guozhang, brings as back to the problem what should be considered a post-condition for each of the admin commands. In my understanding:
1) CreateTopic - broker created /brokers/topics/<topic> (Not the controller picked up changes from zk and broadcasted LeaderAndIsr and UpdateMetadata) 2) AlterTopic - same as 1) - broker changed assignment data in zookeeper or created admin path for topic config change 3) DeleteTopic - admin path /admin/delete_topics is created 4) ReassignPartitions and PreferredReplica - corresponding admin path is created Now what can be considered a completed operation from the client's perspective? 1) Topic is created once corresponding data is in zk (I remember there were some thoughts that it'd be good to consider topic created once all replicas receive information about it and thus clients can produce/consume from it, but as was discussed this seems to be a hard thing to do) 2) Probably same as 1), so right after AlterTopic is issued 3) The topic has been removed from /brokers/topics 4) ReassignPartitions and PrefferedReplica were discussed earlier - in short the former is completed once partition state info in zk matches reassignment request and admin path is empty, the latter - once data in zk shows that head of assignned replicas of the partition and leader is the same replica Thoughts? Thanks, Andrii Biletskyi On Fri, Mar 20, 2015 at 7:10 PM, Guozhang Wang <wangg...@gmail.com> wrote: > I think while loop is fine for supporting blocking, just that we need to > add back off to avoid bombarding brokers with DescribeTopic requests. > > Also I have linked KAFKA-1125 > <https://issues.apache.org/jira/browse/KAFKA-1125> to your proposal, and > when KAFKA-1694 is done this ticket can also be closed. > > Guozhang > > On Fri, Mar 20, 2015 at 9:41 AM, Andrii Biletskyi < > andrii.bilets...@stealth.ly> wrote: > > > Great. > > I want to elaborate this a bit more, to see we are on the same page > > concerning the client code. > > > > So with all topic commands being async a client (AdminClient in our > > case or any other other client people would like to implement) to support > > a blocking operation (which seems to be a natural use-case e.g. for topic > > creation): would have to do: > > 1. issue CreateTopicRequest > > 2. if successful, in a "while" loop send DescribeTopicRequest and > > break the loop once all topics are returned in response (or upon > timeout). > > 3. if unsuccessful throw exception > > Would it be okay? > > > > Thanks, > > Andrii Biletskyi > > > > > > On Fri, Mar 20, 2015 at 6:21 PM, Jun Rao <j...@confluent.io> wrote: > > > > > Andrii, > > > > > > I think you are right. It seems that only ReassignPartitions needs a > > > separate verification request. > > > > > > Thanks, > > > > > > Jun > > > > > > On Thu, Mar 19, 2015 at 9:22 AM, Andrii Biletskyi < > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > Guys, > > > > I like this idea too. Let's stick with that. I'll update KIP > > accordingly. > > > > > > > > I was also thinking we can avoid adding dedicated status check > > > > requests for topic commands. - We have everything in DescribeTopic > > > > for that! E.g.: > > > > User issued CreateTopic - to check the status client sends > > DescribeTopic > > > > and checks whether is something returned for that topic. The same for > > > > alteration, deletion. > > > > Btw, PreferredReplica status can be also checked with > > > DescribeTopicRequest > > > > (head of assigned replicas list == leader). > > > > For ReassignPartitions as discussed we'll need to have a separate > > > Verify... > > > > request. > > > > > > > > Thanks, > > > > Andrii Biletskyi > > > > > > > > > > > > On Thu, Mar 19, 2015 at 6:03 PM, Guozhang Wang <wangg...@gmail.com> > > > wrote: > > > > > > > > > +1 on broker writing to ZK for async handling. I was thinking that > in > > > the > > > > > end state the admin requests would be eventually sent to controller > > > > either > > > > > through re-routing or clients discovering them, instead of letting > > > > > controller listen on ZK admin path. But thinking about it a second > > > time, > > > > I > > > > > think it is actually simpler to let controller manage > > > > > incoming queued-up admin requests through ZK. > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 4:16 PM, Joel Koshy <jjkosh...@gmail.com> > > > wrote: > > > > > > > > > > > +1 as well. I think it helps to keep the rerouting approach > > > orthogonal > > > > > > to this KIP. > > > > > > > > > > > > On Wed, Mar 18, 2015 at 03:40:48PM -0700, Jay Kreps wrote: > > > > > > > I'm +1 on Jun's suggestion as long as it can work for all the > > > > requests. > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 3:35 PM, Jun Rao <j...@confluent.io> > > wrote: > > > > > > > > > > > > > > > Andrii, > > > > > > > > > > > > > > > > I think we agreed on the following. > > > > > > > > > > > > > > > > (a) Admin requests can be sent to and handled by any broker. > > > > > > > > (b) Admin requests are processed asynchronously, at least for > > > now. > > > > > > That is, > > > > > > > > when the client gets a response, it just means that the > request > > > is > > > > > > > > initiated, but not necessarily completed. Then, it's up to > the > > > > client > > > > > > to > > > > > > > > issue another request to check the status for completion. > > > > > > > > > > > > > > > > To support (a), we were thinking of doing request forwarding > to > > > the > > > > > > > > controller (utilizing KAFKA-1912). I am making an alternative > > > > > proposal. > > > > > > > > Basically, the broker can just write to ZooKeeper to inform > the > > > > > > controller > > > > > > > > about the request. For example, to handle > > partitionReassignment, > > > > the > > > > > > broker > > > > > > > > will just write the requested partitions to > > > > > /admin/reassign_partitions > > > > > > > > (like what AdminUtils currently does) and then send a > response > > to > > > > the > > > > > > > > client. This shouldn't take long and the implementation will > be > > > > > simpler > > > > > > > > than forwarding the requests to the controller through RPC. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 3:03 PM, Andrii Biletskyi < > > > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > > > > > Jun, > > > > > > > > > > > > > > > > > > I might be wrong but didn't we agree we will let any broker > > > from > > > > > the > > > > > > > > > cluster handle *long-running* admin requests (at this time > > > > > > > > preferredReplica > > > > > > > > > and > > > > > > > > > reassignPartitions), via zk admin path. Thus CreateTopics > etc > > > > > should > > > > > > be > > > > > > > > > sent > > > > > > > > > only to the controller. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 11:55 PM, Jun Rao < > j...@confluent.io> > > > > > wrote: > > > > > > > > > > > > > > > > > > > Joel, Andril, > > > > > > > > > > > > > > > > > > > > I think we agreed that those admin requests can be issued > > to > > > > any > > > > > > > > broker. > > > > > > > > > > Because of that, there doesn't seem to be a strong need > to > > > know > > > > > the > > > > > > > > > > controller. So, perhaps we can proceed by not making any > > > change > > > > > to > > > > > > the > > > > > > > > > > format of TMR right now. When we start using create topic > > > > request > > > > > > in > > > > > > > > the > > > > > > > > > > producer, we will need a new version of TMR that doesn't > > > > trigger > > > > > > auto > > > > > > > > > topic > > > > > > > > > > creation. But that can be done later. > > > > > > > > > > > > > > > > > > > > As a first cut implementation, I think the broker can > just > > > > write > > > > > > to ZK > > > > > > > > > > directly for > > > > > > > > > > > > > > > createToipic/alterTopic/reassignPartitions/preferredLeaderElection > > > > > > > > > > requests, instead of forwarding them to the controller. > > This > > > > will > > > > > > > > > simplify > > > > > > > > > > the implementation on the broker side. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 11:58 AM, Joel Koshy < > > > > > jjkosh...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > (Thanks Andrii for the summary) > > > > > > > > > > > > > > > > > > > > > > For (1) yes we will circle back on that shortly after > > > syncing > > > > > up > > > > > > in > > > > > > > > > > > person. I think it is close to getting committed > although > > > > > > development > > > > > > > > > > > for KAFKA-1927 can probably begin without it. > > > > > > > > > > > > > > > > > > > > > > There is one more item we covered at the hangout. i.e., > > > > whether > > > > > > we > > > > > > > > > > > want to add the coordinator to the topic metadata > > response > > > or > > > > > > provide > > > > > > > > > > > a clearer ClusterMetadataRequest. > > > > > > > > > > > > > > > > > > > > > > There are two reasons I think we should try and avoid > > > adding > > > > > the > > > > > > > > > > > field: > > > > > > > > > > > - It is irrelevant to topic metadata > > > > > > > > > > > - If we finally do request rerouting in Kafka then the > > > field > > > > > > would > > > > > > > > add > > > > > > > > > > > little to no value. (It still helps to have a > separate > > > > > > > > > > > ClusterMetadataRequest to query for cluster-wide > > > > information > > > > > > such > > > > > > > > as > > > > > > > > > > > 'which broker is the controller?' as Joe mentioned.) > > > > > > > > > > > > > > > > > > > > > > I think it would be cleaner to have an explicit > > > > > > > > ClusterMetadataRequest > > > > > > > > > > > that you can send to any broker in order to obtain the > > > > > controller > > > > > > > > (and > > > > > > > > > > > in the future possibly other cluster-wide > information). I > > > > think > > > > > > the > > > > > > > > > > > main argument against doing this and instead adding it > to > > > the > > > > > > topic > > > > > > > > > > > metadata response was convenience - i.e., you don't > have > > to > > > > > > discover > > > > > > > > > > > the controller in advance. However, I don't see much > > actual > > > > > > > > > > > benefit/convenience in this and in fact think it is a > > > > > non-issue. > > > > > > Let > > > > > > > > > > > me know if I'm overlooking something here. > > > > > > > > > > > > > > > > > > > > > > As an example, say we need to initiate partition > > > reassignment > > > > > by > > > > > > > > > > > issuing the new ReassignPartitionsRequest to the > > controller > > > > > > (assume > > > > > > > > we > > > > > > > > > > > already have the desired manual partition assignment). > > If > > > we > > > > > > are to > > > > > > > > > > > augment topic metadata response then the flow be > > something > > > > like > > > > > > this > > > > > > > > : > > > > > > > > > > > > > > > > > > > > > > - Issue topic metadata request to any broker (and > > discover > > > > the > > > > > > > > > > > controller > > > > > > > > > > > - Connect to controller if required (i.e., if the > broker > > > > above > > > > > != > > > > > > > > > > > controller) > > > > > > > > > > > - Issue the partition reassignment request to the > > > controller. > > > > > > > > > > > > > > > > > > > > > > With an explicit cluster metadata request it would be: > > > > > > > > > > > - Issue cluster metadata request to any broker > > > > > > > > > > > - Connect to controller if required (i.e., if the > broker > > > > above > > > > > != > > > > > > > > > > > controller) > > > > > > > > > > > - Issue the partition reassignment request > > > > > > > > > > > > > > > > > > > > > > So it seems to add little practical value and bloats > > topic > > > > > > metadata > > > > > > > > > > > response with an irrelevant detail. > > > > > > > > > > > > > > > > > > > > > > The other angle to this is the following - is it a > matter > > > of > > > > > > naming? > > > > > > > > > > > Should we just rename topic metadata request/response > to > > > just > > > > > > > > > > > MetadataRequest/Response and add cluster metadata to > it? > > By > > > > > that > > > > > > same > > > > > > > > > > > token should we also allow querying for the consumer > > > > > coordinator > > > > > > (and > > > > > > > > > > > in future transaction coordinator) as well? This leads > > to a > > > > > > bloated > > > > > > > > > > > request which isn't very appealing and altogether > > > confusing. > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > Joel > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 09:34:12AM -0700, Jun Rao > wrote: > > > > > > > > > > > > Andri, > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the summary. > > > > > > > > > > > > > > > > > > > > > > > > 1. I just realized that in order to start working on > > > > > > KAFKA-1927, we > > > > > > > > > > will > > > > > > > > > > > > need to merge the changes to OffsetCommitRequest > (from > > > > 0.8.2) > > > > > > to > > > > > > > > > trunk. > > > > > > > > > > > > This is planned to be done as part of KAFKA-1634. So, > > we > > > > will > > > > > > need > > > > > > > > > > > Guozhang > > > > > > > > > > > > and Joel's help to wrap this up. > > > > > > > > > > > > > > > > > > > > > > > > 2. Thinking about this a bit more, if the semantic of > > > those > > > > > > "write" > > > > > > > > > > > > requests is async (i.e., after the client gets a > > > response, > > > > it > > > > > > just > > > > > > > > > > means > > > > > > > > > > > > that the operation is initiated, but not necessarily > > > > > > completed), we > > > > > > > > > > don't > > > > > > > > > > > > really need to forward the requests to the > controller. > > > > > > Instead, the > > > > > > > > > > > > receiving broker can just write the operation to ZK > as > > > the > > > > > > admin > > > > > > > > > > command > > > > > > > > > > > > line tool previously does. This will simplify the > > > > > > implementation. > > > > > > > > > > > > > > > > > > > > > > > > 8. There is another implementation detail for > describe > > > > topic. > > > > > > > > > Ideally, > > > > > > > > > > we > > > > > > > > > > > > want to read the topic config from the broker cache, > > > > instead > > > > > of > > > > > > > > > > > ZooKeeper. > > > > > > > > > > > > Currently, every broker reads the topic-level config > > for > > > > all > > > > > > > > topics. > > > > > > > > > > > > However, it ignores those for topics not hosted on > > > itself. > > > > > So, > > > > > > we > > > > > > > > may > > > > > > > > > > > need > > > > > > > > > > > > to change TopicConfigManager a bit so that it caches > > the > > > > > > configs > > > > > > > > for > > > > > > > > > > all > > > > > > > > > > > > topics. > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2015 at 1:13 PM, Andrii Biletskyi < > > > > > > > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Guys, > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for a great discussion! > > > > > > > > > > > > > Here are the actions points: > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Q: Get rid of all scala requests objects, use > java > > > > > > protocol > > > > > > > > > > > definitions. > > > > > > > > > > > > > A: Gwen kindly took that (KAFKA-1927). It's > > > important > > > > > to > > > > > > > > speed > > > > > > > > > up > > > > > > > > > > > > > review procedure > > > > > > > > > > > > > there since this ticket blocks other > > important > > > > > > changes. > > > > > > > > > > > > > > > > > > > > > > > > > > 2. Q: Generic re-reroute facility vs client > > maintaining > > > > > > cluster > > > > > > > > > > state. > > > > > > > > > > > > > A: Jay has added pseudo code to KAFKA-1912 - > need > > > to > > > > > > consider > > > > > > > > > > > whether > > > > > > > > > > > > > this will be > > > > > > > > > > > > > easy to implement as a server-side feature > > > > > (comments > > > > > > are > > > > > > > > > > > > > welcomed!). > > > > > > > > > > > > > > > > > > > > > > > > > > 3. Q: Controller field in wire protocol. > > > > > > > > > > > > > A: This might be useful for clients, add this > to > > > > > > > > > > > TopicMetadataResponse > > > > > > > > > > > > > (already in KIP). > > > > > > > > > > > > > > > > > > > > > > > > > > 4. Q: Decoupling topic creation from TMR. > > > > > > > > > > > > > A: I will add proposed by Jun solution (using > > > > clientId > > > > > > for > > > > > > > > > that) > > > > > > > > > > > to the > > > > > > > > > > > > > KIP. > > > > > > > > > > > > > > > > > > > > > > > > > > 5. Q: Bumping new versions of TMR vs grabbing all > > > > protocol > > > > > > > > changes > > > > > > > > > in > > > > > > > > > > > one > > > > > > > > > > > > > version. > > > > > > > > > > > > > A: It was decided to try to gather all changes > to > > > > > > protocol > > > > > > > > > > (before > > > > > > > > > > > > > release). > > > > > > > > > > > > > In case of TMR it worth checking: > KAFKA-2020 > > > and > > > > > > KIP-13 > > > > > > > > > > > (quotas) > > > > > > > > > > > > > > > > > > > > > > > > > > 6. Q: JSON lib is needed to deserialize user's > input > > in > > > > CLI > > > > > > tool. > > > > > > > > > > > > > A: Use jackson for that, /tools project is a > > > separate > > > > > > jar so > > > > > > > > > > > shouldn't > > > > > > > > > > > > > be a big deal. > > > > > > > > > > > > > > > > > > > > > > > > > > 7. Q: VerifyReassingPartitions vs generic status > > check > > > > > > command. > > > > > > > > > > > > > A: For long-running requests like reassign > > > > partitions > > > > > > > > > *progress* > > > > > > > > > > > check > > > > > > > > > > > > > request is useful, > > > > > > > > > > > > > it makes sense to introduce it. > > > > > > > > > > > > > > > > > > > > > > > > > > Please add, correct me if I missed something. > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2015 at 6:20 PM, Andrii Biletskyi < > > > > > > > > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Joel, > > > > > > > > > > > > > > > > > > > > > > > > > > > > You are right, I removed ClusterMetadata because > we > > > > have > > > > > > > > > partially > > > > > > > > > > > > > > what we need in TopicMetadata. Also, as Jay > pointed > > > out > > > > > > > > earlier, > > > > > > > > > we > > > > > > > > > > > > > > would like to have "orthogonal" API, but at the > > same > > > > time > > > > > > we > > > > > > > > need > > > > > > > > > > > > > > to be backward compatible. > > > > > > > > > > > > > > > > > > > > > > > > > > > > But I like your idea and even have some other > > > arguments > > > > > for > > > > > > > > this > > > > > > > > > > > option: > > > > > > > > > > > > > > There is also DescribeTopicRequest which was > > proposed > > > > in > > > > > > this > > > > > > > > > KIP, > > > > > > > > > > > > > > it returns topic configs, partitions, replication > > > > factor > > > > > > plus > > > > > > > > > > > partition > > > > > > > > > > > > > > ISR, ASR, > > > > > > > > > > > > > > leader replica. The later part is really already > > > there > > > > in > > > > > > > > > > > > > > TopicMetadataRequest. > > > > > > > > > > > > > > So again we'll have to add stuff to TMR, not to > > > > duplicate > > > > > > some > > > > > > > > > info > > > > > > > > > > > in > > > > > > > > > > > > > > newly added requests. However, this way we'll end > > up > > > > with > > > > > > > > > "monster" > > > > > > > > > > > > > > request which returns cluster metadata, topic > > > > replication > > > > > > and > > > > > > > > > > config > > > > > > > > > > > info > > > > > > > > > > > > > > plus partition replication data. Seems logical to > > > split > > > > > > TMR to > > > > > > > > > > > > > > - ClusterMetadata (brokers + controller, maybe > smth > > > > else) > > > > > > > > > > > > > > - TopicMetadata (topic info + partition details) > > > > > > > > > > > > > > But since current TMR is involved in lots of > places > > > > > > (including > > > > > > > > > > > network > > > > > > > > > > > > > > client, > > > > > > > > > > > > > > as I understand) this might be very serious > change > > > and > > > > it > > > > > > > > > probably > > > > > > > > > > > makes > > > > > > > > > > > > > > sense to stick with current approach. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2015 at 5:29 PM, Joel Koshy < > > > > > > > > jjkosh...@gmail.com > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > >> I may be missing some context but hopefully this > > > will > > > > > > also be > > > > > > > > > > > covered > > > > > > > > > > > > > >> today: I thought the earlier proposal where > there > > > was > > > > an > > > > > > > > > explicit > > > > > > > > > > > > > >> ClusterMetadata request was clearer and > explicit. > > > > During > > > > > > the > > > > > > > > > > course > > > > > > > > > > > of > > > > > > > > > > > > > >> this thread I think the conclusion was that the > > main > > > > > need > > > > > > was > > > > > > > > > for > > > > > > > > > > > > > >> controller information and that can be rolled > into > > > the > > > > > > topic > > > > > > > > > > > metadata > > > > > > > > > > > > > >> response but that seems a bit irrelevant to > topic > > > > > > metadata. > > > > > > > > > FWIW I > > > > > > > > > > > > > >> think the full broker-list is also irrelevant to > > > topic > > > > > > > > metadata, > > > > > > > > > > but > > > > > > > > > > > > > >> it is already there and in use. I think there is > > > still > > > > > > room > > > > > > > > for > > > > > > > > > an > > > > > > > > > > > > > >> explicit ClusterMetadata request since there may > > be > > > > > other > > > > > > > > > > > > > >> cluster-level information that we may want to > add > > > over > > > > > > time > > > > > > > > (and > > > > > > > > > > > that > > > > > > > > > > > > > >> have nothing to do with topic metadata). > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> On Tue, Mar 17, 2015 at 02:45:30PM +0200, Andrii > > > > > Biletskyi > > > > > > > > > wrote: > > > > > > > > > > > > > >> > Jun, > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > 101. Okay, if you say that such use case is > > > > > important. I > > > > > > > > also > > > > > > > > > > > think > > > > > > > > > > > > > >> > using clientId for these purposes is fine - if > > we > > > > > > already > > > > > > > > have > > > > > > > > > > > this > > > > > > > > > > > > > >> field > > > > > > > > > > > > > >> > as part of all Wire protocol messages, why not > > use > > > > > that. > > > > > > > > > > > > > >> > I will update KIP-4 page if nobody has other > > ideas > > > > > > (which > > > > > > > > may > > > > > > > > > > > come up > > > > > > > > > > > > > >> > during the call today). > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > 102.1 Agree, I'll update the KIP accordingly. > I > > > > think > > > > > > we can > > > > > > > > > add > > > > > > > > > > > new, > > > > > > > > > > > > > >> > fine-grained error codes if some error code > > > received > > > > > in > > > > > > > > > specific > > > > > > > > > > > case > > > > > > > > > > > > > >> > won't give enough context to return a > > descriptive > > > > > error > > > > > > > > > message > > > > > > > > > > > for > > > > > > > > > > > > > >> user. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > Look forward to discussing all outstanding > > issues > > > in > > > > > > detail > > > > > > > > > > today > > > > > > > > > > > > > during > > > > > > > > > > > > > >> > the call. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > Thanks, > > > > > > > > > > > > > >> > Andrii Biletskyi > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > On Mon, Mar 16, 2015 at 10:59 PM, Jun Rao < > > > > > > j...@confluent.io > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > 101. There may be a use case where you only > > want > > > > the > > > > > > > > topics > > > > > > > > > to > > > > > > > > > > > be > > > > > > > > > > > > > >> created > > > > > > > > > > > > > >> > > manually by admins. Currently, you can do > that > > > by > > > > > > > > disabling > > > > > > > > > > auto > > > > > > > > > > > > > topic > > > > > > > > > > > > > >> > > creation and issue topic creation from the > > > > > > TopicCommand. > > > > > > > > If > > > > > > > > > we > > > > > > > > > > > > > >> disable auto > > > > > > > > > > > > > >> > > topic creation completely on the broker and > > > don't > > > > > > have a > > > > > > > > way > > > > > > > > > > to > > > > > > > > > > > > > >> distinguish > > > > > > > > > > > > > >> > > between topic creation requests from the > > regular > > > > > > clients > > > > > > > > and > > > > > > > > > > the > > > > > > > > > > > > > >> admin, we > > > > > > > > > > > > > >> > > can't support manual topic creation any > more. > > I > > > > was > > > > > > > > thinking > > > > > > > > > > > that > > > > > > > > > > > > > >> another > > > > > > > > > > > > > >> > > way of distinguishing the clients making the > > > topic > > > > > > > > creation > > > > > > > > > > > requests > > > > > > > > > > > > > >> is > > > > > > > > > > > > > >> > > using clientId. For example, the admin tool > > can > > > > set > > > > > > it to > > > > > > > > > > > something > > > > > > > > > > > > > >> like > > > > > > > > > > > > > >> > > admin and the broker can treat that clientId > > > > > > specially. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > Also, there is a related discussion in > > > KAFKA-2020. > > > > > > > > > Currently, > > > > > > > > > > > we do > > > > > > > > > > > > > >> the > > > > > > > > > > > > > >> > > following in TopicMetadataResponse: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > 1. If leader is not available, we set the > > > > partition > > > > > > level > > > > > > > > > > error > > > > > > > > > > > code > > > > > > > > > > > > > >> to > > > > > > > > > > > > > >> > > LeaderNotAvailable. > > > > > > > > > > > > > >> > > 2. If a non-leader replica is not available, > > we > > > > take > > > > > > that > > > > > > > > > > > replica > > > > > > > > > > > > > out > > > > > > > > > > > > > >> of > > > > > > > > > > > > > >> > > the assigned replica list and isr in the > > > response. > > > > > As > > > > > > an > > > > > > > > > > > indication > > > > > > > > > > > > > >> for > > > > > > > > > > > > > >> > > doing that, we set the partition level error > > > code > > > > to > > > > > > > > > > > > > >> ReplicaNotAvailable. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > This has a few problems. First, > > > > ReplicaNotAvailable > > > > > > > > probably > > > > > > > > > > > > > >> shouldn't be > > > > > > > > > > > > > >> > > an error, at least for the normal > > > > producer/consumer > > > > > > > > clients > > > > > > > > > > that > > > > > > > > > > > > > just > > > > > > > > > > > > > >> want > > > > > > > > > > > > > >> > > to find out the leader. Second, it can > happen > > > that > > > > > > both > > > > > > > > the > > > > > > > > > > > leader > > > > > > > > > > > > > and > > > > > > > > > > > > > >> > > another replica are not available at the > same > > > > time. > > > > > > There > > > > > > > > is > > > > > > > > > > no > > > > > > > > > > > > > error > > > > > > > > > > > > > >> code > > > > > > > > > > > > > >> > > to indicate both. Third, even if a replica > is > > > not > > > > > > > > available, > > > > > > > > > > > it's > > > > > > > > > > > > > >> still > > > > > > > > > > > > > >> > > useful to return its replica id since some > > > clients > > > > > > (e.g. > > > > > > > > > admin > > > > > > > > > > > tool) > > > > > > > > > > > > > >> may > > > > > > > > > > > > > >> > > still make use of it. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > One way to address this issue is to always > > > return > > > > > the > > > > > > > > > replica > > > > > > > > > > > id for > > > > > > > > > > > > > >> > > leader, assigned replicas, and isr > regardless > > of > > > > > > whether > > > > > > > > the > > > > > > > > > > > > > >> corresponding > > > > > > > > > > > > > >> > > broker is live or not. Since we also return > > the > > > > list > > > > > > of > > > > > > > > live > > > > > > > > > > > > > brokers, > > > > > > > > > > > > > >> the > > > > > > > > > > > > > >> > > client can figure out whether a leader or a > > > > replica > > > > > is > > > > > > > > live > > > > > > > > > or > > > > > > > > > > > not > > > > > > > > > > > > > >> and act > > > > > > > > > > > > > >> > > accordingly. This way, we don't need to set > > the > > > > > > partition > > > > > > > > > > level > > > > > > > > > > > > > error > > > > > > > > > > > > > >> code > > > > > > > > > > > > > >> > > when the leader or a replica is not > available. > > > > This > > > > > > > > doesn't > > > > > > > > > > > change > > > > > > > > > > > > > >> the wire > > > > > > > > > > > > > >> > > protocol, but does change the semantics. > Since > > > we > > > > > are > > > > > > > > > evolving > > > > > > > > > > > the > > > > > > > > > > > > > >> protocol > > > > > > > > > > > > > >> > > of TopicMetadataRequest here, we can > > potentially > > > > > > piggyback > > > > > > > > > the > > > > > > > > > > > > > change. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > 102.1 For those types of errors due to > invalid > > > > > input, > > > > > > > > > > shouldn't > > > > > > > > > > > we > > > > > > > > > > > > > >> just > > > > > > > > > > > > > >> > > guard it at parameter validation time and > > throw > > > > > > > > > > > > > >> InvalidArgumentException > > > > > > > > > > > > > >> > > without even sending the request to the > > broker? > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > Thanks, > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > Jun > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > On Mon, Mar 16, 2015 at 10:37 AM, Andrii > > > > Biletskyi < > > > > > > > > > > > > > >> > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > Jun, > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > Answering your questions: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > 101. If I understand you correctly, you > are > > > > saying > > > > > > > > future > > > > > > > > > > > producer > > > > > > > > > > > > > >> > > versions > > > > > > > > > > > > > >> > > > (which > > > > > > > > > > > > > >> > > > will be ported to TMR_V1) won't be able to > > > > > > automatically > > > > > > > > > > > create > > > > > > > > > > > > > >> topic (if > > > > > > > > > > > > > >> > > > we > > > > > > > > > > > > > >> > > > unconditionally remove topic creation from > > > > there). > > > > > > But > > > > > > > > we > > > > > > > > > > > need to > > > > > > > > > > > > > >> this > > > > > > > > > > > > > >> > > > preserve logic. > > > > > > > > > > > > > >> > > > Ok, about your proposal: I'm not a big fan > > > too, > > > > > > when it > > > > > > > > > > comes > > > > > > > > > > > to > > > > > > > > > > > > > >> > > > differentiating > > > > > > > > > > > > > >> > > > clients directly in protocol schema. And > > also > > > > I'm > > > > > > not > > > > > > > > > sure I > > > > > > > > > > > > > >> understand > > > > > > > > > > > > > >> > > at > > > > > > > > > > > > > >> > > > all why > > > > > > > > > > > > > >> > > > auto.create.topics.enable is a server side > > > > > > > > configuration. > > > > > > > > > > Can > > > > > > > > > > > we > > > > > > > > > > > > > >> > > deprecate > > > > > > > > > > > > > >> > > > this setting > > > > > > > > > > > > > >> > > > in future versions, add this setting to > > > producer > > > > > and > > > > > > > > based > > > > > > > > > > on > > > > > > > > > > > that > > > > > > > > > > > > > >> upon > > > > > > > > > > > > > >> > > > receiving > > > > > > > > > > > > > >> > > > UnknownTopic create topic explicitly by a > > > > separate > > > > > > > > > producer > > > > > > > > > > > call > > > > > > > > > > > > > via > > > > > > > > > > > > > >> > > > adminClient? > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > 102.1. Hm, yes. It's because we want to > > > support > > > > > > batching > > > > > > > > > and > > > > > > > > > > > at > > > > > > > > > > > > > the > > > > > > > > > > > > > >> same > > > > > > > > > > > > > >> > > > time we > > > > > > > > > > > > > >> > > > want to give descriptive error messages > for > > > > > clients. > > > > > > > > Since > > > > > > > > > > > > > >> AdminClient > > > > > > > > > > > > > >> > > > holds the context > > > > > > > > > > > > > >> > > > to construct such messages (e.g. > AdminClient > > > > layer > > > > > > can > > > > > > > > > know > > > > > > > > > > > that > > > > > > > > > > > > > >> > > > InvalidArgumentsCode > > > > > > > > > > > > > >> > > > means two cases: either invalid number - > > e.g. > > > > -1; > > > > > or > > > > > > > > > > > > > >> replication-factor > > > > > > > > > > > > > >> > > was > > > > > > > > > > > > > >> > > > provided while > > > > > > > > > > > > > >> > > > partitions argument wasn't) - I wrapped > > > > responses > > > > > in > > > > > > > > > > > Exceptions. > > > > > > > > > > > > > >> But I'm > > > > > > > > > > > > > >> > > > open to any > > > > > > > > > > > > > >> > > > other ideas, this was just initial > version. > > > > > > > > > > > > > >> > > > 102.2. Yes, I agree. I'll change that to > > > > probably > > > > > > some > > > > > > > > > other > > > > > > > > > > > dto. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > Thanks, > > > > > > > > > > > > > >> > > > Andrii Biletskyi > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > On Fri, Mar 13, 2015 at 7:16 PM, Jun Rao < > > > > > > > > > j...@confluent.io> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > Andrii, > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > 101. That's what I was thinking too, but > > it > > > > may > > > > > > not be > > > > > > > > > > that > > > > > > > > > > > > > >> simple. In > > > > > > > > > > > > > >> > > > > TopicMetadataRequest_V1, > > > > > > > > > > > > > >> > > > > we can let it not trigger auto topic > > > creation. > > > > > > Then, > > > > > > > > in > > > > > > > > > > the > > > > > > > > > > > > > >> producer > > > > > > > > > > > > > >> > > > side, > > > > > > > > > > > > > >> > > > > if it gets an UnknownTopicException, it > > can > > > > > > explicitly > > > > > > > > > > > issue a > > > > > > > > > > > > > >> > > > > createTopicRequest for auto topic > > creation. > > > On > > > > > the > > > > > > > > > > consumer > > > > > > > > > > > > > side, > > > > > > > > > > > > > >> it > > > > > > > > > > > > > >> > > will > > > > > > > > > > > > > >> > > > > never issue createTopicRequest. This > works > > > > when > > > > > > auto > > > > > > > > > topic > > > > > > > > > > > > > >> creation is > > > > > > > > > > > > > >> > > > > enabled on the broker side. However, I > am > > > not > > > > > > sure how > > > > > > > > > > > things > > > > > > > > > > > > > >> will work > > > > > > > > > > > > > >> > > > > when auto topic creation is disabled on > > the > > > > > broker > > > > > > > > side. > > > > > > > > > > In > > > > > > > > > > > this > > > > > > > > > > > > > >> case, > > > > > > > > > > > > > >> > > we > > > > > > > > > > > > > >> > > > > want to have a way to manually create a > > > topic, > > > > > > > > > potentially > > > > > > > > > > > > > through > > > > > > > > > > > > > >> > > admin > > > > > > > > > > > > > >> > > > > commands. However, then we need a way to > > > > > > distinguish > > > > > > > > > > > > > >> createTopicRequest > > > > > > > > > > > > > >> > > > > issued from the producer clients and the > > > admin > > > > > > tools. > > > > > > > > > May > > > > > > > > > > > be we > > > > > > > > > > > > > >> can > > > > > > > > > > > > > >> > > add a > > > > > > > > > > > > > >> > > > > new field in createTopicRequest and set > it > > > > > > differently > > > > > > > > > in > > > > > > > > > > > the > > > > > > > > > > > > > >> producer > > > > > > > > > > > > > >> > > > > client and the admin client. However, I > am > > > not > > > > > > sure if > > > > > > > > > > > that's > > > > > > > > > > > > > the > > > > > > > > > > > > > >> best > > > > > > > > > > > > > >> > > > > approach. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > 2. Yes, refactoring existing requests > is a > > > > > > non-trivial > > > > > > > > > > > amount of > > > > > > > > > > > > > >> work. > > > > > > > > > > > > > >> > > I > > > > > > > > > > > > > >> > > > > posted some comments in KAFKA-1927. We > > will > > > > > > probably > > > > > > > > > have > > > > > > > > > > > to fix > > > > > > > > > > > > > >> > > > KAFKA-1927 > > > > > > > > > > > > > >> > > > > first, before adding the new logic in > > > > > KAFKA-1694. > > > > > > > > > > > Otherwise, the > > > > > > > > > > > > > >> > > changes > > > > > > > > > > > > > >> > > > > will be too big. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > 102. About the AdminClient: > > > > > > > > > > > > > >> > > > > 102.1. It's a bit weird that we return > > > > exception > > > > > > in > > > > > > > > the > > > > > > > > > > > api. It > > > > > > > > > > > > > >> seems > > > > > > > > > > > > > >> > > > that > > > > > > > > > > > > > >> > > > > we should either return error code or > > throw > > > an > > > > > > > > exception > > > > > > > > > > > when > > > > > > > > > > > > > >> getting > > > > > > > > > > > > > >> > > the > > > > > > > > > > > > > >> > > > > response state. > > > > > > > > > > > > > >> > > > > 102.2. We probably shouldn't explicitly > > use > > > > the > > > > > > > > request > > > > > > > > > > > object > > > > > > > > > > > > > in > > > > > > > > > > > > > >> the > > > > > > > > > > > > > >> > > > api. > > > > > > > > > > > > > >> > > > > Not every request evolution requires an > > api > > > > > > change. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > Thanks, > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > Jun > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > On Fri, Mar 13, 2015 at 4:08 AM, Andrii > > > > > Biletskyi > > > > > > < > > > > > > > > > > > > > >> > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > Jun, > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > Thanks for you comments. Answers > inline: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > 100. There are a few fields such as > > > > > > > > ReplicaAssignment, > > > > > > > > > > > > > >> > > > > > > ReassignPartitionRequest, > > > > > > > > > > > > > >> > > > > > > and PartitionsSerialized that are > > > > > represented > > > > > > as a > > > > > > > > > > > string, > > > > > > > > > > > > > but > > > > > > > > > > > > > >> > > > contain > > > > > > > > > > > > > >> > > > > > > composite structures in json. Could > we > > > > > flatten > > > > > > > > them > > > > > > > > > > out > > > > > > > > > > > > > >> directly in > > > > > > > > > > > > > >> > > > the > > > > > > > > > > > > > >> > > > > > > protocol definition as > arrays/records? > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > Yes, now with Admin Client this looks > a > > > bit > > > > > > weird. > > > > > > > > My > > > > > > > > > > > initial > > > > > > > > > > > > > >> > > > motivation > > > > > > > > > > > > > >> > > > > > was: > > > > > > > > > > > > > >> > > > > > ReassignPartitionCommand accepts input > > in > > > > > json, > > > > > > we > > > > > > > > > want > > > > > > > > > > to > > > > > > > > > > > > > >> remain > > > > > > > > > > > > > >> > > > tools' > > > > > > > > > > > > > >> > > > > > interfaces unchanged, where possible. > > > > > > > > > > > > > >> > > > > > If we port it to deserialized format, > in > > > CLI > > > > > > (/tools > > > > > > > > > > > project) > > > > > > > > > > > > > >> we will > > > > > > > > > > > > > >> > > > > have > > > > > > > > > > > > > >> > > > > > to add some > > > > > > > > > > > > > >> > > > > > json library since /tools is written > in > > > java > > > > > and > > > > > > > > we'll > > > > > > > > > > > need to > > > > > > > > > > > > > >> > > > > deserialize > > > > > > > > > > > > > >> > > > > > json file > > > > > > > > > > > > > >> > > > > > provided by a user. Can we quickly > agree > > > on > > > > > what > > > > > > > > this > > > > > > > > > > > library > > > > > > > > > > > > > >> should > > > > > > > > > > > > > >> > > be > > > > > > > > > > > > > >> > > > > > (Jackson, GSON, whatever)? > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > 101. Does TopicMetadataRequest v1 > still > > > > > trigger > > > > > > auto > > > > > > > > > > topic > > > > > > > > > > > > > >> creation? > > > > > > > > > > > > > >> > > > This > > > > > > > > > > > > > >> > > > > > > will be a bit weird now that we > have a > > > > > > separate > > > > > > > > > topic > > > > > > > > > > > > > >> creation api. > > > > > > > > > > > > > >> > > > > Have > > > > > > > > > > > > > >> > > > > > > you thought about how the new > > > > > > createTopicRequest > > > > > > > > and > > > > > > > > > > > > > >> > > > > TopicMetadataRequest > > > > > > > > > > > > > >> > > > > > > v1 will be used in the > > producer/consumer > > > > > > client, > > > > > > > > in > > > > > > > > > > > addition > > > > > > > > > > > > > >> to > > > > > > > > > > > > > >> > > admin > > > > > > > > > > > > > >> > > > > > > tools? For example, ideally, we > don't > > > want > > > > > > > > > > > > > >> TopicMetadataRequest > > > > > > > > > > > > > >> > > from > > > > > > > > > > > > > >> > > > > the > > > > > > > > > > > > > >> > > > > > > consumer to trigger auto topic > > creation. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > I agree, this strange logic should be > > > fixed. > > > > > > I'm not > > > > > > > > > > > confident > > > > > > > > > > > > > >> in > > > > > > > > > > > > > >> > > this > > > > > > > > > > > > > >> > > > > > Kafka part so > > > > > > > > > > > > > >> > > > > > correct me if I'm wrong, but it > doesn't > > > look > > > > > > like a > > > > > > > > > hard > > > > > > > > > > > thing > > > > > > > > > > > > > >> to > > > > > > > > > > > > > >> > > do, I > > > > > > > > > > > > > >> > > > > > think we can > > > > > > > > > > > > > >> > > > > > leverage AdminClient for that in > > Producer > > > > and > > > > > > > > > > > unconditionally > > > > > > > > > > > > > >> remove > > > > > > > > > > > > > >> > > > > topic > > > > > > > > > > > > > >> > > > > > creation from the > > TopicMetadataRequest_V1. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > 2. I think Jay meant getting rid of > > scala > > > > > > classes > > > > > > > > > > > > > >> > > > > > > like HeartbeatRequestAndHeader and > > > > > > > > > > > > > >> HeartbeatResponseAndHeader. We > > > > > > > > > > > > > >> > > did > > > > > > > > > > > > > >> > > > > > that > > > > > > > > > > > > > >> > > > > > > as a stop-gap thing when adding the > > new > > > > > > requests > > > > > > > > for > > > > > > > > > > the > > > > > > > > > > > > > >> consumers. > > > > > > > > > > > > > >> > > > > > > However, the long term plan is to > get > > > rid > > > > of > > > > > > all > > > > > > > > > those > > > > > > > > > > > and > > > > > > > > > > > > > >> just > > > > > > > > > > > > > >> > > reuse > > > > > > > > > > > > > >> > > > > the > > > > > > > > > > > > > >> > > > > > > java request/response in the client. > > > Since > > > > > > this > > > > > > > > KIP > > > > > > > > > > > proposes > > > > > > > > > > > > > >> to > > > > > > > > > > > > > >> > > add a > > > > > > > > > > > > > >> > > > > > > significant number of new requests, > > > > perhaps > > > > > we > > > > > > > > > should > > > > > > > > > > > bite > > > > > > > > > > > > > the > > > > > > > > > > > > > >> > > bullet > > > > > > > > > > > > > >> > > > > to > > > > > > > > > > > > > >> > > > > > > clean up the existing scala requests > > > first > > > > > > before > > > > > > > > > > > adding new > > > > > > > > > > > > > >> ones? > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > Yes, looks like I misunderstood the > > point > > > of > > > > > > > > > > > > > >> ...RequestAndHeader. > > > > > > > > > > > > > >> > > > Okay, I > > > > > > > > > > > > > >> > > > > > will > > > > > > > > > > > > > >> > > > > > rework that. The only thing is that I > > > don't > > > > > see > > > > > > any > > > > > > > > > > > example > > > > > > > > > > > > > how > > > > > > > > > > > > > >> it > > > > > > > > > > > > > >> > > was > > > > > > > > > > > > > >> > > > > done > > > > > > > > > > > > > >> > > > > > for at > > > > > > > > > > > > > >> > > > > > least one existing protocol message. > > Thus, > > > > as > > > > > I > > > > > > > > > > > understand, I > > > > > > > > > > > > > >> have to > > > > > > > > > > > > > >> > > > > think > > > > > > > > > > > > > >> > > > > > how we > > > > > > > > > > > > > >> > > > > > are going to do it. > > > > > > > > > > > > > >> > > > > > Re porting all existing RQ/RP in this > > > patch. > > > > > > Sounds > > > > > > > > > > > > > reasonable, > > > > > > > > > > > > > >> but > > > > > > > > > > > > > >> > > if > > > > > > > > > > > > > >> > > > > it's > > > > > > > > > > > > > >> > > > > > an *obligatory* > > > > > > > > > > > > > >> > > > > > requirement to have Admin KIP done, > I'm > > > > afraid > > > > > > this > > > > > > > > > can > > > > > > > > > > > be a > > > > > > > > > > > > > >> serious > > > > > > > > > > > > > >> > > > > > blocker for us. > > > > > > > > > > > > > >> > > > > > There are 13 protocol messages and all > > > that > > > > > > would > > > > > > > > > > require > > > > > > > > > > > not > > > > > > > > > > > > > >> only > > > > > > > > > > > > > >> > > unit > > > > > > > > > > > > > >> > > > > > tests but quite > > > > > > > > > > > > > >> > > > > > intensive manual testing, no? I'm > afraid > > > I'm > > > > > > not the > > > > > > > > > > > right guy > > > > > > > > > > > > > >> to > > > > > > > > > > > > > >> > > cover > > > > > > > > > > > > > >> > > > > > pretty much all > > > > > > > > > > > > > >> > > > > > Kafka core internals :). Let me know > > your > > > > > > thoughts > > > > > > > > on > > > > > > > > > > this > > > > > > > > > > > > > >> item. Btw > > > > > > > > > > > > > >> > > > > there > > > > > > > > > > > > > >> > > > > > is a ticket to > > > > > > > > > > > > > >> > > > > > follow-up this issue ( > > > > > > > > > > > > > >> > > > > > https://issues.apache.org/jira/browse/KAFKA-2006 > > > > > > > > > > > > > >> > > > ). > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > Thanks, > > > > > > > > > > > > > >> > > > > > Andrii Biletskyi > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > On Fri, Mar 13, 2015 at 6:40 AM, Jun > > Rao < > > > > > > > > > > > j...@confluent.io> > > > > > > > > > > > > > >> wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > Andrii, > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > A few more comments. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > 100. There are a few fields such as > > > > > > > > > ReplicaAssignment, > > > > > > > > > > > > > >> > > > > > > ReassignPartitionRequest, > > > > > > > > > > > > > >> > > > > > > and PartitionsSerialized that are > > > > > represented > > > > > > as a > > > > > > > > > > > string, > > > > > > > > > > > > > but > > > > > > > > > > > > > >> > > > contain > > > > > > > > > > > > > >> > > > > > > composite structures in json. Could > we > > > > > flatten > > > > > > > > them > > > > > > > > > > out > > > > > > > > > > > > > >> directly in > > > > > > > > > > > > > >> > > > the > > > > > > > > > > > > > >> > > > > > > protocol definition as > arrays/records? > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > 101. Does TopicMetadataRequest v1 > > still > > > > > > trigger > > > > > > > > auto > > > > > > > > > > > topic > > > > > > > > > > > > > >> > > creation? > > > > > > > > > > > > > >> > > > > This > > > > > > > > > > > > > >> > > > > > > will be a bit weird now that we > have a > > > > > > separate > > > > > > > > > topic > > > > > > > > > > > > > >> creation api. > > > > > > > > > > > > > >> > > > > Have > > > > > > > > > > > > > >> > > > > > > you thought about how the new > > > > > > createTopicRequest > > > > > > > > and > > > > > > > > > > > > > >> > > > > TopicMetadataRequest > > > > > > > > > > > > > >> > > > > > > v1 will be used in the > > producer/consumer > > > > > > client, > > > > > > > > in > > > > > > > > > > > addition > > > > > > > > > > > > > >> to > > > > > > > > > > > > > >> > > admin > > > > > > > > > > > > > >> > > > > > > tools? For example, ideally, we > don't > > > want > > > > > > > > > > > > > >> TopicMetadataRequest > > > > > > > > > > > > > >> > > from > > > > > > > > > > > > > >> > > > > the > > > > > > > > > > > > > >> > > > > > > consumer to trigger auto topic > > creation. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > 2. I think Jay meant getting rid of > > > scala > > > > > > classes > > > > > > > > > > > > > >> > > > > > > like HeartbeatRequestAndHeader and > > > > > > > > > > > > > >> HeartbeatResponseAndHeader. We > > > > > > > > > > > > > >> > > did > > > > > > > > > > > > > >> > > > > > that > > > > > > > > > > > > > >> > > > > > > as a stop-gap thing when adding the > > new > > > > > > requests > > > > > > > > for > > > > > > > > > > the > > > > > > > > > > > > > >> consumers. > > > > > > > > > > > > > >> > > > > > > However, the long term plan is to > get > > > rid > > > > of > > > > > > all > > > > > > > > > those > > > > > > > > > > > and > > > > > > > > > > > > > >> just > > > > > > > > > > > > > >> > > reuse > > > > > > > > > > > > > >> > > > > the > > > > > > > > > > > > > >> > > > > > > java request/response in the client. > > > Since > > > > > > this > > > > > > > > KIP > > > > > > > > > > > proposes > > > > > > > > > > > > > >> to > > > > > > > > > > > > > >> > > add a > > > > > > > > > > > > > >> > > > > > > significant number of new requests, > > > > perhaps > > > > > we > > > > > > > > > should > > > > > > > > > > > bite > > > > > > > > > > > > > the > > > > > > > > > > > > > >> > > bullet > > > > > > > > > > > > > >> > > > > to > > > > > > > > > > > > > >> > > > > > > clean up the existing scala requests > > > first > > > > > > before > > > > > > > > > > > adding new > > > > > > > > > > > > > >> ones? > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > Thanks, > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > Jun > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > On Thu, Mar 12, 2015 at 3:37 PM, > > Andrii > > > > > > Biletskyi > > > > > > > > < > > > > > > > > > > > > > >> > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > Hi, > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > As said above - I list again all > > > > comments > > > > > > from > > > > > > > > > this > > > > > > > > > > > thread > > > > > > > > > > > > > >> so we > > > > > > > > > > > > > >> > > > > > > > can see what's left and finalize > all > > > > > pending > > > > > > > > > issues. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > Comments from Jay: > > > > > > > > > > > > > >> > > > > > > > 1. This is much needed > > functionality, > > > > but > > > > > > there > > > > > > > > > are > > > > > > > > > > a > > > > > > > > > > > lot > > > > > > > > > > > > > >> of the > > > > > > > > > > > > > >> > > so > > > > > > > > > > > > > >> > > > > > let's > > > > > > > > > > > > > >> > > > > > > > really think these protocols > > through. > > > We > > > > > > really > > > > > > > > > want > > > > > > > > > > > to > > > > > > > > > > > > > end > > > > > > > > > > > > > >> up > > > > > > > > > > > > > >> > > > with a > > > > > > > > > > > > > >> > > > > > set > > > > > > > > > > > > > >> > > > > > > > of well thought-out, orthoganol > > apis. > > > > For > > > > > > this > > > > > > > > > > reason > > > > > > > > > > > I > > > > > > > > > > > > > >> think it > > > > > > > > > > > > > >> > > is > > > > > > > > > > > > > >> > > > > > > really > > > > > > > > > > > > > >> > > > > > > > important to think through the end > > > state > > > > > > even if > > > > > > > > > > that > > > > > > > > > > > > > >> includes > > > > > > > > > > > > > >> > > APIs > > > > > > > > > > > > > >> > > > > we > > > > > > > > > > > > > >> > > > > > > > won't implement in the first > phase. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Definitely behind this. Would > > > > > appreciate > > > > > > if > > > > > > > > > there > > > > > > > > > > > are > > > > > > > > > > > > > >> concrete > > > > > > > > > > > > > >> > > > > > > comments > > > > > > > > > > > > > >> > > > > > > > how this can be improved. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 2. Let's please please please wait > > > until > > > > > we > > > > > > have > > > > > > > > > > > switched > > > > > > > > > > > > > >> the > > > > > > > > > > > > > >> > > > server > > > > > > > > > > > > > >> > > > > > over > > > > > > > > > > > > > >> > > > > > > > to the new java protocol > > definitions. > > > If > > > > > we > > > > > > add > > > > > > > > > > upteen > > > > > > > > > > > > > more > > > > > > > > > > > > > >> ad > > > > > > > > > > > > > >> > > hoc > > > > > > > > > > > > > >> > > > > > scala > > > > > > > > > > > > > >> > > > > > > > objects that is just generating > more > > > > work > > > > > > for > > > > > > > > the > > > > > > > > > > > > > >> conversion we > > > > > > > > > > > > > >> > > > know > > > > > > > > > > > > > >> > > > > we > > > > > > > > > > > > > >> > > > > > > > have to do. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - > > removed > > > > > scala > > > > > > > > > > protocol > > > > > > > > > > > > > >> classes. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 3. This proposal introduces a new > > type > > > > of > > > > > > > > optional > > > > > > > > > > > > > >> parameter. > > > > > > > > > > > > > >> > > This > > > > > > > > > > > > > >> > > > is > > > > > > > > > > > > > >> > > > > > > > inconsistent with everything else > in > > > the > > > > > > > > protocol > > > > > > > > > > > where we > > > > > > > > > > > > > >> use -1 > > > > > > > > > > > > > >> > > > or > > > > > > > > > > > > > >> > > > > > some > > > > > > > > > > > > > >> > > > > > > > other marker value. You could > argue > > > > either > > > > > > way > > > > > > > > but > > > > > > > > > > > let's > > > > > > > > > > > > > >> stick > > > > > > > > > > > > > >> > > with > > > > > > > > > > > > > >> > > > > > that > > > > > > > > > > > > > >> > > > > > > > for consistency. For clients that > > > > > > implemented > > > > > > > > the > > > > > > > > > > > protocol > > > > > > > > > > > > > >> in a > > > > > > > > > > > > > >> > > > > better > > > > > > > > > > > > > >> > > > > > > way > > > > > > > > > > > > > >> > > > > > > > than our scala code these basic > > > > primitives > > > > > > are > > > > > > > > > hard > > > > > > > > > > to > > > > > > > > > > > > > >> change. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - > > removed > > > > > > MaybeOf > > > > > > > > > type > > > > > > > > > > > and > > > > > > > > > > > > > >> changed > > > > > > > > > > > > > >> > > > > > protocol > > > > > > > > > > > > > >> > > > > > > > accordingly. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 4. ClusterMetadata: This seems to > > > > > duplicate > > > > > > > > > > > > > >> TopicMetadataRequest > > > > > > > > > > > > > >> > > > > which > > > > > > > > > > > > > >> > > > > > > has > > > > > > > > > > > > > >> > > > > > > > brokers, topics, and partitions. I > > > think > > > > > we > > > > > > > > should > > > > > > > > > > > rename > > > > > > > > > > > > > >> that > > > > > > > > > > > > > >> > > > > request > > > > > > > > > > > > > >> > > > > > > > ClusterMetadataRequest (or just > > > > > > MetadataRequest) > > > > > > > > > and > > > > > > > > > > > > > >> include the > > > > > > > > > > > > > >> > > id > > > > > > > > > > > > > >> > > > > of > > > > > > > > > > > > > >> > > > > > > the > > > > > > > > > > > > > >> > > > > > > > controller. Or are there other > > things > > > we > > > > > > could > > > > > > > > add > > > > > > > > > > > here? > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: I agree. Updated the KIP. Let's > > > > extends > > > > > > > > > > > TopicMetadata > > > > > > > > > > > > > to > > > > > > > > > > > > > >> > > > version 2 > > > > > > > > > > > > > >> > > > > > and > > > > > > > > > > > > > >> > > > > > > > include controller. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 5. We have a tendency to try to > > make a > > > > lot > > > > > > of > > > > > > > > > > requests > > > > > > > > > > > > > that > > > > > > > > > > > > > >> can > > > > > > > > > > > > > >> > > > only > > > > > > > > > > > > > >> > > > > go > > > > > > > > > > > > > >> > > > > > > to > > > > > > > > > > > > > >> > > > > > > > particular nodes. This adds a lot > of > > > > > burden > > > > > > for > > > > > > > > > > client > > > > > > > > > > > > > >> > > > > implementations > > > > > > > > > > > > > >> > > > > > > (it > > > > > > > > > > > > > >> > > > > > > > sounds easy but each discovery can > > > fail > > > > in > > > > > > many > > > > > > > > > > parts > > > > > > > > > > > so > > > > > > > > > > > > > it > > > > > > > > > > > > > >> ends > > > > > > > > > > > > > >> > > up > > > > > > > > > > > > > >> > > > > > > being a > > > > > > > > > > > > > >> > > > > > > > full state machine to do right). I > > > think > > > > > we > > > > > > > > should > > > > > > > > > > > > > consider > > > > > > > > > > > > > >> > > making > > > > > > > > > > > > > >> > > > > > admin > > > > > > > > > > > > > >> > > > > > > > commands and ideally as many of > the > > > > other > > > > > > apis > > > > > > > > as > > > > > > > > > > > possible > > > > > > > > > > > > > >> > > > available > > > > > > > > > > > > > >> > > > > on > > > > > > > > > > > > > >> > > > > > > all > > > > > > > > > > > > > >> > > > > > > > brokers and just redirect to the > > > > > controller > > > > > > on > > > > > > > > the > > > > > > > > > > > broker > > > > > > > > > > > > > >> side. > > > > > > > > > > > > > >> > > > > Perhaps > > > > > > > > > > > > > >> > > > > > > > there would be a general way to > > > > > encapsulate > > > > > > this > > > > > > > > > > > > > re-routing > > > > > > > > > > > > > >> > > > behavior. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: It's a very interesting idea, > but > > > > seems > > > > > > there > > > > > > > > > are > > > > > > > > > > > some > > > > > > > > > > > > > >> > > concerns > > > > > > > > > > > > > >> > > > > > about > > > > > > > > > > > > > >> > > > > > > > this > > > > > > > > > > > > > >> > > > > > > > feature (like performance > > > > considerations, > > > > > > how > > > > > > > > this > > > > > > > > > > > will > > > > > > > > > > > > > >> > > complicate > > > > > > > > > > > > > >> > > > > > server > > > > > > > > > > > > > >> > > > > > > > etc). > > > > > > > > > > > > > >> > > > > > > > I believe this shouldn't be a > > blocker. > > > > If > > > > > > this > > > > > > > > > > > feature is > > > > > > > > > > > > > >> > > > implemented > > > > > > > > > > > > > >> > > > > > at > > > > > > > > > > > > > >> > > > > > > > some > > > > > > > > > > > > > >> > > > > > > > point it won't affect Admin > changes > > - > > > at > > > > > > least > > > > > > > > no > > > > > > > > > > > changes > > > > > > > > > > > > > to > > > > > > > > > > > > > >> > > public > > > > > > > > > > > > > >> > > > > API > > > > > > > > > > > > > >> > > > > > > > will be required. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 6. We should probably normalize > the > > > key > > > > > > value > > > > > > > > > pairs > > > > > > > > > > > used > > > > > > > > > > > > > for > > > > > > > > > > > > > >> > > > configs > > > > > > > > > > > > > >> > > > > > > rather > > > > > > > > > > > > > >> > > > > > > > than embedding a new formatting. > So > > > two > > > > > > strings > > > > > > > > > > rather > > > > > > > > > > > > > than > > > > > > > > > > > > > >> one > > > > > > > > > > > > > >> > > > with > > > > > > > > > > > > > >> > > > > an > > > > > > > > > > > > > >> > > > > > > > internal equals sign. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - > > > > normalized > > > > > > > > configs > > > > > > > > > > and > > > > > > > > > > > > > >> changed > > > > > > > > > > > > > >> > > > > protocol > > > > > > > > > > > > > >> > > > > > > > accordingly. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 7. Is the postcondition of these > > APIs > > > > that > > > > > > the > > > > > > > > > > > command has > > > > > > > > > > > > > >> begun > > > > > > > > > > > > > >> > > or > > > > > > > > > > > > > >> > > > > > that > > > > > > > > > > > > > >> > > > > > > > the command has been completed? It > > is > > > a > > > > > lot > > > > > > more > > > > > > > > > > > usable if > > > > > > > > > > > > > >> the > > > > > > > > > > > > > >> > > > > command > > > > > > > > > > > > > >> > > > > > > has > > > > > > > > > > > > > >> > > > > > > > been completed so you know that if > > you > > > > > > create a > > > > > > > > > > topic > > > > > > > > > > > and > > > > > > > > > > > > > >> then > > > > > > > > > > > > > >> > > > > publish > > > > > > > > > > > > > >> > > > > > to > > > > > > > > > > > > > >> > > > > > > > it you won't get an exception > about > > > > there > > > > > > being > > > > > > > > no > > > > > > > > > > > such > > > > > > > > > > > > > >> topic. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: For long running requests (like > > > > > reassign > > > > > > > > > > > partitions) - > > > > > > > > > > > > > >> the > > > > > > > > > > > > > >> > > post > > > > > > > > > > > > > >> > > > > > > > condition is > > > > > > > > > > > > > >> > > > > > > > command has begun - so we don't > > block > > > > the > > > > > > > > client. > > > > > > > > > In > > > > > > > > > > > case > > > > > > > > > > > > > >> of your > > > > > > > > > > > > > >> > > > > > > example - > > > > > > > > > > > > > >> > > > > > > > topic commands, this will be > > > refactored > > > > > and > > > > > > > > topic > > > > > > > > > > > commands > > > > > > > > > > > > > >> will > > > > > > > > > > > > > >> > > be > > > > > > > > > > > > > >> > > > > > > executed > > > > > > > > > > > > > >> > > > > > > > immediately, since the Controller > > will > > > > > serve > > > > > > > > Admin > > > > > > > > > > > > > requests > > > > > > > > > > > > > >> > > > > > > > (follow-up ticket KAFKA-1777). > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 8. Describe topic and list topics > > > > > duplicate > > > > > > a > > > > > > > > lot > > > > > > > > > of > > > > > > > > > > > stuff > > > > > > > > > > > > > >> in the > > > > > > > > > > > > > >> > > > > > > metadata > > > > > > > > > > > > > >> > > > > > > > request. Is there a reason to give > > > back > > > > > > topics > > > > > > > > > > marked > > > > > > > > > > > for > > > > > > > > > > > > > >> > > > deletion? I > > > > > > > > > > > > > >> > > > > > > feel > > > > > > > > > > > > > >> > > > > > > > like if we just make the > > > post-condition > > > > of > > > > > > the > > > > > > > > > > delete > > > > > > > > > > > > > >> command be > > > > > > > > > > > > > >> > > > that > > > > > > > > > > > > > >> > > > > > the > > > > > > > > > > > > > >> > > > > > > > topic is deleted that will get rid > > of > > > > the > > > > > > need > > > > > > > > for > > > > > > > > > > > this > > > > > > > > > > > > > >> right? > > > > > > > > > > > > > >> > > And > > > > > > > > > > > > > >> > > > it > > > > > > > > > > > > > >> > > > > > > will > > > > > > > > > > > > > >> > > > > > > > be much more intuitive. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - > > removed > > > > > > topics > > > > > > > > > marked > > > > > > > > > > > for > > > > > > > > > > > > > >> deletion > > > > > > > > > > > > > >> > > > in > > > > > > > > > > > > > >> > > > > > > > ListTopicsRequest. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 9. Should we consider batching > these > > > > > > requests? > > > > > > > > We > > > > > > > > > > have > > > > > > > > > > > > > >> generally > > > > > > > > > > > > > >> > > > > tried > > > > > > > > > > > > > >> > > > > > to > > > > > > > > > > > > > >> > > > > > > > allow multiple operations to be > > > batched. > > > > > My > > > > > > > > > > suspicion > > > > > > > > > > > is > > > > > > > > > > > > > >> that > > > > > > > > > > > > > >> > > > without > > > > > > > > > > > > > >> > > > > > > this > > > > > > > > > > > > > >> > > > > > > > we will get a lot of code that > does > > > > > > something > > > > > > > > like > > > > > > > > > > > > > >> > > > > > > > for(topic: > > > adminClient.listTopics()) > > > > > > > > > > > > > >> > > > > > > > > > adminClient.describeTopic(topic) > > > > > > > > > > > > > >> > > > > > > > this code will work great when you > > > test > > > > > on 5 > > > > > > > > > topics > > > > > > > > > > > but > > > > > > > > > > > > > not > > > > > > > > > > > > > >> do as > > > > > > > > > > > > > >> > > > > well > > > > > > > > > > > > > >> > > > > > if > > > > > > > > > > > > > >> > > > > > > > you have 50k. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Updated the KIP - please check > > > "Topic > > > > > > Admin > > > > > > > > > > Schema" > > > > > > > > > > > > > >> section. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 10. I think we should also discuss > > how > > > > we > > > > > > want > > > > > > > > to > > > > > > > > > > > expose a > > > > > > > > > > > > > >> > > > > programmatic > > > > > > > > > > > > > >> > > > > > > JVM > > > > > > > > > > > > > >> > > > > > > > client api for these operations. > > > > Currently > > > > > > > > people > > > > > > > > > > > rely on > > > > > > > > > > > > > >> > > > AdminUtils > > > > > > > > > > > > > >> > > > > > > which > > > > > > > > > > > > > >> > > > > > > > is totally sketchy. I think we > > > probably > > > > > need > > > > > > > > > another > > > > > > > > > > > > > client > > > > > > > > > > > > > >> under > > > > > > > > > > > > > >> > > > > > > clients/ > > > > > > > > > > > > > >> > > > > > > > that exposes administrative > > > > functionality. > > > > > > We > > > > > > > > will > > > > > > > > > > > need > > > > > > > > > > > > > >> this just > > > > > > > > > > > > > >> > > > to > > > > > > > > > > > > > >> > > > > > > > properly test the new apis, I > > suspect. > > > > We > > > > > > should > > > > > > > > > > > figure > > > > > > > > > > > > > out > > > > > > > > > > > > > >> that > > > > > > > > > > > > > >> > > > API. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Updated the KIP - please check > > > "Admin > > > > > > Client" > > > > > > > > > > > section > > > > > > > > > > > > > >> with an > > > > > > > > > > > > > >> > > > > > initial > > > > > > > > > > > > > >> > > > > > > > API proposal. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 11. The other information that > would > > > be > > > > > > really > > > > > > > > > > useful > > > > > > > > > > > to > > > > > > > > > > > > > get > > > > > > > > > > > > > >> > > would > > > > > > > > > > > > > >> > > > be > > > > > > > > > > > > > >> > > > > > > > information about partitions--how > > much > > > > > data > > > > > > is > > > > > > > > in > > > > > > > > > > the > > > > > > > > > > > > > >> partition, > > > > > > > > > > > > > >> > > > what > > > > > > > > > > > > > >> > > > > > are > > > > > > > > > > > > > >> > > > > > > > the segment offsets, what is the > > > log-end > > > > > > offset > > > > > > > > > > (i.e. > > > > > > > > > > > last > > > > > > > > > > > > > >> > > offset), > > > > > > > > > > > > > >> > > > > > what > > > > > > > > > > > > > >> > > > > > > is > > > > > > > > > > > > > >> > > > > > > > the compaction point, etc. I think > > > that > > > > > done > > > > > > > > right > > > > > > > > > > > this > > > > > > > > > > > > > >> would be > > > > > > > > > > > > > >> > > > the > > > > > > > > > > > > > >> > > > > > > > successor to the very awkward > > > > > OffsetRequest > > > > > > we > > > > > > > > > have > > > > > > > > > > > today. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: I removed > > > ConsumerGroupOffsetsRequest > > > > > in > > > > > > the > > > > > > > > > > latest > > > > > > > > > > > > > >> patch. I > > > > > > > > > > > > > >> > > > > believe > > > > > > > > > > > > > >> > > > > > > > this should > > > > > > > > > > > > > >> > > > > > > > be resolved in a separate KIP / > jira > > > > > ticket. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 12. Generally we can do good error > > > > > handling > > > > > > > > > without > > > > > > > > > > > > > needing > > > > > > > > > > > > > >> > > custom > > > > > > > > > > > > > >> > > > > > > > server-side > > > > > > > > > > > > > >> > > > > > > > messages. I.e. generally the > client > > > has > > > > > the > > > > > > > > > context > > > > > > > > > > to > > > > > > > > > > > > > know > > > > > > > > > > > > > >> that > > > > > > > > > > > > > >> > > if > > > > > > > > > > > > > >> > > > > it > > > > > > > > > > > > > >> > > > > > > got > > > > > > > > > > > > > >> > > > > > > > an error that the topic doesn't > > exist > > > to > > > > > say > > > > > > > > > "Topic > > > > > > > > > > X > > > > > > > > > > > > > >> doesn't > > > > > > > > > > > > > >> > > > exist" > > > > > > > > > > > > > >> > > > > > > rather > > > > > > > > > > > > > >> > > > > > > > than "error code 14" (or > whatever). > > > > Maybe > > > > > > there > > > > > > > > > are > > > > > > > > > > > > > specific > > > > > > > > > > > > > >> > > cases > > > > > > > > > > > > > >> > > > > > where > > > > > > > > > > > > > >> > > > > > > > this is hard? If we want to add > > > > > server-side > > > > > > > > error > > > > > > > > > > > messages > > > > > > > > > > > > > >> we > > > > > > > > > > > > > >> > > > really > > > > > > > > > > > > > >> > > > > do > > > > > > > > > > > > > >> > > > > > > > need to do this in a consistent > way > > > > across > > > > > > the > > > > > > > > > > > protocol. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Updated the KIP - please check > > > > > "Protocol > > > > > > > > > Errors" > > > > > > > > > > > > > >> section. I > > > > > > > > > > > > > >> > > > added > > > > > > > > > > > > > >> > > > > > the > > > > > > > > > > > > > >> > > > > > > > comprehensive, fine-grained list > of > > > > error > > > > > > codes. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > Comments from Guozhang: > > > > > > > > > > > > > >> > > > > > > > 13. Describe topic request: it > would > > > be > > > > > > great to > > > > > > > > > go > > > > > > > > > > > beyond > > > > > > > > > > > > > >> just > > > > > > > > > > > > > >> > > > > > batching > > > > > > > > > > > > > >> > > > > > > on > > > > > > > > > > > > > >> > > > > > > > topic name regex for this request. > > For > > > > > > example, > > > > > > > > a > > > > > > > > > > very > > > > > > > > > > > > > >> common use > > > > > > > > > > > > > >> > > > > case > > > > > > > > > > > > > >> > > > > > of > > > > > > > > > > > > > >> > > > > > > > the topic command is to list all > > > topics > > > > > > whose > > > > > > > > > config > > > > > > > > > > > A's > > > > > > > > > > > > > >> value is > > > > > > > > > > > > > >> > > > B. > > > > > > > > > > > > > >> > > > > > With > > > > > > > > > > > > > >> > > > > > > > topic name regex then we have to > > first > > > > > > retrieve > > > > > > > > > > > __all__ > > > > > > > > > > > > > >> topics's > > > > > > > > > > > > > >> > > > > > > > description info and then filter > at > > > the > > > > > > client > > > > > > > > > end, > > > > > > > > > > > which > > > > > > > > > > > > > >> will > > > > > > > > > > > > > >> > > be a > > > > > > > > > > > > > >> > > > > > huge > > > > > > > > > > > > > >> > > > > > > > burden on ZK. > > > > > > > > > > > > > >> > > > > > > > AND > > > > > > > > > > > > > >> > > > > > > > 14. Config K-Vs in create topic: > > this > > > is > > > > > > related > > > > > > > > > to > > > > > > > > > > > the > > > > > > > > > > > > > >> previous > > > > > > > > > > > > > >> > > > > point; > > > > > > > > > > > > > >> > > > > > > > maybe we can add another metadata > > K-V > > > or > > > > > > just a > > > > > > > > > > > metadata > > > > > > > > > > > > > >> string > > > > > > > > > > > > > >> > > > along > > > > > > > > > > > > > >> > > > > > > side > > > > > > > > > > > > > >> > > > > > > > with config K-V in create topic > like > > > we > > > > > did > > > > > > for > > > > > > > > > > offset > > > > > > > > > > > > > >> commit > > > > > > > > > > > > > >> > > > > request. > > > > > > > > > > > > > >> > > > > > > This > > > > > > > > > > > > > >> > > > > > > > field can be quite useful in > storing > > > > > > information > > > > > > > > > > like > > > > > > > > > > > > > >> "owner" of > > > > > > > > > > > > > >> > > > the > > > > > > > > > > > > > >> > > > > > > topic > > > > > > > > > > > > > >> > > > > > > > who issue the create command, etc, > > > which > > > > > is > > > > > > > > quite > > > > > > > > > > > > > important > > > > > > > > > > > > > >> for a > > > > > > > > > > > > > >> > > > > > > > multi-tenant setting. Then in the > > > > describe > > > > > > topic > > > > > > > > > > > request > > > > > > > > > > > > > we > > > > > > > > > > > > > >> can > > > > > > > > > > > > > >> > > > also > > > > > > > > > > > > > >> > > > > > > batch > > > > > > > > > > > > > >> > > > > > > > on regex of the metadata field. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: As discussed it is very > > interesting > > > > but > > > > > > can > > > > > > > > be > > > > > > > > > > > > > >> implemented > > > > > > > > > > > > > >> > > later > > > > > > > > > > > > > >> > > > > > after > > > > > > > > > > > > > >> > > > > > > > we have some basic functionality > > > there. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 15. Today all the admin operations > > are > > > > > > async in > > > > > > > > > the > > > > > > > > > > > sense > > > > > > > > > > > > > >> that > > > > > > > > > > > > > >> > > > > command > > > > > > > > > > > > > >> > > > > > > will > > > > > > > > > > > > > >> > > > > > > > return once it is written in ZK, > and > > > > that > > > > > > is why > > > > > > > > > we > > > > > > > > > > > need > > > > > > > > > > > > > >> extra > > > > > > > > > > > > > >> > > > > > > verification > > > > > > > > > > > > > >> > > > > > > > like > testUtil.waitForTopicCreated() > > / > > > > > verify > > > > > > > > > > partition > > > > > > > > > > > > > >> > > reassignment > > > > > > > > > > > > > >> > > > > > > > request, etc. With admin requests > we > > > > could > > > > > > add a > > > > > > > > > > flag > > > > > > > > > > > to > > > > > > > > > > > > > >> enable / > > > > > > > > > > > > > >> > > > > > disable > > > > > > > > > > > > > >> > > > > > > > synchronous requests; when it is > > > turned > > > > > on, > > > > > > the > > > > > > > > > > > response > > > > > > > > > > > > > >> will not > > > > > > > > > > > > > >> > > > > > return > > > > > > > > > > > > > >> > > > > > > > until the request has been > > completed. > > > > And > > > > > > for > > > > > > > > > async > > > > > > > > > > > > > >> requests we > > > > > > > > > > > > > >> > > can > > > > > > > > > > > > > >> > > > > > add a > > > > > > > > > > > > > >> > > > > > > > "token" field in the response, and > > > then > > > > > only > > > > > > > > need > > > > > > > > > a > > > > > > > > > > > > > general > > > > > > > > > > > > > >> > > "admin > > > > > > > > > > > > > >> > > > > > > > verification request" with the > given > > > > token > > > > > > to > > > > > > > > > check > > > > > > > > > > > if the > > > > > > > > > > > > > >> async > > > > > > > > > > > > > >> > > > > > request > > > > > > > > > > > > > >> > > > > > > > has been completed. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: I see your point. My idea was > to > > > > > provide > > > > > > > > > specific > > > > > > > > > > > > > >> > > > Verify...Request > > > > > > > > > > > > > >> > > > > > per > > > > > > > > > > > > > >> > > > > > > > each > > > > > > > > > > > > > >> > > > > > > > long running request, where > needed. > > We > > > > can > > > > > > do it > > > > > > > > > the > > > > > > > > > > > way > > > > > > > > > > > > > you > > > > > > > > > > > > > >> > > > suggest. > > > > > > > > > > > > > >> > > > > > The > > > > > > > > > > > > > >> > > > > > > > only > > > > > > > > > > > > > >> > > > > > > > concern is that introducing a > token > > we > > > > > again > > > > > > > > will > > > > > > > > > > make > > > > > > > > > > > > > >> schema > > > > > > > > > > > > > >> > > > > > "dynamic". > > > > > > > > > > > > > >> > > > > > > We > > > > > > > > > > > > > >> > > > > > > > wanted > > > > > > > > > > > > > >> > > > > > > > to do similar thing introducing > > single > > > > > > > > > AdminRequest > > > > > > > > > > > for > > > > > > > > > > > > > all > > > > > > > > > > > > > >> topic > > > > > > > > > > > > > >> > > > > > > commands > > > > > > > > > > > > > >> > > > > > > > but rejected > > > > > > > > > > > > > >> > > > > > > > this idea because we wanted to > have > > > > schema > > > > > > > > > defined. > > > > > > > > > > So > > > > > > > > > > > > > this > > > > > > > > > > > > > >> is > > > > > > > > > > > > > >> > > > more a > > > > > > > > > > > > > >> > > > > > > > choice between: > > > > > > > > > > > > > >> > > > > > > > a) have fixed schema but introduce > > > each > > > > > > time new > > > > > > > > > > > > > >> Verify...Request > > > > > > > > > > > > > >> > > > for > > > > > > > > > > > > > >> > > > > > > > long-running requests > > > > > > > > > > > > > >> > > > > > > > b) use one request for > verification > > > but > > > > > > > > generalize > > > > > > > > > > it > > > > > > > > > > > with > > > > > > > > > > > > > >> token > > > > > > > > > > > > > >> > > > > > > > I'm fine with whatever decision > > > > community > > > > > > come > > > > > > > > to. > > > > > > > > > > > Just > > > > > > > > > > > > > let > > > > > > > > > > > > > >> me > > > > > > > > > > > > > >> > > know > > > > > > > > > > > > > >> > > > > > your > > > > > > > > > > > > > >> > > > > > > > thoughts. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > Comment from Gwen: > > > > > > > > > > > > > >> > > > > > > > 16. Specifically for ownership, I > > > think > > > > > the > > > > > > plan > > > > > > > > > is > > > > > > > > > > > to add > > > > > > > > > > > > > >> ACL > > > > > > > > > > > > > >> > > (it > > > > > > > > > > > > > >> > > > > > sounds > > > > > > > > > > > > > >> > > > > > > > like you are describing ACL) via > an > > > > > external > > > > > > > > > system > > > > > > > > > > > > > (Argus, > > > > > > > > > > > > > >> > > > Sentry). > > > > > > > > > > > > > >> > > > > > > > I remember KIP-11 described this, > > but > > > I > > > > > > can't > > > > > > > > find > > > > > > > > > > > the KIP > > > > > > > > > > > > > >> any > > > > > > > > > > > > > >> > > > > longer. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Okay, no problem. Not sure > though > > > how > > > > > we > > > > > > are > > > > > > > > > > going > > > > > > > > > > > to > > > > > > > > > > > > > >> handle > > > > > > > > > > > > > >> > > it. > > > > > > > > > > > > > >> > > > > > Wait > > > > > > > > > > > > > >> > > > > > > > which KIP > > > > > > > > > > > > > >> > > > > > > > will be committed first and > include > > > > > changes > > > > > > to > > > > > > > > > > > > > >> TopicMetadata from > > > > > > > > > > > > > >> > > > the > > > > > > > > > > > > > >> > > > > > > later > > > > > > > > > > > > > >> > > > > > > > one? > > > > > > > > > > > > > >> > > > > > > > Anyway, I added this note to "Open > > > > > > Questions" > > > > > > > > > > section > > > > > > > > > > > so > > > > > > > > > > > > > we > > > > > > > > > > > > > >> don't > > > > > > > > > > > > > >> > > > > miss > > > > > > > > > > > > > >> > > > > > > this > > > > > > > > > > > > > >> > > > > > > > piece. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > Thanks, > > > > > > > > > > > > > >> > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > On Fri, Mar 13, 2015 at 12:34 AM, > > > Andrii > > > > > > > > > Biletskyi < > > > > > > > > > > > > > >> > > > > > > > andrii.bilets...@stealth.ly> > wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > Hi all, > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > Today I uploaded the patch that > > > covers > > > > > > some of > > > > > > > > > the > > > > > > > > > > > > > >> discussed > > > > > > > > > > > > > >> > > and > > > > > > > > > > > > > >> > > > > > agreed > > > > > > > > > > > > > >> > > > > > > > > items: > > > > > > > > > > > > > >> > > > > > > > > - removed MaybeOf optional type > > > > > > > > > > > > > >> > > > > > > > > - switched to java protocol > > > > definitions > > > > > > > > > > > > > >> > > > > > > > > - simplified messages > (normalized > > > > > configs, > > > > > > > > > removed > > > > > > > > > > > topic > > > > > > > > > > > > > >> marked > > > > > > > > > > > > > >> > > > for > > > > > > > > > > > > > >> > > > > > > > > deletion) > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > I also updated the KIP-4 with > > > > respective > > > > > > > > changes > > > > > > > > > > and > > > > > > > > > > > > > >> wrote down > > > > > > > > > > > > > >> > > > my > > > > > > > > > > > > > >> > > > > > > > > proposal for > > > > > > > > > > > > > >> > > > > > > > > pending items: > > > > > > > > > > > > > >> > > > > > > > > - Batch Admin Operations -> > > updated > > > > Wire > > > > > > > > > Protocol > > > > > > > > > > > schema > > > > > > > > > > > > > >> > > proposal > > > > > > > > > > > > > >> > > > > > > > > - Remove ClusterMetadata -> > > changed > > > to > > > > > > extend > > > > > > > > > > > > > >> > > > TopicMetadataRequest > > > > > > > > > > > > > >> > > > > > > > > - Admin Client -> updated my > > initial > > > > > > proposal > > > > > > > > to > > > > > > > > > > > reflect > > > > > > > > > > > > > >> > > batching > > > > > > > > > > > > > >> > > > > > > > > - Error codes -> proposed > > > fine-grained > > > > > > error > > > > > > > > > code > > > > > > > > > > > > > instead > > > > > > > > > > > > > >> of > > > > > > > > > > > > > >> > > > > > > > > AdminRequestFailed > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > I will also send a separate > email > > to > > > > > > cover all > > > > > > > > > > > comments > > > > > > > > > > > > > >> from > > > > > > > > > > > > > >> > > this > > > > > > > > > > > > > >> > > > > > > thread. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > Thanks, > > > > > > > > > > > > > >> > > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > On Thu, Mar 12, 2015 at 9:26 PM, > > > Gwen > > > > > > Shapira > > > > > > > > < > > > > > > > > > > > > > >> > > > > gshap...@cloudera.com > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > >> Found KIP-11 ( > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface > > > > > > > > > > > > > >> > > > > > > > >> ) > > > > > > > > > > > > > >> > > > > > > > >> It actually specifies changes > to > > > the > > > > > > Metadata > > > > > > > > > > > protocol, > > > > > > > > > > > > > >> so > > > > > > > > > > > > > >> > > > making > > > > > > > > > > > > > >> > > > > > sure > > > > > > > > > > > > > >> > > > > > > > >> both KIPs are consistent in > this > > > > regard > > > > > > will > > > > > > > > be > > > > > > > > > > > good. > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > >> On Thu, Mar 12, 2015 at 12:21 > PM, > > > > Gwen > > > > > > > > Shapira > > > > > > > > > < > > > > > > > > > > > > > >> > > > > > gshap...@cloudera.com > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > > > > > > > >> > > > > > > > >> > Specifically for ownership, I > > > think > > > > > the > > > > > > > > plan > > > > > > > > > is > > > > > > > > > > > to > > > > > > > > > > > > > add > > > > > > > > > > > > > >> ACL > > > > > > > > > > > > > >> > > (it > > > > > > > > > > > > > >> > > > > > > sounds > > > > > > > > > > > > > >> > > > > > > > >> > like you are describing ACL) > > via > > > an > > > > > > > > external > > > > > > > > > > > system > > > > > > > > > > > > > >> (Argus, > > > > > > > > > > > > > >> > > > > > Sentry). > > > > > > > > > > > > > >> > > > > > > > >> > I remember KIP-11 described > > this, > > > > > but I > > > > > > > > can't > > > > > > > > > > > find > > > > > > > > > > > > > the > > > > > > > > > > > > > >> KIP > > > > > > > > > > > > > >> > > any > > > > > > > > > > > > > >> > > > > > > longer. > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > >> > Regardless, I think KIP-4 > > focuses > > > > on > > > > > > > > getting > > > > > > > > > > > > > >> information > > > > > > > > > > > > > >> > > that > > > > > > > > > > > > > >> > > > > > > already > > > > > > > > > > > > > >> > > > > > > > >> > exists from Kafka brokers, > not > > on > > > > > > adding > > > > > > > > > > > information > > > > > > > > > > > > > >> that > > > > > > > > > > > > > >> > > > > perhaps > > > > > > > > > > > > > >> > > > > > > > >> > should exist but doesn't yet? > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > >> > Gwen > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > >> > On Thu, Mar 12, 2015 at 6:37 > > AM, > > > > > > Guozhang > > > > > > > > > Wang > > > > > > > > > > < > > > > > > > > > > > > > >> > > > > > wangg...@gmail.com> > > > > > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > > > > > > > >> > > > > > > > >> >> Folks, > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > >> > > > > > > > >> >> Just want to elaborate a bit > > > more > > > > on > > > > > > the > > > > > > > > > > > > > create-topic > > > > > > > > > > > > > >> > > > metadata > > > > > > > > > > > > > >> > > > > > and > > > > > > > > > > > > > >> > > > > > > > >> batching > > > > > > > > > > > > > >> > > > > > > > >> >> describe-topic based on > > config / > > > > > > metadata > > > > > > > > in > > > > > > > > > > my > > > > > > > > > > > > > >> previous > > > > > > > > > > > > > >> > > > email > > > > > > > > > > > > > >> > > > > as > > > > > > > > > > > > > >> > > > > > > we > > > > > > > > > > > > > >> > > > > > > > >> work > > > > > > > > > > > > > >> > > > > > > > >> >> on KAFKA-1694. The main > > > motivation > > > > > is > > > > > > to > > > > > > > > > have > > > > > > > > > > > some > > > > > > > > > > > > > >> sort of > > > > > > > > > > > > > >> > > > > topic > > > > > > > > > > > > > >> > > > > > > > >> management > > > > > > > > > > > > > >> > > > > > > > >> >> mechanisms, which I think is > > > quite > > > > > > > > important > > > > > > > > > > in > > > > > > > > > > > a > > > > > > > > > > > > > >> > > > multi-tenant > > > > > > > > > > > > > >> > > > > / > > > > > > > > > > > > > >> > > > > > > > cloud > > > > > > > > > > > > > >> > > > > > > > >> >> architecture: today anyone > can > > > > > create > > > > > > > > topics > > > > > > > > > > in > > > > > > > > > > > a > > > > > > > > > > > > > >> shared > > > > > > > > > > > > > >> > > > Kafka > > > > > > > > > > > > > >> > > > > > > > >> cluster, but > > > > > > > > > > > > > >> > > > > > > > >> >> there is no concept or > > > "ownership" > > > > > of > > > > > > > > topics > > > > > > > > > > > that > > > > > > > > > > > > > are > > > > > > > > > > > > > >> > > created > > > > > > > > > > > > > >> > > > > by > > > > > > > > > > > > > >> > > > > > > > >> different > > > > > > > > > > > > > >> > > > > > > > >> >> users. For example, at > > LinkedIn > > > we > > > > > > > > basically > > > > > > > > > > > > > >> distinguish > > > > > > > > > > > > > >> > > > topic > > > > > > > > > > > > > >> > > > > > > owners > > > > > > > > > > > > > >> > > > > > > > >> via > > > > > > > > > > > > > >> > > > > > > > >> >> some casual topic name > prefix, > > > > which > > > > > > is a > > > > > > > > > bit > > > > > > > > > > > > > awkward > > > > > > > > > > > > > >> and > > > > > > > > > > > > > >> > > > does > > > > > > > > > > > > > >> > > > > > not > > > > > > > > > > > > > >> > > > > > > > fly > > > > > > > > > > > > > >> > > > > > > > >> as > > > > > > > > > > > > > >> > > > > > > > >> >> we scale our customers. It > > would > > > > be > > > > > > great > > > > > > > > to > > > > > > > > > > use > > > > > > > > > > > > > >> > > > > describe-topics > > > > > > > > > > > > > >> > > > > > > such > > > > > > > > > > > > > >> > > > > > > > >> as: > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > >> > > > > > > > >> >> Describe all topics that is > > > > created > > > > > > by me. > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > >> > > > > > > > >> >> Describe all topics whose > > > > retention > > > > > > time > > > > > > > > is > > > > > > > > > > > > > overriden > > > > > > > > > > > > > >> to X. > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > >> > > > > > > > >> >> Describe all topics whose > > > writable > > > > > > group > > > > > > > > > > include > > > > > > > > > > > > > user > > > > > > > > > > > > > >> Y > > > > > > > > > > > > > >> > > (this > > > > > > > > > > > > > >> > > > > is > > > > > > > > > > > > > >> > > > > > > > >> related to > > > > > > > > > > > > > >> > > > > > > > >> >> authorization), etc.. > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > >> > > > > > > > >> >> One possible way to achieve > > this > > > > is > > > > > to > > > > > > > > add a > > > > > > > > > > > > > metadata > > > > > > > > > > > > > >> file > > > > > > > > > > > > > >> > > in > > > > > > > > > > > > > >> > > > > the > > > > > > > > > > > > > >> > > > > > > > >> >> create-topic request, whose > > > value > > > > > will > > > > > > > > also > > > > > > > > > be > > > > > > > > > > > > > >> written ZK > > > > > > > > > > > > > >> > > as > > > > > > > > > > > > > >> > > > we > > > > > > > > > > > > > >> > > > > > > > create > > > > > > > > > > > > > >> > > > > > > > >> the > > > > > > > > > > > > > >> > > > > > > > >> >> topic; then describe-topics > > can > > > > > > choose to > > > > > > > > > > batch > > > > > > > > > > > > > topics > > > > > > > > > > > > > >> > > based > > > > > > > > > > > > > >> > > > on > > > > > > > > > > > > > >> > > > > > 1) > > > > > > > > > > > > > >> > > > > > > > name > > > > > > > > > > > > > >> > > > > > > > >> >> regex, 2) config K-V > matching, > > > 3) > > > > > > metadata > > > > > > > > > > > regex, > > > > > > > > > > > > > etc. > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > >> > > > > > > > >> >> Thoughts? > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > >> > > > > > > > >> >> Guozhang > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > >> > > > > > > > >> >> On Thu, Mar 5, 2015 at 4:37 > > PM, > > > > > > Guozhang > > > > > > > > > Wang > > > > > > > > > > < > > > > > > > > > > > > > >> > > > > > wangg...@gmail.com> > > > > > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > >> > > > > > > > >> >>> Thanks for the updated > wiki. > > A > > > > few > > > > > > > > comments > > > > > > > > > > > below: > > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > > >> > > > > > > > >> >>> 1. Error description in > > > > response: I > > > > > > think > > > > > > > > > if > > > > > > > > > > > some > > > > > > > > > > > > > >> > > errorCode > > > > > > > > > > > > > >> > > > > > could > > > > > > > > > > > > > >> > > > > > > > >> indicate > > > > > > > > > > > > > >> > > > > > > > >> >>> several different error > cases > > > > then > > > > > we > > > > > > > > > should > > > > > > > > > > > really > > > > > > > > > > > > > >> change > > > > > > > > > > > > > >> > > > it > > > > > > > > > > > > > >> > > > > to > > > > > > > > > > > > > >> > > > > > > > >> multiple > > > > > > > > > > > > > >> > > > > > > > >> >>> codes. In general the > > errorCode > > > > > > itself > > > > > > > > > would > > > > > > > > > > be > > > > > > > > > > > > > >> precise > > > > > > > > > > > > > >> > > and > > > > > > > > > > > > > >> > > > > > > > >> sufficient for > > > > > > > > > > > > > >> > > > > > > > >> >>> describing the server side > > > > errors. > > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > > >> > > > > > > > >> >>> 2. Describe topic request: > it > > > > would > > > > > > be > > > > > > > > > great > > > > > > > > > > > to go > > > > > > > > > > > > > >> beyond > > > > > > > > > > > > > >> > > > just > > > > > > > > > > > > > >> > > > > > > > >> batching on > > > > > > > > > > > > > >> > > > > > > > >> >>> topic name regex for this > > > > request. > > > > > > For > > > > > > > > > > > example, a > > > > > > > > > > > > > >> very > > > > > > > > > > > > > >> > > > common > > > > > > > > > > > > > >> > > > > > use > > > > > > > > > > > > > >> > > > > > > > >> case of > > > > > > > > > > > > > >> > > > > > > > >> >>> the topic command is to > list > > > all > > > > > > topics > > > > > > > > > whose > > > > > > > > > > > > > config > > > > > > > > > > > > > >> A's > > > > > > > > > > > > > >> > > > value > > > > > > > > > > > > > >> > > > > > is > > > > > > > > > > > > > >> > > > > > > B. > > > > > > > > > > > > > >> > > > > > > > >> With > > > > > > > > > > > > > >> > > > > > > > >> >>> topic name regex then we > have > > > to > > > > > > first > > > > > > > > > > retrieve > > > > > > > > > > > > > >> __all__ > > > > > > > > > > > > > >> > > > > topics's > > > > > > > > > > > > > >> > > > > > > > >> >>> description info and then > > > filter > > > > at > > > > > > the > > > > > > > > > > client > > > > > > > > > > > end, > > > > > > > > > > > > > >> which > > > > > > > > > > > > > >> > > > will > > > > > > > > > > > > > >> > > > > > be > > > > > > > > > > > > > >> > > > > > > a > > > > > > > > > > > > > >> > > > > > > > >> huge > > > > > > > > > > > > > >> > > > > > > > >> >>> burden on ZK. > > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > > >> > > > > > > > >> >>> 3. Config K-Vs in create > > topic: > > > > > this > > > > > > is > > > > > > > > > > > related to > > > > > > > > > > > > > >> the > > > > > > > > > > > > > >> > > > > previous > > > > > > > > > > > > > >> > > > > > > > point; > > > > > > > > > > > > > >> > > > > > > > >> >>> maybe we can add another > > > metadata > > > > > > K-V or > > > > > > > > > > just a > > > > > > > > > > > > > >> metadata > > > > > > > > > > > > > >> > > > > string > > > > > > > > > > > > > >> > > > > > > > along > > > > > > > > > > > > > >> > > > > > > > >> side > > > > > > > > > > > > > >> > > > > > > > >> >>> with config K-V in create > > topic > > > > > like > > > > > > we > > > > > > > > did > > > > > > > > > > for > > > > > > > > > > > > > >> offset > > > > > > > > > > > > > >> > > > commit > > > > > > > > > > > > > >> > > > > > > > >> request. This > > > > > > > > > > > > > >> > > > > > > > >> >>> field can be quite useful > in > > > > > storing > > > > > > > > > > > information > > > > > > > > > > > > > like > > > > > > > > > > > > > >> > > > "owner" > > > > > > > > > > > > > >> > > > > of > > > > > > > > > > > > > >> > > > > > > the > > > > > > > > > > > > > >> > > > > > > > >> topic > > > > > > > > > > > > > >> > > > > > > > >> >>> who issue the create > command, > > > > etc, > > > > > > which > > > > > > > > is > > > > > > > > > > > quite > > > > > > > > > > > > > >> > > important > > > > > > > > > > > > > >> > > > > for > > > > > > > > > > > > > >> > > > > > a > > > > > > > > > > > > > >> > > > > > > > >> >>> multi-tenant setting. Then > in > > > the > > > > > > > > describe > > > > > > > > > > > topic > > > > > > > > > > > > > >> request > > > > > > > > > > > > > >> > > we > > > > > > > > > > > > > >> > > > > can > > > > > > > > > > > > > >> > > > > > > also > > > > > > > > > > > > > >> > > > > > > > >> batch > > > > > > > > > > > > > >> > > > > > > > >> >>> on regex of the metadata > > field. > > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > > >> > > > > > > > >> >>> 4. Today all the admin > > > operations > > > > > are > > > > > > > > async > > > > > > > > > > in > > > > > > > > > > > the > > > > > > > > > > > > > >> sense > > > > > > > > > > > > > >> > > > that > > > > > > > > > > > > > >> > > > > > > > command > > > > > > > > > > > > > >> > > > > > > > >> will > > > > > > > > > > > > > >> > > > > > > > >> >>> return once it is written > in > > > ZK, > > > > > and > > > > > > that > > > > > > > > > is > > > > > > > > > > > why we > > > > > > > > > > > > > >> need > > > > > > > > > > > > > >> > > > extra > > > > > > > > > > > > > >> > > > > > > > >> verification > > > > > > > > > > > > > >> > > > > > > > >> >>> like > > > > > testUtil.waitForTopicCreated() / > > > > > > > > > verify > > > > > > > > > > > > > >> partition > > > > > > > > > > > > > >> > > > > > > reassignment > > > > > > > > > > > > > >> > > > > > > > >> >>> request, etc. With admin > > > requests > > > > > we > > > > > > > > could > > > > > > > > > > add > > > > > > > > > > > a > > > > > > > > > > > > > >> flag to > > > > > > > > > > > > > >> > > > > enable > > > > > > > > > > > > > >> > > > > > / > > > > > > > > > > > > > >> > > > > > > > >> disable > > > > > > > > > > > > > >> > > > > > > > >> >>> synchronous requests; when > it > > > is > > > > > > turned > > > > > > > > on, > > > > > > > > > > the > > > > > > > > > > > > > >> response > > > > > > > > > > > > > >> > > > will > > > > > > > > > > > > > >> > > > > > not > > > > > > > > > > > > > >> > > > > > > > >> return > > > > > > > > > > > > > >> > > > > > > > >> >>> until the request has been > > > > > > completed. And > > > > > > > > > for > > > > > > > > > > > async > > > > > > > > > > > > > >> > > requests > > > > > > > > > > > > > >> > > > > we > > > > > > > > > > > > > >> > > > > > > can > > > > > > > > > > > > > >> > > > > > > > >> add a > > > > > > > > > > > > > >> > > > > > > > >> >>> "token" field in the > > response, > > > > and > > > > > > then > > > > > > > > > only > > > > > > > > > > > need a > > > > > > > > > > > > > >> > > general > > > > > > > > > > > > > >> > > > > > "admin > > > > > > > > > > > > > >> > > > > > > > >> >>> verification request" with > > the > > > > > given > > > > > > > > token > > > > > > > > > to > > > > > > > > > > > check > > > > > > > > > > > > > >> if the > > > > > > > > > > > > > >> > > > > async > > > > > > > > > > > > > >> > > > > > > > >> request > > > > > > > > > > > > > >> > > > > > > > >> >>> has been completed. > > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > > >> > > > > > > > >> >>> 5. +1 for extending > Metadata > > > > > request > > > > > > to > > > > > > > > > > include > > > > > > > > > > > > > >> > > controller / > > > > > > > > > > > > > >> > > > > > > > >> coordinator > > > > > > > > > > > > > >> > > > > > > > >> >>> information, and then we > can > > > > remove > > > > > > the > > > > > > > > > > > > > >> ConsumerMetadata / > > > > > > > > > > > > > >> > > > > > > > >> ClusterMetadata > > > > > > > > > > > > > >> > > > > > > > >> >>> requests. > > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > > >> > > > > > > > >> >>> Guozhang > > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > > >> > > > > > > > >> >>> On Tue, Mar 3, 2015 at > 10:23 > > > AM, > > > > > Joel > > > > > > > > > Koshy < > > > > > > > > > > > > > >> > > > > > jjkosh...@gmail.com> > > > > > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > > >> > > > > > > > >> >>>> Thanks for sending that > out > > > Joe > > > > - > > > > > I > > > > > > > > don't > > > > > > > > > > > think I > > > > > > > > > > > > > >> will be > > > > > > > > > > > > > >> > > > > able > > > > > > > > > > > > > >> > > > > > to > > > > > > > > > > > > > >> > > > > > > > >> make > > > > > > > > > > > > > >> > > > > > > > >> >>>> it today, so if notes can > be > > > > sent > > > > > > out > > > > > > > > > > > afterward > > > > > > > > > > > > > that > > > > > > > > > > > > > >> > > would > > > > > > > > > > > > > >> > > > be > > > > > > > > > > > > > >> > > > > > > > great. > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > >> > > > > > > > >> >>>> On Mon, Mar 02, 2015 at > > > > 09:16:13AM > > > > > > > > -0800, > > > > > > > > > > Gwen > > > > > > > > > > > > > >> Shapira > > > > > > > > > > > > > >> > > > wrote: > > > > > > > > > > > > > >> > > > > > > > >> >>>> > Thanks for sending this > > out > > > > Joe. > > > > > > > > Looking > > > > > > > > > > > forward > > > > > > > > > > > > > >> to > > > > > > > > > > > > > >> > > > > chatting > > > > > > > > > > > > > >> > > > > > > with > > > > > > > > > > > > > >> > > > > > > > >> >>>> everyone :) > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > On Mon, Mar 2, 2015 at > > 6:46 > > > > AM, > > > > > > Joe > > > > > > > > > Stein > > > > > > > > > > < > > > > > > > > > > > > > >> > > > > > > joe.st...@stealth.ly> > > > > > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > Hey, I just sent out a > > > > google > > > > > > > > hangout > > > > > > > > > > > invite > > > > > > > > > > > > > to > > > > > > > > > > > > > >> all > > > > > > > > > > > > > >> > > > pmc, > > > > > > > > > > > > > >> > > > > > > > >> committers > > > > > > > > > > > > > >> > > > > > > > >> >>>> and > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > everyone I found > working > > > on > > > > a > > > > > > KIP. > > > > > > > > If > > > > > > > > > I > > > > > > > > > > > missed > > > > > > > > > > > > > >> anyone > > > > > > > > > > > > > >> > > > in > > > > > > > > > > > > > >> > > > > > the > > > > > > > > > > > > > >> > > > > > > > >> invite > > > > > > > > > > > > > >> > > > > > > > >> >>>> please > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > let me know and can > > update > > > > it, > > > > > > np. > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > We should do this > every > > > > > Tuesday > > > > > > @ > > > > > > > > 2pm > > > > > > > > > > > Eastern > > > > > > > > > > > > > >> Time. > > > > > > > > > > > > > >> > > > Maybe > > > > > > > > > > > > > >> > > > > > we > > > > > > > > > > > > > >> > > > > > > > can > > > > > > > > > > > > > >> > > > > > > > >> get > > > > > > > > > > > > > >> > > > > > > > >> >>>> INFRA > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > help to make a google > > > > account > > > > > > so we > > > > > > > > > can > > > > > > > > > > > manage > > > > > > > > > > > > > >> > > better? > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > To discuss > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > in progress and > related > > > JIRA > > > > > > that > > > > > > > > are > > > > > > > > > > > > > >> interdependent > > > > > > > > > > > > > >> > > > and > > > > > > > > > > > > > >> > > > > > > common > > > > > > > > > > > > > >> > > > > > > > >> work. > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > ~ Joe Stein > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > On Tue, Feb 24, 2015 > at > > > 2:59 > > > > > > PM, Jay > > > > > > > > > > > Kreps < > > > > > > > > > > > > > >> > > > > > > > jay.kr...@gmail.com> > > > > > > > > > > > > > >> > > > > > > > >> >>>> wrote: > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> Let's stay on Google > > > > hangouts > > > > > > that > > > > > > > > > will > > > > > > > > > > > also > > > > > > > > > > > > > >> record > > > > > > > > > > > > > >> > > > and > > > > > > > > > > > > > >> > > > > > make > > > > > > > > > > > > > >> > > > > > > > the > > > > > > > > > > > > > >> > > > > > > > >> >>>> sessions > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> available on youtube. > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> -Jay > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> On Tue, Feb 24, 2015 > at > > > > 11:49 > > > > > > AM, > > > > > > > > > Jeff > > > > > > > > > > > > > Holoman > > > > > > > > > > > > > >> < > > > > > > > > > > > > > >> > > > > > > > >> >>>> jholo...@cloudera.com> > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> wrote: > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > Jay / Joe > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > We're happy to send > > > out a > > > > > > Webex > > > > > > > > for > > > > > > > > > > > this > > > > > > > > > > > > > >> purpose. > > > > > > > > > > > > > >> > > We > > > > > > > > > > > > > >> > > > > > could > > > > > > > > > > > > > >> > > > > > > > >> record > > > > > > > > > > > > > >> > > > > > > > >> >>>> the > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > sessions if there > is > > > > > > interest and > > > > > > > > > > > publish > > > > > > > > > > > > > >> them > > > > > > > > > > > > > >> > > out. > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > Thanks > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > Jeff > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > On Tue, Feb 24, > 2015 > > at > > > > > > 11:28 AM, > > > > > > > > > Jay > > > > > > > > > > > > > Kreps < > > > > > > > > > > > > > >> > > > > > > > >> jay.kr...@gmail.com> > > > > > > > > > > > > > >> > > > > > > > >> >>>> wrote: > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > Let's try to get > > the > > > > > > technical > > > > > > > > > > > hang-ups > > > > > > > > > > > > > >> sorted > > > > > > > > > > > > > >> > > > out, > > > > > > > > > > > > > >> > > > > > > > though. > > > > > > > > > > > > > >> > > > > > > > >> I > > > > > > > > > > > > > >> > > > > > > > >> >>>> really > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > think > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > there is some > > benefit > > > > to > > > > > > live > > > > > > > > > > > discussion > > > > > > > > > > > > > vs > > > > > > > > > > > > > >> > > > > writing. I > > > > > > > > > > > > > >> > > > > > > am > > > > > > > > > > > > > >> > > > > > > > >> >>>> hopeful that > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> if > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > we post > > instructions > > > > and > > > > > > give > > > > > > > > > > > ourselves a > > > > > > > > > > > > > >> few > > > > > > > > > > > > > >> > > > > attempts > > > > > > > > > > > > > >> > > > > > > we > > > > > > > > > > > > > >> > > > > > > > >> can > > > > > > > > > > > > > >> > > > > > > > >> >>>> get it > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > working. > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > Tuesday at that > > time > > > > > would > > > > > > work > > > > > > > > > for > > > > > > > > > > > > > >> me...any > > > > > > > > > > > > > >> > > > > > objections? > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > -Jay > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > On Tue, Feb 24, > > 2015 > > > at > > > > > > 8:18 > > > > > > > > AM, > > > > > > > > > > Joe > > > > > > > > > > > > > Stein > > > > > > > > > > > > > >> < > > > > > > > > > > > > > >> > > > > > > > >> joe.st...@stealth.ly > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> wrote: > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > Weekly would be > > > great > > > > > > maybe > > > > > > > > > like > > > > > > > > > > > every > > > > > > > > > > > > > >> > > Tuesday ~ > > > > > > > > > > > > > >> > > > > 1pm > > > > > > > > > > > > > >> > > > > > > ET > > > > > > > > > > > > > >> > > > > > > > / > > > > > > > > > > > > > >> > > > > > > > >> 10am > > > > > > > > > > > > > >> > > > > > > > >> >>>> PT > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> ???? > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > I don't mind > > google > > > > > > hangout > > > > > > > > but > > > > > > > > > > > there > > > > > > > > > > > > > is > > > > > > > > > > > > > >> > > always > > > > > > > > > > > > > >> > > > > some > > > > > > > > > > > > > >> > > > > > > > >> issue or > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> whatever > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > so > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > we know the > > apache > > > > irc > > > > > > > > channel > > > > > > > > > > > works. > > > > > > > > > > > > > We > > > > > > > > > > > > > >> can > > > > > > > > > > > > > >> > > > start > > > > > > > > > > > > > >> > > > > > > there > > > > > > > > > > > > > >> > > > > > > > >> and > > > > > > > > > > > > > >> > > > > > > > >> >>>> see how > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> it > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > goes? We can > pull > > > > > > transcripts > > > > > > > > > too > > > > > > > > > > > and > > > > > > > > > > > > > >> > > associate > > > > > > > > > > > > > >> > > > to > > > > > > > > > > > > > >> > > > > > > > >> tickets if > > > > > > > > > > > > > >> > > > > > > > >> >>>> need be > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > makes > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > it helpful for > > > > things. > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > ~ Joestein > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > On Tue, Feb 24, > > > 2015 > > > > at > > > > > > 11:10 > > > > > > > > > AM, > > > > > > > > > > > Jay > > > > > > > > > > > > > >> Kreps < > > > > > > > > > > > > > >> > > > > > > > >> >>>> jay.kr...@gmail.com> > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > wrote: > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > We'd talked > > about > > > > > > doing a > > > > > > > > > > Google > > > > > > > > > > > > > >> Hangout to > > > > > > > > > > > > > >> > > > chat > > > > > > > > > > > > > >> > > > > > > about > > > > > > > > > > > > > >> > > > > > > > >> this. > > > > > > > > > > > > > >> > > > > > > > >> >>>> What > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > about > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > generalizing > > > that a > > > > > > little > > > > > > > > > > > > > further...I > > > > > > > > > > > > > >> > > > actually > > > > > > > > > > > > > >> > > > > > > think > > > > > > > > > > > > > >> > > > > > > > it > > > > > > > > > > > > > >> > > > > > > > >> >>>> would be > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > good > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > for > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > everyone > > > spending a > > > > > > > > > reasonable > > > > > > > > > > > chunk > > > > > > > > > > > > > of > > > > > > > > > > > > > >> > > their > > > > > > > > > > > > > >> > > > > week > > > > > > > > > > > > > >> > > > > > > on > > > > > > > > > > > > > >> > > > > > > > >> Kafka > > > > > > > > > > > > > >> > > > > > > > >> >>>> stuff > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> to > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > maybe > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > sync up once > a > > > > week. > > > > > I > > > > > > > > think > > > > > > > > > we > > > > > > > > > > > could > > > > > > > > > > > > > >> use > > > > > > > > > > > > > >> > > time > > > > > > > > > > > > > >> > > > > to > > > > > > > > > > > > > >> > > > > > > talk > > > > > > > > > > > > > >> > > > > > > > >> >>>> through > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> design > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > stuff, make > > sure > > > we > > > > > > are on > > > > > > > > > top > > > > > > > > > > of > > > > > > > > > > > > > code > > > > > > > > > > > > > >> > > > reviews, > > > > > > > > > > > > > >> > > > > > talk > > > > > > > > > > > > > >> > > > > > > > >> through > > > > > > > > > > > > > >> > > > > > > > >> >>>> any > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > tricky > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > issues, etc. > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > We can make > it > > > > > publicly > > > > > > > > > > > available so > > > > > > > > > > > > > >> that > > > > > > > > > > > > > >> > > any > > > > > > > > > > > > > >> > > > > one > > > > > > > > > > > > > >> > > > > > > can > > > > > > > > > > > > > >> > > > > > > > >> follow > > > > > > > > > > > > > >> > > > > > > > >> >>>> along > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > who > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > likes. > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > Any interest > in > > > > doing > > > > > > this? > > > > > > > > > If > > > > > > > > > > so > > > > > > > > > > > > > I'll > > > > > > > > > > > > > >> try > > > > > > > > > > > > > >> > > to > > > > > > > > > > > > > >> > > > > set > > > > > > > > > > > > > >> > > > > > it > > > > > > > > > > > > > >> > > > > > > > up > > > > > > > > > > > > > >> > > > > > > > >> >>>> starting > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> next > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > week. > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > -Jay > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > On Tue, Feb > 24, > > > > 2015 > > > > > at > > > > > > > > 3:57 > > > > > > > > > > AM, > > > > > > > > > > > > > Andrii > > > > > > > > > > > > > >> > > > > Biletskyi > > > > > > > > > > > > > >> > > > > > < > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > andrii.bilets...@stealth.ly> > > > > > > > > > > > wrote: > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > Hi all, > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > I've > updated > > > KIP > > > > > > page, > > > > > > > > > fixed > > > > > > > > > > / > > > > > > > > > > > > > >> aligned > > > > > > > > > > > > > >> > > > > document > > > > > > > > > > > > > >> > > > > > > > >> structure. > > > > > > > > > > > > > >> > > > > > > > >> >>>> Also I > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > added > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > some > > > > > > > ... > > > > > > [Message clipped] > > > > > > -- > -- Guozhang >
