Guys, I like this idea too. Let's stick with that. I'll update KIP accordingly.
I was also thinking we can avoid adding dedicated status check requests for topic commands. - We have everything in DescribeTopic for that! E.g.: User issued CreateTopic - to check the status client sends DescribeTopic and checks whether is something returned for that topic. The same for alteration, deletion. Btw, PreferredReplica status can be also checked with DescribeTopicRequest (head of assigned replicas list == leader). For ReassignPartitions as discussed we'll need to have a separate Verify... request. Thanks, Andrii Biletskyi On Thu, Mar 19, 2015 at 6:03 PM, Guozhang Wang <wangg...@gmail.com> wrote: > +1 on broker writing to ZK for async handling. I was thinking that in the > end state the admin requests would be eventually sent to controller either > through re-routing or clients discovering them, instead of letting > controller listen on ZK admin path. But thinking about it a second time, I > think it is actually simpler to let controller manage > incoming queued-up admin requests through ZK. > > Guozhang > > > On Wed, Mar 18, 2015 at 4:16 PM, Joel Koshy <jjkosh...@gmail.com> wrote: > > > +1 as well. I think it helps to keep the rerouting approach orthogonal > > to this KIP. > > > > On Wed, Mar 18, 2015 at 03:40:48PM -0700, Jay Kreps wrote: > > > I'm +1 on Jun's suggestion as long as it can work for all the requests. > > > > > > On Wed, Mar 18, 2015 at 3:35 PM, Jun Rao <j...@confluent.io> wrote: > > > > > > > Andrii, > > > > > > > > I think we agreed on the following. > > > > > > > > (a) Admin requests can be sent to and handled by any broker. > > > > (b) Admin requests are processed asynchronously, at least for now. > > That is, > > > > when the client gets a response, it just means that the request is > > > > initiated, but not necessarily completed. Then, it's up to the client > > to > > > > issue another request to check the status for completion. > > > > > > > > To support (a), we were thinking of doing request forwarding to the > > > > controller (utilizing KAFKA-1912). I am making an alternative > proposal. > > > > Basically, the broker can just write to ZooKeeper to inform the > > controller > > > > about the request. For example, to handle partitionReassignment, the > > broker > > > > will just write the requested partitions to > /admin/reassign_partitions > > > > (like what AdminUtils currently does) and then send a response to the > > > > client. This shouldn't take long and the implementation will be > simpler > > > > than forwarding the requests to the controller through RPC. > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > > > > > On Wed, Mar 18, 2015 at 3:03 PM, Andrii Biletskyi < > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > Jun, > > > > > > > > > > I might be wrong but didn't we agree we will let any broker from > the > > > > > cluster handle *long-running* admin requests (at this time > > > > preferredReplica > > > > > and > > > > > reassignPartitions), via zk admin path. Thus CreateTopics etc > should > > be > > > > > sent > > > > > only to the controller. > > > > > > > > > > Thanks, > > > > > Andrii Biletskyi > > > > > > > > > > On Wed, Mar 18, 2015 at 11:55 PM, Jun Rao <j...@confluent.io> > wrote: > > > > > > > > > > > Joel, Andril, > > > > > > > > > > > > I think we agreed that those admin requests can be issued to any > > > > broker. > > > > > > Because of that, there doesn't seem to be a strong need to know > the > > > > > > controller. So, perhaps we can proceed by not making any change > to > > the > > > > > > format of TMR right now. When we start using create topic request > > in > > > > the > > > > > > producer, we will need a new version of TMR that doesn't trigger > > auto > > > > > topic > > > > > > creation. But that can be done later. > > > > > > > > > > > > As a first cut implementation, I think the broker can just write > > to ZK > > > > > > directly for > > > > > > > createToipic/alterTopic/reassignPartitions/preferredLeaderElection > > > > > > requests, instead of forwarding them to the controller. This will > > > > > simplify > > > > > > the implementation on the broker side. > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 11:58 AM, Joel Koshy < > jjkosh...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > (Thanks Andrii for the summary) > > > > > > > > > > > > > > For (1) yes we will circle back on that shortly after syncing > up > > in > > > > > > > person. I think it is close to getting committed although > > development > > > > > > > for KAFKA-1927 can probably begin without it. > > > > > > > > > > > > > > There is one more item we covered at the hangout. i.e., whether > > we > > > > > > > want to add the coordinator to the topic metadata response or > > provide > > > > > > > a clearer ClusterMetadataRequest. > > > > > > > > > > > > > > There are two reasons I think we should try and avoid adding > the > > > > > > > field: > > > > > > > - It is irrelevant to topic metadata > > > > > > > - If we finally do request rerouting in Kafka then the field > > would > > > > add > > > > > > > little to no value. (It still helps to have a separate > > > > > > > ClusterMetadataRequest to query for cluster-wide information > > such > > > > as > > > > > > > 'which broker is the controller?' as Joe mentioned.) > > > > > > > > > > > > > > I think it would be cleaner to have an explicit > > > > ClusterMetadataRequest > > > > > > > that you can send to any broker in order to obtain the > controller > > > > (and > > > > > > > in the future possibly other cluster-wide information). I think > > the > > > > > > > main argument against doing this and instead adding it to the > > topic > > > > > > > metadata response was convenience - i.e., you don't have to > > discover > > > > > > > the controller in advance. However, I don't see much actual > > > > > > > benefit/convenience in this and in fact think it is a > non-issue. > > Let > > > > > > > me know if I'm overlooking something here. > > > > > > > > > > > > > > As an example, say we need to initiate partition reassignment > by > > > > > > > issuing the new ReassignPartitionsRequest to the controller > > (assume > > > > we > > > > > > > already have the desired manual partition assignment). If we > > are to > > > > > > > augment topic metadata response then the flow be something like > > this > > > > : > > > > > > > > > > > > > > - Issue topic metadata request to any broker (and discover the > > > > > > > controller > > > > > > > - Connect to controller if required (i.e., if the broker above > != > > > > > > > controller) > > > > > > > - Issue the partition reassignment request to the controller. > > > > > > > > > > > > > > With an explicit cluster metadata request it would be: > > > > > > > - Issue cluster metadata request to any broker > > > > > > > - Connect to controller if required (i.e., if the broker above > != > > > > > > > controller) > > > > > > > - Issue the partition reassignment request > > > > > > > > > > > > > > So it seems to add little practical value and bloats topic > > metadata > > > > > > > response with an irrelevant detail. > > > > > > > > > > > > > > The other angle to this is the following - is it a matter of > > naming? > > > > > > > Should we just rename topic metadata request/response to just > > > > > > > MetadataRequest/Response and add cluster metadata to it? By > that > > same > > > > > > > token should we also allow querying for the consumer > coordinator > > (and > > > > > > > in future transaction coordinator) as well? This leads to a > > bloated > > > > > > > request which isn't very appealing and altogether confusing. > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Joel > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 09:34:12AM -0700, Jun Rao wrote: > > > > > > > > Andri, > > > > > > > > > > > > > > > > Thanks for the summary. > > > > > > > > > > > > > > > > 1. I just realized that in order to start working on > > KAFKA-1927, we > > > > > > will > > > > > > > > need to merge the changes to OffsetCommitRequest (from 0.8.2) > > to > > > > > trunk. > > > > > > > > This is planned to be done as part of KAFKA-1634. So, we will > > need > > > > > > > Guozhang > > > > > > > > and Joel's help to wrap this up. > > > > > > > > > > > > > > > > 2. Thinking about this a bit more, if the semantic of those > > "write" > > > > > > > > requests is async (i.e., after the client gets a response, it > > just > > > > > > means > > > > > > > > that the operation is initiated, but not necessarily > > completed), we > > > > > > don't > > > > > > > > really need to forward the requests to the controller. > > Instead, the > > > > > > > > receiving broker can just write the operation to ZK as the > > admin > > > > > > command > > > > > > > > line tool previously does. This will simplify the > > implementation. > > > > > > > > > > > > > > > > 8. There is another implementation detail for describe topic. > > > > > Ideally, > > > > > > we > > > > > > > > want to read the topic config from the broker cache, instead > of > > > > > > > ZooKeeper. > > > > > > > > Currently, every broker reads the topic-level config for all > > > > topics. > > > > > > > > However, it ignores those for topics not hosted on itself. > So, > > we > > > > may > > > > > > > need > > > > > > > > to change TopicConfigManager a bit so that it caches the > > configs > > > > for > > > > > > all > > > > > > > > topics. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2015 at 1:13 PM, Andrii Biletskyi < > > > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > > > > > Guys, > > > > > > > > > > > > > > > > > > Thanks for a great discussion! > > > > > > > > > Here are the actions points: > > > > > > > > > > > > > > > > > > 1. Q: Get rid of all scala requests objects, use java > > protocol > > > > > > > definitions. > > > > > > > > > A: Gwen kindly took that (KAFKA-1927). It's important > to > > > > speed > > > > > up > > > > > > > > > review procedure > > > > > > > > > there since this ticket blocks other important > > changes. > > > > > > > > > > > > > > > > > > 2. Q: Generic re-reroute facility vs client maintaining > > cluster > > > > > > state. > > > > > > > > > A: Jay has added pseudo code to KAFKA-1912 - need to > > consider > > > > > > > whether > > > > > > > > > this will be > > > > > > > > > easy to implement as a server-side feature > (comments > > are > > > > > > > > > welcomed!). > > > > > > > > > > > > > > > > > > 3. Q: Controller field in wire protocol. > > > > > > > > > A: This might be useful for clients, add this to > > > > > > > TopicMetadataResponse > > > > > > > > > (already in KIP). > > > > > > > > > > > > > > > > > > 4. Q: Decoupling topic creation from TMR. > > > > > > > > > A: I will add proposed by Jun solution (using clientId > > for > > > > > that) > > > > > > > to the > > > > > > > > > KIP. > > > > > > > > > > > > > > > > > > 5. Q: Bumping new versions of TMR vs grabbing all protocol > > > > changes > > > > > in > > > > > > > one > > > > > > > > > version. > > > > > > > > > A: It was decided to try to gather all changes to > > protocol > > > > > > (before > > > > > > > > > release). > > > > > > > > > In case of TMR it worth checking: KAFKA-2020 and > > KIP-13 > > > > > > > (quotas) > > > > > > > > > > > > > > > > > > 6. Q: JSON lib is needed to deserialize user's input in CLI > > tool. > > > > > > > > > A: Use jackson for that, /tools project is a separate > > jar so > > > > > > > shouldn't > > > > > > > > > be a big deal. > > > > > > > > > > > > > > > > > > 7. Q: VerifyReassingPartitions vs generic status check > > command. > > > > > > > > > A: For long-running requests like reassign partitions > > > > > *progress* > > > > > > > check > > > > > > > > > request is useful, > > > > > > > > > it makes sense to introduce it. > > > > > > > > > > > > > > > > > > Please add, correct me if I missed something. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2015 at 6:20 PM, Andrii Biletskyi < > > > > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > > > > > > > Joel, > > > > > > > > > > > > > > > > > > > > You are right, I removed ClusterMetadata because we have > > > > > partially > > > > > > > > > > what we need in TopicMetadata. Also, as Jay pointed out > > > > earlier, > > > > > we > > > > > > > > > > would like to have "orthogonal" API, but at the same time > > we > > > > need > > > > > > > > > > to be backward compatible. > > > > > > > > > > > > > > > > > > > > But I like your idea and even have some other arguments > for > > > > this > > > > > > > option: > > > > > > > > > > There is also DescribeTopicRequest which was proposed in > > this > > > > > KIP, > > > > > > > > > > it returns topic configs, partitions, replication factor > > plus > > > > > > > partition > > > > > > > > > > ISR, ASR, > > > > > > > > > > leader replica. The later part is really already there in > > > > > > > > > > TopicMetadataRequest. > > > > > > > > > > So again we'll have to add stuff to TMR, not to duplicate > > some > > > > > info > > > > > > > in > > > > > > > > > > newly added requests. However, this way we'll end up with > > > > > "monster" > > > > > > > > > > request which returns cluster metadata, topic replication > > and > > > > > > config > > > > > > > info > > > > > > > > > > plus partition replication data. Seems logical to split > > TMR to > > > > > > > > > > - ClusterMetadata (brokers + controller, maybe smth else) > > > > > > > > > > - TopicMetadata (topic info + partition details) > > > > > > > > > > But since current TMR is involved in lots of places > > (including > > > > > > > network > > > > > > > > > > client, > > > > > > > > > > as I understand) this might be very serious change and it > > > > > probably > > > > > > > makes > > > > > > > > > > sense to stick with current approach. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2015 at 5:29 PM, Joel Koshy < > > > > jjkosh...@gmail.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > >> I may be missing some context but hopefully this will > > also be > > > > > > > covered > > > > > > > > > >> today: I thought the earlier proposal where there was an > > > > > explicit > > > > > > > > > >> ClusterMetadata request was clearer and explicit. During > > the > > > > > > course > > > > > > > of > > > > > > > > > >> this thread I think the conclusion was that the main > need > > was > > > > > for > > > > > > > > > >> controller information and that can be rolled into the > > topic > > > > > > > metadata > > > > > > > > > >> response but that seems a bit irrelevant to topic > > metadata. > > > > > FWIW I > > > > > > > > > >> think the full broker-list is also irrelevant to topic > > > > metadata, > > > > > > but > > > > > > > > > >> it is already there and in use. I think there is still > > room > > > > for > > > > > an > > > > > > > > > >> explicit ClusterMetadata request since there may be > other > > > > > > > > > >> cluster-level information that we may want to add over > > time > > > > (and > > > > > > > that > > > > > > > > > >> have nothing to do with topic metadata). > > > > > > > > > >> > > > > > > > > > >> On Tue, Mar 17, 2015 at 02:45:30PM +0200, Andrii > Biletskyi > > > > > wrote: > > > > > > > > > >> > Jun, > > > > > > > > > >> > > > > > > > > > > >> > 101. Okay, if you say that such use case is > important. I > > > > also > > > > > > > think > > > > > > > > > >> > using clientId for these purposes is fine - if we > > already > > > > have > > > > > > > this > > > > > > > > > >> field > > > > > > > > > >> > as part of all Wire protocol messages, why not use > that. > > > > > > > > > >> > I will update KIP-4 page if nobody has other ideas > > (which > > > > may > > > > > > > come up > > > > > > > > > >> > during the call today). > > > > > > > > > >> > > > > > > > > > > >> > 102.1 Agree, I'll update the KIP accordingly. I think > > we can > > > > > add > > > > > > > new, > > > > > > > > > >> > fine-grained error codes if some error code received > in > > > > > specific > > > > > > > case > > > > > > > > > >> > won't give enough context to return a descriptive > error > > > > > message > > > > > > > for > > > > > > > > > >> user. > > > > > > > > > >> > > > > > > > > > > >> > Look forward to discussing all outstanding issues in > > detail > > > > > > today > > > > > > > > > during > > > > > > > > > >> > the call. > > > > > > > > > >> > > > > > > > > > > >> > Thanks, > > > > > > > > > >> > Andrii Biletskyi > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > On Mon, Mar 16, 2015 at 10:59 PM, Jun Rao < > > j...@confluent.io > > > > > > > > > > > > wrote: > > > > > > > > > >> > > > > > > > > > > >> > > 101. There may be a use case where you only want the > > > > topics > > > > > to > > > > > > > be > > > > > > > > > >> created > > > > > > > > > >> > > manually by admins. Currently, you can do that by > > > > disabling > > > > > > auto > > > > > > > > > topic > > > > > > > > > >> > > creation and issue topic creation from the > > TopicCommand. > > > > If > > > > > we > > > > > > > > > >> disable auto > > > > > > > > > >> > > topic creation completely on the broker and don't > > have a > > > > way > > > > > > to > > > > > > > > > >> distinguish > > > > > > > > > >> > > between topic creation requests from the regular > > clients > > > > and > > > > > > the > > > > > > > > > >> admin, we > > > > > > > > > >> > > can't support manual topic creation any more. I was > > > > thinking > > > > > > > that > > > > > > > > > >> another > > > > > > > > > >> > > way of distinguishing the clients making the topic > > > > creation > > > > > > > requests > > > > > > > > > >> is > > > > > > > > > >> > > using clientId. For example, the admin tool can set > > it to > > > > > > > something > > > > > > > > > >> like > > > > > > > > > >> > > admin and the broker can treat that clientId > > specially. > > > > > > > > > >> > > > > > > > > > > > >> > > Also, there is a related discussion in KAFKA-2020. > > > > > Currently, > > > > > > > we do > > > > > > > > > >> the > > > > > > > > > >> > > following in TopicMetadataResponse: > > > > > > > > > >> > > > > > > > > > > > >> > > 1. If leader is not available, we set the partition > > level > > > > > > error > > > > > > > code > > > > > > > > > >> to > > > > > > > > > >> > > LeaderNotAvailable. > > > > > > > > > >> > > 2. If a non-leader replica is not available, we take > > that > > > > > > > replica > > > > > > > > > out > > > > > > > > > >> of > > > > > > > > > >> > > the assigned replica list and isr in the response. > As > > an > > > > > > > indication > > > > > > > > > >> for > > > > > > > > > >> > > doing that, we set the partition level error code to > > > > > > > > > >> ReplicaNotAvailable. > > > > > > > > > >> > > > > > > > > > > > >> > > This has a few problems. First, ReplicaNotAvailable > > > > probably > > > > > > > > > >> shouldn't be > > > > > > > > > >> > > an error, at least for the normal producer/consumer > > > > clients > > > > > > that > > > > > > > > > just > > > > > > > > > >> want > > > > > > > > > >> > > to find out the leader. Second, it can happen that > > both > > > > the > > > > > > > leader > > > > > > > > > and > > > > > > > > > >> > > another replica are not available at the same time. > > There > > > > is > > > > > > no > > > > > > > > > error > > > > > > > > > >> code > > > > > > > > > >> > > to indicate both. Third, even if a replica is not > > > > available, > > > > > > > it's > > > > > > > > > >> still > > > > > > > > > >> > > useful to return its replica id since some clients > > (e.g. > > > > > admin > > > > > > > tool) > > > > > > > > > >> may > > > > > > > > > >> > > still make use of it. > > > > > > > > > >> > > > > > > > > > > > >> > > One way to address this issue is to always return > the > > > > > replica > > > > > > > id for > > > > > > > > > >> > > leader, assigned replicas, and isr regardless of > > whether > > > > the > > > > > > > > > >> corresponding > > > > > > > > > >> > > broker is live or not. Since we also return the list > > of > > > > live > > > > > > > > > brokers, > > > > > > > > > >> the > > > > > > > > > >> > > client can figure out whether a leader or a replica > is > > > > live > > > > > or > > > > > > > not > > > > > > > > > >> and act > > > > > > > > > >> > > accordingly. This way, we don't need to set the > > partition > > > > > > level > > > > > > > > > error > > > > > > > > > >> code > > > > > > > > > >> > > when the leader or a replica is not available. This > > > > doesn't > > > > > > > change > > > > > > > > > >> the wire > > > > > > > > > >> > > protocol, but does change the semantics. Since we > are > > > > > evolving > > > > > > > the > > > > > > > > > >> protocol > > > > > > > > > >> > > of TopicMetadataRequest here, we can potentially > > piggyback > > > > > the > > > > > > > > > change. > > > > > > > > > >> > > > > > > > > > > > >> > > 102.1 For those types of errors due to invalid > input, > > > > > > shouldn't > > > > > > > we > > > > > > > > > >> just > > > > > > > > > >> > > guard it at parameter validation time and throw > > > > > > > > > >> InvalidArgumentException > > > > > > > > > >> > > without even sending the request to the broker? > > > > > > > > > >> > > > > > > > > > > > >> > > Thanks, > > > > > > > > > >> > > > > > > > > > > > >> > > Jun > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > On Mon, Mar 16, 2015 at 10:37 AM, Andrii Biletskyi < > > > > > > > > > >> > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > >> > > > > > > > > > > > >> > > > Jun, > > > > > > > > > >> > > > > > > > > > > > > >> > > > Answering your questions: > > > > > > > > > >> > > > > > > > > > > > > >> > > > 101. If I understand you correctly, you are saying > > > > future > > > > > > > producer > > > > > > > > > >> > > versions > > > > > > > > > >> > > > (which > > > > > > > > > >> > > > will be ported to TMR_V1) won't be able to > > automatically > > > > > > > create > > > > > > > > > >> topic (if > > > > > > > > > >> > > > we > > > > > > > > > >> > > > unconditionally remove topic creation from there). > > But > > > > we > > > > > > > need to > > > > > > > > > >> this > > > > > > > > > >> > > > preserve logic. > > > > > > > > > >> > > > Ok, about your proposal: I'm not a big fan too, > > when it > > > > > > comes > > > > > > > to > > > > > > > > > >> > > > differentiating > > > > > > > > > >> > > > clients directly in protocol schema. And also I'm > > not > > > > > sure I > > > > > > > > > >> understand > > > > > > > > > >> > > at > > > > > > > > > >> > > > all why > > > > > > > > > >> > > > auto.create.topics.enable is a server side > > > > configuration. > > > > > > Can > > > > > > > we > > > > > > > > > >> > > deprecate > > > > > > > > > >> > > > this setting > > > > > > > > > >> > > > in future versions, add this setting to producer > and > > > > based > > > > > > on > > > > > > > that > > > > > > > > > >> upon > > > > > > > > > >> > > > receiving > > > > > > > > > >> > > > UnknownTopic create topic explicitly by a separate > > > > > producer > > > > > > > call > > > > > > > > > via > > > > > > > > > >> > > > adminClient? > > > > > > > > > >> > > > > > > > > > > > > >> > > > 102.1. Hm, yes. It's because we want to support > > batching > > > > > and > > > > > > > at > > > > > > > > > the > > > > > > > > > >> same > > > > > > > > > >> > > > time we > > > > > > > > > >> > > > want to give descriptive error messages for > clients. > > > > Since > > > > > > > > > >> AdminClient > > > > > > > > > >> > > > holds the context > > > > > > > > > >> > > > to construct such messages (e.g. AdminClient layer > > can > > > > > know > > > > > > > that > > > > > > > > > >> > > > InvalidArgumentsCode > > > > > > > > > >> > > > means two cases: either invalid number - e.g. -1; > or > > > > > > > > > >> replication-factor > > > > > > > > > >> > > was > > > > > > > > > >> > > > provided while > > > > > > > > > >> > > > partitions argument wasn't) - I wrapped responses > in > > > > > > > Exceptions. > > > > > > > > > >> But I'm > > > > > > > > > >> > > > open to any > > > > > > > > > >> > > > other ideas, this was just initial version. > > > > > > > > > >> > > > 102.2. Yes, I agree. I'll change that to probably > > some > > > > > other > > > > > > > dto. > > > > > > > > > >> > > > > > > > > > > > > >> > > > Thanks, > > > > > > > > > >> > > > Andrii Biletskyi > > > > > > > > > >> > > > > > > > > > > > > >> > > > On Fri, Mar 13, 2015 at 7:16 PM, Jun Rao < > > > > > j...@confluent.io> > > > > > > > > > wrote: > > > > > > > > > >> > > > > > > > > > > > > >> > > > > Andrii, > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > 101. That's what I was thinking too, but it may > > not be > > > > > > that > > > > > > > > > >> simple. In > > > > > > > > > >> > > > > TopicMetadataRequest_V1, > > > > > > > > > >> > > > > we can let it not trigger auto topic creation. > > Then, > > > > in > > > > > > the > > > > > > > > > >> producer > > > > > > > > > >> > > > side, > > > > > > > > > >> > > > > if it gets an UnknownTopicException, it can > > explicitly > > > > > > > issue a > > > > > > > > > >> > > > > createTopicRequest for auto topic creation. On > the > > > > > > consumer > > > > > > > > > side, > > > > > > > > > >> it > > > > > > > > > >> > > will > > > > > > > > > >> > > > > never issue createTopicRequest. This works when > > auto > > > > > topic > > > > > > > > > >> creation is > > > > > > > > > >> > > > > enabled on the broker side. However, I am not > > sure how > > > > > > > things > > > > > > > > > >> will work > > > > > > > > > >> > > > > when auto topic creation is disabled on the > broker > > > > side. > > > > > > In > > > > > > > this > > > > > > > > > >> case, > > > > > > > > > >> > > we > > > > > > > > > >> > > > > want to have a way to manually create a topic, > > > > > potentially > > > > > > > > > through > > > > > > > > > >> > > admin > > > > > > > > > >> > > > > commands. However, then we need a way to > > distinguish > > > > > > > > > >> createTopicRequest > > > > > > > > > >> > > > > issued from the producer clients and the admin > > tools. > > > > > May > > > > > > > be we > > > > > > > > > >> can > > > > > > > > > >> > > add a > > > > > > > > > >> > > > > new field in createTopicRequest and set it > > differently > > > > > in > > > > > > > the > > > > > > > > > >> producer > > > > > > > > > >> > > > > client and the admin client. However, I am not > > sure if > > > > > > > that's > > > > > > > > > the > > > > > > > > > >> best > > > > > > > > > >> > > > > approach. > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > 2. Yes, refactoring existing requests is a > > non-trivial > > > > > > > amount of > > > > > > > > > >> work. > > > > > > > > > >> > > I > > > > > > > > > >> > > > > posted some comments in KAFKA-1927. We will > > probably > > > > > have > > > > > > > to fix > > > > > > > > > >> > > > KAFKA-1927 > > > > > > > > > >> > > > > first, before adding the new logic in > KAFKA-1694. > > > > > > > Otherwise, the > > > > > > > > > >> > > changes > > > > > > > > > >> > > > > will be too big. > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > 102. About the AdminClient: > > > > > > > > > >> > > > > 102.1. It's a bit weird that we return exception > > in > > > > the > > > > > > > api. It > > > > > > > > > >> seems > > > > > > > > > >> > > > that > > > > > > > > > >> > > > > we should either return error code or throw an > > > > exception > > > > > > > when > > > > > > > > > >> getting > > > > > > > > > >> > > the > > > > > > > > > >> > > > > response state. > > > > > > > > > >> > > > > 102.2. We probably shouldn't explicitly use the > > > > request > > > > > > > object > > > > > > > > > in > > > > > > > > > >> the > > > > > > > > > >> > > > api. > > > > > > > > > >> > > > > Not every request evolution requires an api > > change. > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > Thanks, > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > Jun > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > On Fri, Mar 13, 2015 at 4:08 AM, Andrii > Biletskyi > > < > > > > > > > > > >> > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > Jun, > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > Thanks for you comments. Answers inline: > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > 100. There are a few fields such as > > > > ReplicaAssignment, > > > > > > > > > >> > > > > > > ReassignPartitionRequest, > > > > > > > > > >> > > > > > > and PartitionsSerialized that are > represented > > as a > > > > > > > string, > > > > > > > > > but > > > > > > > > > >> > > > contain > > > > > > > > > >> > > > > > > composite structures in json. Could we > flatten > > > > them > > > > > > out > > > > > > > > > >> directly in > > > > > > > > > >> > > > the > > > > > > > > > >> > > > > > > protocol definition as arrays/records? > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > Yes, now with Admin Client this looks a bit > > weird. > > > > My > > > > > > > initial > > > > > > > > > >> > > > motivation > > > > > > > > > >> > > > > > was: > > > > > > > > > >> > > > > > ReassignPartitionCommand accepts input in > json, > > we > > > > > want > > > > > > to > > > > > > > > > >> remain > > > > > > > > > >> > > > tools' > > > > > > > > > >> > > > > > interfaces unchanged, where possible. > > > > > > > > > >> > > > > > If we port it to deserialized format, in CLI > > (/tools > > > > > > > project) > > > > > > > > > >> we will > > > > > > > > > >> > > > > have > > > > > > > > > >> > > > > > to add some > > > > > > > > > >> > > > > > json library since /tools is written in java > and > > > > we'll > > > > > > > need to > > > > > > > > > >> > > > > deserialize > > > > > > > > > >> > > > > > json file > > > > > > > > > >> > > > > > provided by a user. Can we quickly agree on > what > > > > this > > > > > > > library > > > > > > > > > >> should > > > > > > > > > >> > > be > > > > > > > > > >> > > > > > (Jackson, GSON, whatever)? > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > 101. Does TopicMetadataRequest v1 still > trigger > > auto > > > > > > topic > > > > > > > > > >> creation? > > > > > > > > > >> > > > This > > > > > > > > > >> > > > > > > will be a bit weird now that we have a > > separate > > > > > topic > > > > > > > > > >> creation api. > > > > > > > > > >> > > > > Have > > > > > > > > > >> > > > > > > you thought about how the new > > createTopicRequest > > > > and > > > > > > > > > >> > > > > TopicMetadataRequest > > > > > > > > > >> > > > > > > v1 will be used in the producer/consumer > > client, > > > > in > > > > > > > addition > > > > > > > > > >> to > > > > > > > > > >> > > admin > > > > > > > > > >> > > > > > > tools? For example, ideally, we don't want > > > > > > > > > >> TopicMetadataRequest > > > > > > > > > >> > > from > > > > > > > > > >> > > > > the > > > > > > > > > >> > > > > > > consumer to trigger auto topic creation. > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > I agree, this strange logic should be fixed. > > I'm not > > > > > > > confident > > > > > > > > > >> in > > > > > > > > > >> > > this > > > > > > > > > >> > > > > > Kafka part so > > > > > > > > > >> > > > > > correct me if I'm wrong, but it doesn't look > > like a > > > > > hard > > > > > > > thing > > > > > > > > > >> to > > > > > > > > > >> > > do, I > > > > > > > > > >> > > > > > think we can > > > > > > > > > >> > > > > > leverage AdminClient for that in Producer and > > > > > > > unconditionally > > > > > > > > > >> remove > > > > > > > > > >> > > > > topic > > > > > > > > > >> > > > > > creation from the TopicMetadataRequest_V1. > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > 2. I think Jay meant getting rid of scala > > classes > > > > > > > > > >> > > > > > > like HeartbeatRequestAndHeader and > > > > > > > > > >> HeartbeatResponseAndHeader. We > > > > > > > > > >> > > did > > > > > > > > > >> > > > > > that > > > > > > > > > >> > > > > > > as a stop-gap thing when adding the new > > requests > > > > for > > > > > > the > > > > > > > > > >> consumers. > > > > > > > > > >> > > > > > > However, the long term plan is to get rid of > > all > > > > > those > > > > > > > and > > > > > > > > > >> just > > > > > > > > > >> > > reuse > > > > > > > > > >> > > > > the > > > > > > > > > >> > > > > > > java request/response in the client. Since > > this > > > > KIP > > > > > > > proposes > > > > > > > > > >> to > > > > > > > > > >> > > add a > > > > > > > > > >> > > > > > > significant number of new requests, perhaps > we > > > > > should > > > > > > > bite > > > > > > > > > the > > > > > > > > > >> > > bullet > > > > > > > > > >> > > > > to > > > > > > > > > >> > > > > > > clean up the existing scala requests first > > before > > > > > > > adding new > > > > > > > > > >> ones? > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > Yes, looks like I misunderstood the point of > > > > > > > > > >> ...RequestAndHeader. > > > > > > > > > >> > > > Okay, I > > > > > > > > > >> > > > > > will > > > > > > > > > >> > > > > > rework that. The only thing is that I don't > see > > any > > > > > > > example > > > > > > > > > how > > > > > > > > > >> it > > > > > > > > > >> > > was > > > > > > > > > >> > > > > done > > > > > > > > > >> > > > > > for at > > > > > > > > > >> > > > > > least one existing protocol message. Thus, as > I > > > > > > > understand, I > > > > > > > > > >> have to > > > > > > > > > >> > > > > think > > > > > > > > > >> > > > > > how we > > > > > > > > > >> > > > > > are going to do it. > > > > > > > > > >> > > > > > Re porting all existing RQ/RP in this patch. > > Sounds > > > > > > > > > reasonable, > > > > > > > > > >> but > > > > > > > > > >> > > if > > > > > > > > > >> > > > > it's > > > > > > > > > >> > > > > > an *obligatory* > > > > > > > > > >> > > > > > requirement to have Admin KIP done, I'm afraid > > this > > > > > can > > > > > > > be a > > > > > > > > > >> serious > > > > > > > > > >> > > > > > blocker for us. > > > > > > > > > >> > > > > > There are 13 protocol messages and all that > > would > > > > > > require > > > > > > > not > > > > > > > > > >> only > > > > > > > > > >> > > unit > > > > > > > > > >> > > > > > tests but quite > > > > > > > > > >> > > > > > intensive manual testing, no? I'm afraid I'm > > not the > > > > > > > right guy > > > > > > > > > >> to > > > > > > > > > >> > > cover > > > > > > > > > >> > > > > > pretty much all > > > > > > > > > >> > > > > > Kafka core internals :). Let me know your > > thoughts > > > > on > > > > > > this > > > > > > > > > >> item. Btw > > > > > > > > > >> > > > > there > > > > > > > > > >> > > > > > is a ticket to > > > > > > > > > >> > > > > > follow-up this issue ( > > > > > > > > > >> > > https://issues.apache.org/jira/browse/KAFKA-2006 > > > > > > > > > >> > > > ). > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > Thanks, > > > > > > > > > >> > > > > > Andrii Biletskyi > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > On Fri, Mar 13, 2015 at 6:40 AM, Jun Rao < > > > > > > > j...@confluent.io> > > > > > > > > > >> wrote: > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > Andrii, > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > A few more comments. > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > 100. There are a few fields such as > > > > > ReplicaAssignment, > > > > > > > > > >> > > > > > > ReassignPartitionRequest, > > > > > > > > > >> > > > > > > and PartitionsSerialized that are > represented > > as a > > > > > > > string, > > > > > > > > > but > > > > > > > > > >> > > > contain > > > > > > > > > >> > > > > > > composite structures in json. Could we > flatten > > > > them > > > > > > out > > > > > > > > > >> directly in > > > > > > > > > >> > > > the > > > > > > > > > >> > > > > > > protocol definition as arrays/records? > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > 101. Does TopicMetadataRequest v1 still > > trigger > > > > auto > > > > > > > topic > > > > > > > > > >> > > creation? > > > > > > > > > >> > > > > This > > > > > > > > > >> > > > > > > will be a bit weird now that we have a > > separate > > > > > topic > > > > > > > > > >> creation api. > > > > > > > > > >> > > > > Have > > > > > > > > > >> > > > > > > you thought about how the new > > createTopicRequest > > > > and > > > > > > > > > >> > > > > TopicMetadataRequest > > > > > > > > > >> > > > > > > v1 will be used in the producer/consumer > > client, > > > > in > > > > > > > addition > > > > > > > > > >> to > > > > > > > > > >> > > admin > > > > > > > > > >> > > > > > > tools? For example, ideally, we don't want > > > > > > > > > >> TopicMetadataRequest > > > > > > > > > >> > > from > > > > > > > > > >> > > > > the > > > > > > > > > >> > > > > > > consumer to trigger auto topic creation. > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > 2. I think Jay meant getting rid of scala > > classes > > > > > > > > > >> > > > > > > like HeartbeatRequestAndHeader and > > > > > > > > > >> HeartbeatResponseAndHeader. We > > > > > > > > > >> > > did > > > > > > > > > >> > > > > > that > > > > > > > > > >> > > > > > > as a stop-gap thing when adding the new > > requests > > > > for > > > > > > the > > > > > > > > > >> consumers. > > > > > > > > > >> > > > > > > However, the long term plan is to get rid of > > all > > > > > those > > > > > > > and > > > > > > > > > >> just > > > > > > > > > >> > > reuse > > > > > > > > > >> > > > > the > > > > > > > > > >> > > > > > > java request/response in the client. Since > > this > > > > KIP > > > > > > > proposes > > > > > > > > > >> to > > > > > > > > > >> > > add a > > > > > > > > > >> > > > > > > significant number of new requests, perhaps > we > > > > > should > > > > > > > bite > > > > > > > > > the > > > > > > > > > >> > > bullet > > > > > > > > > >> > > > > to > > > > > > > > > >> > > > > > > clean up the existing scala requests first > > before > > > > > > > adding new > > > > > > > > > >> ones? > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > Thanks, > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > Jun > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > On Thu, Mar 12, 2015 at 3:37 PM, Andrii > > Biletskyi > > > > < > > > > > > > > > >> > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > Hi, > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > As said above - I list again all comments > > from > > > > > this > > > > > > > thread > > > > > > > > > >> so we > > > > > > > > > >> > > > > > > > can see what's left and finalize all > pending > > > > > issues. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > Comments from Jay: > > > > > > > > > >> > > > > > > > 1. This is much needed functionality, but > > there > > > > > are > > > > > > a > > > > > > > lot > > > > > > > > > >> of the > > > > > > > > > >> > > so > > > > > > > > > >> > > > > > let's > > > > > > > > > >> > > > > > > > really think these protocols through. We > > really > > > > > want > > > > > > > to > > > > > > > > > end > > > > > > > > > >> up > > > > > > > > > >> > > > with a > > > > > > > > > >> > > > > > set > > > > > > > > > >> > > > > > > > of well thought-out, orthoganol apis. For > > this > > > > > > reason > > > > > > > I > > > > > > > > > >> think it > > > > > > > > > >> > > is > > > > > > > > > >> > > > > > > really > > > > > > > > > >> > > > > > > > important to think through the end state > > even if > > > > > > that > > > > > > > > > >> includes > > > > > > > > > >> > > APIs > > > > > > > > > >> > > > > we > > > > > > > > > >> > > > > > > > won't implement in the first phase. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > A: Definitely behind this. Would > appreciate > > if > > > > > there > > > > > > > are > > > > > > > > > >> concrete > > > > > > > > > >> > > > > > > comments > > > > > > > > > >> > > > > > > > how this can be improved. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > 2. Let's please please please wait until > we > > have > > > > > > > switched > > > > > > > > > >> the > > > > > > > > > >> > > > server > > > > > > > > > >> > > > > > over > > > > > > > > > >> > > > > > > > to the new java protocol definitions. If > we > > add > > > > > > upteen > > > > > > > > > more > > > > > > > > > >> ad > > > > > > > > > >> > > hoc > > > > > > > > > >> > > > > > scala > > > > > > > > > >> > > > > > > > objects that is just generating more work > > for > > > > the > > > > > > > > > >> conversion we > > > > > > > > > >> > > > know > > > > > > > > > >> > > > > we > > > > > > > > > >> > > > > > > > have to do. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - removed > scala > > > > > > protocol > > > > > > > > > >> classes. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > 3. This proposal introduces a new type of > > > > optional > > > > > > > > > >> parameter. > > > > > > > > > >> > > This > > > > > > > > > >> > > > is > > > > > > > > > >> > > > > > > > inconsistent with everything else in the > > > > protocol > > > > > > > where we > > > > > > > > > >> use -1 > > > > > > > > > >> > > > or > > > > > > > > > >> > > > > > some > > > > > > > > > >> > > > > > > > other marker value. You could argue either > > way > > > > but > > > > > > > let's > > > > > > > > > >> stick > > > > > > > > > >> > > with > > > > > > > > > >> > > > > > that > > > > > > > > > >> > > > > > > > for consistency. For clients that > > implemented > > > > the > > > > > > > protocol > > > > > > > > > >> in a > > > > > > > > > >> > > > > better > > > > > > > > > >> > > > > > > way > > > > > > > > > >> > > > > > > > than our scala code these basic primitives > > are > > > > > hard > > > > > > to > > > > > > > > > >> change. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - removed > > MaybeOf > > > > > type > > > > > > > and > > > > > > > > > >> changed > > > > > > > > > >> > > > > > protocol > > > > > > > > > >> > > > > > > > accordingly. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > 4. ClusterMetadata: This seems to > duplicate > > > > > > > > > >> TopicMetadataRequest > > > > > > > > > >> > > > > which > > > > > > > > > >> > > > > > > has > > > > > > > > > >> > > > > > > > brokers, topics, and partitions. I think > we > > > > should > > > > > > > rename > > > > > > > > > >> that > > > > > > > > > >> > > > > request > > > > > > > > > >> > > > > > > > ClusterMetadataRequest (or just > > MetadataRequest) > > > > > and > > > > > > > > > >> include the > > > > > > > > > >> > > id > > > > > > > > > >> > > > > of > > > > > > > > > >> > > > > > > the > > > > > > > > > >> > > > > > > > controller. Or are there other things we > > could > > > > add > > > > > > > here? > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > A: I agree. Updated the KIP. Let's extends > > > > > > > TopicMetadata > > > > > > > > > to > > > > > > > > > >> > > > version 2 > > > > > > > > > >> > > > > > and > > > > > > > > > >> > > > > > > > include controller. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > 5. We have a tendency to try to make a lot > > of > > > > > > requests > > > > > > > > > that > > > > > > > > > >> can > > > > > > > > > >> > > > only > > > > > > > > > >> > > > > go > > > > > > > > > >> > > > > > > to > > > > > > > > > >> > > > > > > > particular nodes. This adds a lot of > burden > > for > > > > > > client > > > > > > > > > >> > > > > implementations > > > > > > > > > >> > > > > > > (it > > > > > > > > > >> > > > > > > > sounds easy but each discovery can fail in > > many > > > > > > parts > > > > > > > so > > > > > > > > > it > > > > > > > > > >> ends > > > > > > > > > >> > > up > > > > > > > > > >> > > > > > > being a > > > > > > > > > >> > > > > > > > full state machine to do right). I think > we > > > > should > > > > > > > > > consider > > > > > > > > > >> > > making > > > > > > > > > >> > > > > > admin > > > > > > > > > >> > > > > > > > commands and ideally as many of the other > > apis > > > > as > > > > > > > possible > > > > > > > > > >> > > > available > > > > > > > > > >> > > > > on > > > > > > > > > >> > > > > > > all > > > > > > > > > >> > > > > > > > brokers and just redirect to the > controller > > on > > > > the > > > > > > > broker > > > > > > > > > >> side. > > > > > > > > > >> > > > > Perhaps > > > > > > > > > >> > > > > > > > there would be a general way to > encapsulate > > this > > > > > > > > > re-routing > > > > > > > > > >> > > > behavior. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > A: It's a very interesting idea, but seems > > there > > > > > are > > > > > > > some > > > > > > > > > >> > > concerns > > > > > > > > > >> > > > > > about > > > > > > > > > >> > > > > > > > this > > > > > > > > > >> > > > > > > > feature (like performance considerations, > > how > > > > this > > > > > > > will > > > > > > > > > >> > > complicate > > > > > > > > > >> > > > > > server > > > > > > > > > >> > > > > > > > etc). > > > > > > > > > >> > > > > > > > I believe this shouldn't be a blocker. If > > this > > > > > > > feature is > > > > > > > > > >> > > > implemented > > > > > > > > > >> > > > > > at > > > > > > > > > >> > > > > > > > some > > > > > > > > > >> > > > > > > > point it won't affect Admin changes - at > > least > > > > no > > > > > > > changes > > > > > > > > > to > > > > > > > > > >> > > public > > > > > > > > > >> > > > > API > > > > > > > > > >> > > > > > > > will be required. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > 6. We should probably normalize the key > > value > > > > > pairs > > > > > > > used > > > > > > > > > for > > > > > > > > > >> > > > configs > > > > > > > > > >> > > > > > > rather > > > > > > > > > >> > > > > > > > than embedding a new formatting. So two > > strings > > > > > > rather > > > > > > > > > than > > > > > > > > > >> one > > > > > > > > > >> > > > with > > > > > > > > > >> > > > > an > > > > > > > > > >> > > > > > > > internal equals sign. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - normalized > > > > configs > > > > > > and > > > > > > > > > >> changed > > > > > > > > > >> > > > > protocol > > > > > > > > > >> > > > > > > > accordingly. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > 7. Is the postcondition of these APIs that > > the > > > > > > > command has > > > > > > > > > >> begun > > > > > > > > > >> > > or > > > > > > > > > >> > > > > > that > > > > > > > > > >> > > > > > > > the command has been completed? It is a > lot > > more > > > > > > > usable if > > > > > > > > > >> the > > > > > > > > > >> > > > > command > > > > > > > > > >> > > > > > > has > > > > > > > > > >> > > > > > > > been completed so you know that if you > > create a > > > > > > topic > > > > > > > and > > > > > > > > > >> then > > > > > > > > > >> > > > > publish > > > > > > > > > >> > > > > > to > > > > > > > > > >> > > > > > > > it you won't get an exception about there > > being > > > > no > > > > > > > such > > > > > > > > > >> topic. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > A: For long running requests (like > reassign > > > > > > > partitions) - > > > > > > > > > >> the > > > > > > > > > >> > > post > > > > > > > > > >> > > > > > > > condition is > > > > > > > > > >> > > > > > > > command has begun - so we don't block the > > > > client. > > > > > In > > > > > > > case > > > > > > > > > >> of your > > > > > > > > > >> > > > > > > example - > > > > > > > > > >> > > > > > > > topic commands, this will be refactored > and > > > > topic > > > > > > > commands > > > > > > > > > >> will > > > > > > > > > >> > > be > > > > > > > > > >> > > > > > > executed > > > > > > > > > >> > > > > > > > immediately, since the Controller will > serve > > > > Admin > > > > > > > > > requests > > > > > > > > > >> > > > > > > > (follow-up ticket KAFKA-1777). > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > 8. Describe topic and list topics > duplicate > > a > > > > lot > > > > > of > > > > > > > stuff > > > > > > > > > >> in the > > > > > > > > > >> > > > > > > metadata > > > > > > > > > >> > > > > > > > request. Is there a reason to give back > > topics > > > > > > marked > > > > > > > for > > > > > > > > > >> > > > deletion? I > > > > > > > > > >> > > > > > > feel > > > > > > > > > >> > > > > > > > like if we just make the post-condition of > > the > > > > > > delete > > > > > > > > > >> command be > > > > > > > > > >> > > > that > > > > > > > > > >> > > > > > the > > > > > > > > > >> > > > > > > > topic is deleted that will get rid of the > > need > > > > for > > > > > > > this > > > > > > > > > >> right? > > > > > > > > > >> > > And > > > > > > > > > >> > > > it > > > > > > > > > >> > > > > > > will > > > > > > > > > >> > > > > > > > be much more intuitive. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - removed > > topics > > > > > marked > > > > > > > for > > > > > > > > > >> deletion > > > > > > > > > >> > > > in > > > > > > > > > >> > > > > > > > ListTopicsRequest. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > 9. Should we consider batching these > > requests? > > > > We > > > > > > have > > > > > > > > > >> generally > > > > > > > > > >> > > > > tried > > > > > > > > > >> > > > > > to > > > > > > > > > >> > > > > > > > allow multiple operations to be batched. > My > > > > > > suspicion > > > > > > > is > > > > > > > > > >> that > > > > > > > > > >> > > > without > > > > > > > > > >> > > > > > > this > > > > > > > > > >> > > > > > > > we will get a lot of code that does > > something > > > > like > > > > > > > > > >> > > > > > > > for(topic: adminClient.listTopics()) > > > > > > > > > >> > > > > > > > adminClient.describeTopic(topic) > > > > > > > > > >> > > > > > > > this code will work great when you test > on 5 > > > > > topics > > > > > > > but > > > > > > > > > not > > > > > > > > > >> do as > > > > > > > > > >> > > > > well > > > > > > > > > >> > > > > > if > > > > > > > > > >> > > > > > > > you have 50k. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > A: Updated the KIP - please check "Topic > > Admin > > > > > > Schema" > > > > > > > > > >> section. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > 10. I think we should also discuss how we > > want > > > > to > > > > > > > expose a > > > > > > > > > >> > > > > programmatic > > > > > > > > > >> > > > > > > JVM > > > > > > > > > >> > > > > > > > client api for these operations. Currently > > > > people > > > > > > > rely on > > > > > > > > > >> > > > AdminUtils > > > > > > > > > >> > > > > > > which > > > > > > > > > >> > > > > > > > is totally sketchy. I think we probably > need > > > > > another > > > > > > > > > client > > > > > > > > > >> under > > > > > > > > > >> > > > > > > clients/ > > > > > > > > > >> > > > > > > > that exposes administrative functionality. > > We > > > > will > > > > > > > need > > > > > > > > > >> this just > > > > > > > > > >> > > > to > > > > > > > > > >> > > > > > > > properly test the new apis, I suspect. We > > should > > > > > > > figure > > > > > > > > > out > > > > > > > > > >> that > > > > > > > > > >> > > > API. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > A: Updated the KIP - please check "Admin > > Client" > > > > > > > section > > > > > > > > > >> with an > > > > > > > > > >> > > > > > initial > > > > > > > > > >> > > > > > > > API proposal. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > 11. The other information that would be > > really > > > > > > useful > > > > > > > to > > > > > > > > > get > > > > > > > > > >> > > would > > > > > > > > > >> > > > be > > > > > > > > > >> > > > > > > > information about partitions--how much > data > > is > > > > in > > > > > > the > > > > > > > > > >> partition, > > > > > > > > > >> > > > what > > > > > > > > > >> > > > > > are > > > > > > > > > >> > > > > > > > the segment offsets, what is the log-end > > offset > > > > > > (i.e. > > > > > > > last > > > > > > > > > >> > > offset), > > > > > > > > > >> > > > > > what > > > > > > > > > >> > > > > > > is > > > > > > > > > >> > > > > > > > the compaction point, etc. I think that > done > > > > right > > > > > > > this > > > > > > > > > >> would be > > > > > > > > > >> > > > the > > > > > > > > > >> > > > > > > > successor to the very awkward > OffsetRequest > > we > > > > > have > > > > > > > today. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > A: I removed ConsumerGroupOffsetsRequest > in > > the > > > > > > latest > > > > > > > > > >> patch. I > > > > > > > > > >> > > > > believe > > > > > > > > > >> > > > > > > > this should > > > > > > > > > >> > > > > > > > be resolved in a separate KIP / jira > ticket. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > 12. Generally we can do good error > handling > > > > > without > > > > > > > > > needing > > > > > > > > > >> > > custom > > > > > > > > > >> > > > > > > > server-side > > > > > > > > > >> > > > > > > > messages. I.e. generally the client has > the > > > > > context > > > > > > to > > > > > > > > > know > > > > > > > > > >> that > > > > > > > > > >> > > if > > > > > > > > > >> > > > > it > > > > > > > > > >> > > > > > > got > > > > > > > > > >> > > > > > > > an error that the topic doesn't exist to > say > > > > > "Topic > > > > > > X > > > > > > > > > >> doesn't > > > > > > > > > >> > > > exist" > > > > > > > > > >> > > > > > > rather > > > > > > > > > >> > > > > > > > than "error code 14" (or whatever). Maybe > > there > > > > > are > > > > > > > > > specific > > > > > > > > > >> > > cases > > > > > > > > > >> > > > > > where > > > > > > > > > >> > > > > > > > this is hard? If we want to add > server-side > > > > error > > > > > > > messages > > > > > > > > > >> we > > > > > > > > > >> > > > really > > > > > > > > > >> > > > > do > > > > > > > > > >> > > > > > > > need to do this in a consistent way across > > the > > > > > > > protocol. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > A: Updated the KIP - please check > "Protocol > > > > > Errors" > > > > > > > > > >> section. I > > > > > > > > > >> > > > added > > > > > > > > > >> > > > > > the > > > > > > > > > >> > > > > > > > comprehensive, fine-grained list of error > > codes. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > Comments from Guozhang: > > > > > > > > > >> > > > > > > > 13. Describe topic request: it would be > > great to > > > > > go > > > > > > > beyond > > > > > > > > > >> just > > > > > > > > > >> > > > > > batching > > > > > > > > > >> > > > > > > on > > > > > > > > > >> > > > > > > > topic name regex for this request. For > > example, > > > > a > > > > > > very > > > > > > > > > >> common use > > > > > > > > > >> > > > > case > > > > > > > > > >> > > > > > of > > > > > > > > > >> > > > > > > > the topic command is to list all topics > > whose > > > > > config > > > > > > > A's > > > > > > > > > >> value is > > > > > > > > > >> > > > B. > > > > > > > > > >> > > > > > With > > > > > > > > > >> > > > > > > > topic name regex then we have to first > > retrieve > > > > > > > __all__ > > > > > > > > > >> topics's > > > > > > > > > >> > > > > > > > description info and then filter at the > > client > > > > > end, > > > > > > > which > > > > > > > > > >> will > > > > > > > > > >> > > be a > > > > > > > > > >> > > > > > huge > > > > > > > > > >> > > > > > > > burden on ZK. > > > > > > > > > >> > > > > > > > AND > > > > > > > > > >> > > > > > > > 14. Config K-Vs in create topic: this is > > related > > > > > to > > > > > > > the > > > > > > > > > >> previous > > > > > > > > > >> > > > > point; > > > > > > > > > >> > > > > > > > maybe we can add another metadata K-V or > > just a > > > > > > > metadata > > > > > > > > > >> string > > > > > > > > > >> > > > along > > > > > > > > > >> > > > > > > side > > > > > > > > > >> > > > > > > > with config K-V in create topic like we > did > > for > > > > > > offset > > > > > > > > > >> commit > > > > > > > > > >> > > > > request. > > > > > > > > > >> > > > > > > This > > > > > > > > > >> > > > > > > > field can be quite useful in storing > > information > > > > > > like > > > > > > > > > >> "owner" of > > > > > > > > > >> > > > the > > > > > > > > > >> > > > > > > topic > > > > > > > > > >> > > > > > > > who issue the create command, etc, which > is > > > > quite > > > > > > > > > important > > > > > > > > > >> for a > > > > > > > > > >> > > > > > > > multi-tenant setting. Then in the describe > > topic > > > > > > > request > > > > > > > > > we > > > > > > > > > >> can > > > > > > > > > >> > > > also > > > > > > > > > >> > > > > > > batch > > > > > > > > > >> > > > > > > > on regex of the metadata field. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > A: As discussed it is very interesting but > > can > > > > be > > > > > > > > > >> implemented > > > > > > > > > >> > > later > > > > > > > > > >> > > > > > after > > > > > > > > > >> > > > > > > > we have some basic functionality there. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > 15. Today all the admin operations are > > async in > > > > > the > > > > > > > sense > > > > > > > > > >> that > > > > > > > > > >> > > > > command > > > > > > > > > >> > > > > > > will > > > > > > > > > >> > > > > > > > return once it is written in ZK, and that > > is why > > > > > we > > > > > > > need > > > > > > > > > >> extra > > > > > > > > > >> > > > > > > verification > > > > > > > > > >> > > > > > > > like testUtil.waitForTopicCreated() / > verify > > > > > > partition > > > > > > > > > >> > > reassignment > > > > > > > > > >> > > > > > > > request, etc. With admin requests we could > > add a > > > > > > flag > > > > > > > to > > > > > > > > > >> enable / > > > > > > > > > >> > > > > > disable > > > > > > > > > >> > > > > > > > synchronous requests; when it is turned > on, > > the > > > > > > > response > > > > > > > > > >> will not > > > > > > > > > >> > > > > > return > > > > > > > > > >> > > > > > > > until the request has been completed. And > > for > > > > > async > > > > > > > > > >> requests we > > > > > > > > > >> > > can > > > > > > > > > >> > > > > > add a > > > > > > > > > >> > > > > > > > "token" field in the response, and then > only > > > > need > > > > > a > > > > > > > > > general > > > > > > > > > >> > > "admin > > > > > > > > > >> > > > > > > > verification request" with the given token > > to > > > > > check > > > > > > > if the > > > > > > > > > >> async > > > > > > > > > >> > > > > > request > > > > > > > > > >> > > > > > > > has been completed. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > A: I see your point. My idea was to > provide > > > > > specific > > > > > > > > > >> > > > Verify...Request > > > > > > > > > >> > > > > > per > > > > > > > > > >> > > > > > > > each > > > > > > > > > >> > > > > > > > long running request, where needed. We can > > do it > > > > > the > > > > > > > way > > > > > > > > > you > > > > > > > > > >> > > > suggest. > > > > > > > > > >> > > > > > The > > > > > > > > > >> > > > > > > > only > > > > > > > > > >> > > > > > > > concern is that introducing a token we > again > > > > will > > > > > > make > > > > > > > > > >> schema > > > > > > > > > >> > > > > > "dynamic". > > > > > > > > > >> > > > > > > We > > > > > > > > > >> > > > > > > > wanted > > > > > > > > > >> > > > > > > > to do similar thing introducing single > > > > > AdminRequest > > > > > > > for > > > > > > > > > all > > > > > > > > > >> topic > > > > > > > > > >> > > > > > > commands > > > > > > > > > >> > > > > > > > but rejected > > > > > > > > > >> > > > > > > > this idea because we wanted to have schema > > > > > defined. > > > > > > So > > > > > > > > > this > > > > > > > > > >> is > > > > > > > > > >> > > > more a > > > > > > > > > >> > > > > > > > choice between: > > > > > > > > > >> > > > > > > > a) have fixed schema but introduce each > > time new > > > > > > > > > >> Verify...Request > > > > > > > > > >> > > > for > > > > > > > > > >> > > > > > > > long-running requests > > > > > > > > > >> > > > > > > > b) use one request for verification but > > > > generalize > > > > > > it > > > > > > > with > > > > > > > > > >> token > > > > > > > > > >> > > > > > > > I'm fine with whatever decision community > > come > > > > to. > > > > > > > Just > > > > > > > > > let > > > > > > > > > >> me > > > > > > > > > >> > > know > > > > > > > > > >> > > > > > your > > > > > > > > > >> > > > > > > > thoughts. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > Comment from Gwen: > > > > > > > > > >> > > > > > > > 16. Specifically for ownership, I think > the > > plan > > > > > is > > > > > > > to add > > > > > > > > > >> ACL > > > > > > > > > >> > > (it > > > > > > > > > >> > > > > > sounds > > > > > > > > > >> > > > > > > > like you are describing ACL) via an > external > > > > > system > > > > > > > > > (Argus, > > > > > > > > > >> > > > Sentry). > > > > > > > > > >> > > > > > > > I remember KIP-11 described this, but I > > can't > > > > find > > > > > > > the KIP > > > > > > > > > >> any > > > > > > > > > >> > > > > longer. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > A: Okay, no problem. Not sure though how > we > > are > > > > > > going > > > > > > > to > > > > > > > > > >> handle > > > > > > > > > >> > > it. > > > > > > > > > >> > > > > > Wait > > > > > > > > > >> > > > > > > > which KIP > > > > > > > > > >> > > > > > > > will be committed first and include > changes > > to > > > > > > > > > >> TopicMetadata from > > > > > > > > > >> > > > the > > > > > > > > > >> > > > > > > later > > > > > > > > > >> > > > > > > > one? > > > > > > > > > >> > > > > > > > Anyway, I added this note to "Open > > Questions" > > > > > > section > > > > > > > so > > > > > > > > > we > > > > > > > > > >> don't > > > > > > > > > >> > > > > miss > > > > > > > > > >> > > > > > > this > > > > > > > > > >> > > > > > > > piece. > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > Thanks, > > > > > > > > > >> > > > > > > > Andrii Biletskyi > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > On Fri, Mar 13, 2015 at 12:34 AM, Andrii > > > > > Biletskyi < > > > > > > > > > >> > > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > Hi all, > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > Today I uploaded the patch that covers > > some of > > > > > the > > > > > > > > > >> discussed > > > > > > > > > >> > > and > > > > > > > > > >> > > > > > agreed > > > > > > > > > >> > > > > > > > > items: > > > > > > > > > >> > > > > > > > > - removed MaybeOf optional type > > > > > > > > > >> > > > > > > > > - switched to java protocol definitions > > > > > > > > > >> > > > > > > > > - simplified messages (normalized > configs, > > > > > removed > > > > > > > topic > > > > > > > > > >> marked > > > > > > > > > >> > > > for > > > > > > > > > >> > > > > > > > > deletion) > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > I also updated the KIP-4 with respective > > > > changes > > > > > > and > > > > > > > > > >> wrote down > > > > > > > > > >> > > > my > > > > > > > > > >> > > > > > > > > proposal for > > > > > > > > > >> > > > > > > > > pending items: > > > > > > > > > >> > > > > > > > > - Batch Admin Operations -> updated Wire > > > > > Protocol > > > > > > > schema > > > > > > > > > >> > > proposal > > > > > > > > > >> > > > > > > > > - Remove ClusterMetadata -> changed to > > extend > > > > > > > > > >> > > > TopicMetadataRequest > > > > > > > > > >> > > > > > > > > - Admin Client -> updated my initial > > proposal > > > > to > > > > > > > reflect > > > > > > > > > >> > > batching > > > > > > > > > >> > > > > > > > > - Error codes -> proposed fine-grained > > error > > > > > code > > > > > > > > > instead > > > > > > > > > >> of > > > > > > > > > >> > > > > > > > > AdminRequestFailed > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > I will also send a separate email to > > cover all > > > > > > > comments > > > > > > > > > >> from > > > > > > > > > >> > > this > > > > > > > > > >> > > > > > > thread. > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > Thanks, > > > > > > > > > >> > > > > > > > > Andrii Biletskyi > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > On Thu, Mar 12, 2015 at 9:26 PM, Gwen > > Shapira > > > > < > > > > > > > > > >> > > > > gshap...@cloudera.com > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > wrote: > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > >> Found KIP-11 ( > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface > > > > > > > > > >> > > > > > > > >> ) > > > > > > > > > >> > > > > > > > >> It actually specifies changes to the > > Metadata > > > > > > > protocol, > > > > > > > > > >> so > > > > > > > > > >> > > > making > > > > > > > > > >> > > > > > sure > > > > > > > > > >> > > > > > > > >> both KIPs are consistent in this regard > > will > > > > be > > > > > > > good. > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > > > >> On Thu, Mar 12, 2015 at 12:21 PM, Gwen > > > > Shapira > > > > > < > > > > > > > > > >> > > > > > gshap...@cloudera.com > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > > > >> > > > > > > > >> > Specifically for ownership, I think > the > > > > plan > > > > > is > > > > > > > to > > > > > > > > > add > > > > > > > > > >> ACL > > > > > > > > > >> > > (it > > > > > > > > > >> > > > > > > sounds > > > > > > > > > >> > > > > > > > >> > like you are describing ACL) via an > > > > external > > > > > > > system > > > > > > > > > >> (Argus, > > > > > > > > > >> > > > > > Sentry). > > > > > > > > > >> > > > > > > > >> > I remember KIP-11 described this, > but I > > > > can't > > > > > > > find > > > > > > > > > the > > > > > > > > > >> KIP > > > > > > > > > >> > > any > > > > > > > > > >> > > > > > > longer. > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > >> > > > > > > > >> > Regardless, I think KIP-4 focuses on > > > > getting > > > > > > > > > >> information > > > > > > > > > >> > > that > > > > > > > > > >> > > > > > > already > > > > > > > > > >> > > > > > > > >> > exists from Kafka brokers, not on > > adding > > > > > > > information > > > > > > > > > >> that > > > > > > > > > >> > > > > perhaps > > > > > > > > > >> > > > > > > > >> > should exist but doesn't yet? > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > >> > > > > > > > >> > Gwen > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > >> > > > > > > > >> > On Thu, Mar 12, 2015 at 6:37 AM, > > Guozhang > > > > > Wang > > > > > > < > > > > > > > > > >> > > > > > wangg...@gmail.com> > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > > > >> > > > > > > > >> >> Folks, > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > >> > > > > > > > >> >> Just want to elaborate a bit more on > > the > > > > > > > > > create-topic > > > > > > > > > >> > > > metadata > > > > > > > > > >> > > > > > and > > > > > > > > > >> > > > > > > > >> batching > > > > > > > > > >> > > > > > > > >> >> describe-topic based on config / > > metadata > > > > in > > > > > > my > > > > > > > > > >> previous > > > > > > > > > >> > > > email > > > > > > > > > >> > > > > as > > > > > > > > > >> > > > > > > we > > > > > > > > > >> > > > > > > > >> work > > > > > > > > > >> > > > > > > > >> >> on KAFKA-1694. The main motivation > is > > to > > > > > have > > > > > > > some > > > > > > > > > >> sort of > > > > > > > > > >> > > > > topic > > > > > > > > > >> > > > > > > > >> management > > > > > > > > > >> > > > > > > > >> >> mechanisms, which I think is quite > > > > important > > > > > > in > > > > > > > a > > > > > > > > > >> > > > multi-tenant > > > > > > > > > >> > > > > / > > > > > > > > > >> > > > > > > > cloud > > > > > > > > > >> > > > > > > > >> >> architecture: today anyone can > create > > > > topics > > > > > > in > > > > > > > a > > > > > > > > > >> shared > > > > > > > > > >> > > > Kafka > > > > > > > > > >> > > > > > > > >> cluster, but > > > > > > > > > >> > > > > > > > >> >> there is no concept or "ownership" > of > > > > topics > > > > > > > that > > > > > > > > > are > > > > > > > > > >> > > created > > > > > > > > > >> > > > > by > > > > > > > > > >> > > > > > > > >> different > > > > > > > > > >> > > > > > > > >> >> users. For example, at LinkedIn we > > > > basically > > > > > > > > > >> distinguish > > > > > > > > > >> > > > topic > > > > > > > > > >> > > > > > > owners > > > > > > > > > >> > > > > > > > >> via > > > > > > > > > >> > > > > > > > >> >> some casual topic name prefix, which > > is a > > > > > bit > > > > > > > > > awkward > > > > > > > > > >> and > > > > > > > > > >> > > > does > > > > > > > > > >> > > > > > not > > > > > > > > > >> > > > > > > > fly > > > > > > > > > >> > > > > > > > >> as > > > > > > > > > >> > > > > > > > >> >> we scale our customers. It would be > > great > > > > to > > > > > > use > > > > > > > > > >> > > > > describe-topics > > > > > > > > > >> > > > > > > such > > > > > > > > > >> > > > > > > > >> as: > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > >> > > > > > > > >> >> Describe all topics that is created > > by me. > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > >> > > > > > > > >> >> Describe all topics whose retention > > time > > > > is > > > > > > > > > overriden > > > > > > > > > >> to X. > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > >> > > > > > > > >> >> Describe all topics whose writable > > group > > > > > > include > > > > > > > > > user > > > > > > > > > >> Y > > > > > > > > > >> > > (this > > > > > > > > > >> > > > > is > > > > > > > > > >> > > > > > > > >> related to > > > > > > > > > >> > > > > > > > >> >> authorization), etc.. > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > >> > > > > > > > >> >> One possible way to achieve this is > to > > > > add a > > > > > > > > > metadata > > > > > > > > > >> file > > > > > > > > > >> > > in > > > > > > > > > >> > > > > the > > > > > > > > > >> > > > > > > > >> >> create-topic request, whose value > will > > > > also > > > > > be > > > > > > > > > >> written ZK > > > > > > > > > >> > > as > > > > > > > > > >> > > > we > > > > > > > > > >> > > > > > > > create > > > > > > > > > >> > > > > > > > >> the > > > > > > > > > >> > > > > > > > >> >> topic; then describe-topics can > > choose to > > > > > > batch > > > > > > > > > topics > > > > > > > > > >> > > based > > > > > > > > > >> > > > on > > > > > > > > > >> > > > > > 1) > > > > > > > > > >> > > > > > > > name > > > > > > > > > >> > > > > > > > >> >> regex, 2) config K-V matching, 3) > > metadata > > > > > > > regex, > > > > > > > > > etc. > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > >> > > > > > > > >> >> Thoughts? > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > >> > > > > > > > >> >> Guozhang > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > >> > > > > > > > >> >> On Thu, Mar 5, 2015 at 4:37 PM, > > Guozhang > > > > > Wang > > > > > > < > > > > > > > > > >> > > > > > wangg...@gmail.com> > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > >> > > > > > > > >> >>> Thanks for the updated wiki. A few > > > > comments > > > > > > > below: > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > >> > > > > > > > >> >>> 1. Error description in response: I > > think > > > > > if > > > > > > > some > > > > > > > > > >> > > errorCode > > > > > > > > > >> > > > > > could > > > > > > > > > >> > > > > > > > >> indicate > > > > > > > > > >> > > > > > > > >> >>> several different error cases then > we > > > > > should > > > > > > > really > > > > > > > > > >> change > > > > > > > > > >> > > > it > > > > > > > > > >> > > > > to > > > > > > > > > >> > > > > > > > >> multiple > > > > > > > > > >> > > > > > > > >> >>> codes. In general the errorCode > > itself > > > > > would > > > > > > be > > > > > > > > > >> precise > > > > > > > > > >> > > and > > > > > > > > > >> > > > > > > > >> sufficient for > > > > > > > > > >> > > > > > > > >> >>> describing the server side errors. > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > >> > > > > > > > >> >>> 2. Describe topic request: it would > > be > > > > > great > > > > > > > to go > > > > > > > > > >> beyond > > > > > > > > > >> > > > just > > > > > > > > > >> > > > > > > > >> batching on > > > > > > > > > >> > > > > > > > >> >>> topic name regex for this request. > > For > > > > > > > example, a > > > > > > > > > >> very > > > > > > > > > >> > > > common > > > > > > > > > >> > > > > > use > > > > > > > > > >> > > > > > > > >> case of > > > > > > > > > >> > > > > > > > >> >>> the topic command is to list all > > topics > > > > > whose > > > > > > > > > config > > > > > > > > > >> A's > > > > > > > > > >> > > > value > > > > > > > > > >> > > > > > is > > > > > > > > > >> > > > > > > B. > > > > > > > > > >> > > > > > > > >> With > > > > > > > > > >> > > > > > > > >> >>> topic name regex then we have to > > first > > > > > > retrieve > > > > > > > > > >> __all__ > > > > > > > > > >> > > > > topics's > > > > > > > > > >> > > > > > > > >> >>> description info and then filter at > > the > > > > > > client > > > > > > > end, > > > > > > > > > >> which > > > > > > > > > >> > > > will > > > > > > > > > >> > > > > > be > > > > > > > > > >> > > > > > > a > > > > > > > > > >> > > > > > > > >> huge > > > > > > > > > >> > > > > > > > >> >>> burden on ZK. > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > >> > > > > > > > >> >>> 3. Config K-Vs in create topic: > this > > is > > > > > > > related to > > > > > > > > > >> the > > > > > > > > > >> > > > > previous > > > > > > > > > >> > > > > > > > point; > > > > > > > > > >> > > > > > > > >> >>> maybe we can add another metadata > > K-V or > > > > > > just a > > > > > > > > > >> metadata > > > > > > > > > >> > > > > string > > > > > > > > > >> > > > > > > > along > > > > > > > > > >> > > > > > > > >> side > > > > > > > > > >> > > > > > > > >> >>> with config K-V in create topic > like > > we > > > > did > > > > > > for > > > > > > > > > >> offset > > > > > > > > > >> > > > commit > > > > > > > > > >> > > > > > > > >> request. This > > > > > > > > > >> > > > > > > > >> >>> field can be quite useful in > storing > > > > > > > information > > > > > > > > > like > > > > > > > > > >> > > > "owner" > > > > > > > > > >> > > > > of > > > > > > > > > >> > > > > > > the > > > > > > > > > >> > > > > > > > >> topic > > > > > > > > > >> > > > > > > > >> >>> who issue the create command, etc, > > which > > > > is > > > > > > > quite > > > > > > > > > >> > > important > > > > > > > > > >> > > > > for > > > > > > > > > >> > > > > > a > > > > > > > > > >> > > > > > > > >> >>> multi-tenant setting. Then in the > > > > describe > > > > > > > topic > > > > > > > > > >> request > > > > > > > > > >> > > we > > > > > > > > > >> > > > > can > > > > > > > > > >> > > > > > > also > > > > > > > > > >> > > > > > > > >> batch > > > > > > > > > >> > > > > > > > >> >>> on regex of the metadata field. > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > >> > > > > > > > >> >>> 4. Today all the admin operations > are > > > > async > > > > > > in > > > > > > > the > > > > > > > > > >> sense > > > > > > > > > >> > > > that > > > > > > > > > >> > > > > > > > command > > > > > > > > > >> > > > > > > > >> will > > > > > > > > > >> > > > > > > > >> >>> return once it is written in ZK, > and > > that > > > > > is > > > > > > > why we > > > > > > > > > >> need > > > > > > > > > >> > > > extra > > > > > > > > > >> > > > > > > > >> verification > > > > > > > > > >> > > > > > > > >> >>> like > testUtil.waitForTopicCreated() / > > > > > verify > > > > > > > > > >> partition > > > > > > > > > >> > > > > > > reassignment > > > > > > > > > >> > > > > > > > >> >>> request, etc. With admin requests > we > > > > could > > > > > > add > > > > > > > a > > > > > > > > > >> flag to > > > > > > > > > >> > > > > enable > > > > > > > > > >> > > > > > / > > > > > > > > > >> > > > > > > > >> disable > > > > > > > > > >> > > > > > > > >> >>> synchronous requests; when it is > > turned > > > > on, > > > > > > the > > > > > > > > > >> response > > > > > > > > > >> > > > will > > > > > > > > > >> > > > > > not > > > > > > > > > >> > > > > > > > >> return > > > > > > > > > >> > > > > > > > >> >>> until the request has been > > completed. And > > > > > for > > > > > > > async > > > > > > > > > >> > > requests > > > > > > > > > >> > > > > we > > > > > > > > > >> > > > > > > can > > > > > > > > > >> > > > > > > > >> add a > > > > > > > > > >> > > > > > > > >> >>> "token" field in the response, and > > then > > > > > only > > > > > > > need a > > > > > > > > > >> > > general > > > > > > > > > >> > > > > > "admin > > > > > > > > > >> > > > > > > > >> >>> verification request" with the > given > > > > token > > > > > to > > > > > > > check > > > > > > > > > >> if the > > > > > > > > > >> > > > > async > > > > > > > > > >> > > > > > > > >> request > > > > > > > > > >> > > > > > > > >> >>> has been completed. > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > >> > > > > > > > >> >>> 5. +1 for extending Metadata > request > > to > > > > > > include > > > > > > > > > >> > > controller / > > > > > > > > > >> > > > > > > > >> coordinator > > > > > > > > > >> > > > > > > > >> >>> information, and then we can remove > > the > > > > > > > > > >> ConsumerMetadata / > > > > > > > > > >> > > > > > > > >> ClusterMetadata > > > > > > > > > >> > > > > > > > >> >>> requests. > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > >> > > > > > > > >> >>> Guozhang > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > >> > > > > > > > >> >>> On Tue, Mar 3, 2015 at 10:23 AM, > Joel > > > > > Koshy < > > > > > > > > > >> > > > > > jjkosh...@gmail.com> > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > >> > > > > > > > >> >>>> Thanks for sending that out Joe - > I > > > > don't > > > > > > > think I > > > > > > > > > >> will be > > > > > > > > > >> > > > > able > > > > > > > > > >> > > > > > to > > > > > > > > > >> > > > > > > > >> make > > > > > > > > > >> > > > > > > > >> >>>> it today, so if notes can be sent > > out > > > > > > > afterward > > > > > > > > > that > > > > > > > > > >> > > would > > > > > > > > > >> > > > be > > > > > > > > > >> > > > > > > > great. > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > >> > > > > > > > >> >>>> On Mon, Mar 02, 2015 at 09:16:13AM > > > > -0800, > > > > > > Gwen > > > > > > > > > >> Shapira > > > > > > > > > >> > > > wrote: > > > > > > > > > >> > > > > > > > >> >>>> > Thanks for sending this out Joe. > > > > Looking > > > > > > > forward > > > > > > > > > >> to > > > > > > > > > >> > > > > chatting > > > > > > > > > >> > > > > > > with > > > > > > > > > >> > > > > > > > >> >>>> everyone :) > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > >> > > > > > > > >> >>>> > On Mon, Mar 2, 2015 at 6:46 AM, > > Joe > > > > > Stein > > > > > > < > > > > > > > > > >> > > > > > > joe.st...@stealth.ly> > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > > > >> > > > > > > > >> >>>> > > Hey, I just sent out a google > > > > hangout > > > > > > > invite > > > > > > > > > to > > > > > > > > > >> all > > > > > > > > > >> > > > pmc, > > > > > > > > > >> > > > > > > > >> committers > > > > > > > > > >> > > > > > > > >> >>>> and > > > > > > > > > >> > > > > > > > >> >>>> > > everyone I found working on a > > KIP. > > > > If > > > > > I > > > > > > > missed > > > > > > > > > >> anyone > > > > > > > > > >> > > > in > > > > > > > > > >> > > > > > the > > > > > > > > > >> > > > > > > > >> invite > > > > > > > > > >> > > > > > > > >> >>>> please > > > > > > > > > >> > > > > > > > >> >>>> > > let me know and can update it, > > np. > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > >> > > > > > > > >> >>>> > > We should do this every > Tuesday > > @ > > > > 2pm > > > > > > > Eastern > > > > > > > > > >> Time. > > > > > > > > > >> > > > Maybe > > > > > > > > > >> > > > > > we > > > > > > > > > >> > > > > > > > can > > > > > > > > > >> > > > > > > > >> get > > > > > > > > > >> > > > > > > > >> >>>> INFRA > > > > > > > > > >> > > > > > > > >> >>>> > > help to make a google account > > so we > > > > > can > > > > > > > manage > > > > > > > > > >> > > better? > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > >> > > > > > > > >> >>>> > > To discuss > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > > > > > > > >> > > > > > > > >> >>>> > > in progress and related JIRA > > that > > > > are > > > > > > > > > >> interdependent > > > > > > > > > >> > > > and > > > > > > > > > >> > > > > > > common > > > > > > > > > >> > > > > > > > >> work. > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > >> > > > > > > > >> >>>> > > ~ Joe Stein > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > >> > > > > > > > >> >>>> > > On Tue, Feb 24, 2015 at 2:59 > > PM, Jay > > > > > > > Kreps < > > > > > > > > > >> > > > > > > > jay.kr...@gmail.com> > > > > > > > > > >> > > > > > > > >> >>>> wrote: > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > >> > > > > > > > >> >>>> > >> Let's stay on Google hangouts > > that > > > > > will > > > > > > > also > > > > > > > > > >> record > > > > > > > > > >> > > > and > > > > > > > > > >> > > > > > make > > > > > > > > > >> > > > > > > > the > > > > > > > > > >> > > > > > > > >> >>>> sessions > > > > > > > > > >> > > > > > > > >> >>>> > >> available on youtube. > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > >> > > > > > > > >> >>>> > >> -Jay > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > >> > > > > > > > >> >>>> > >> On Tue, Feb 24, 2015 at 11:49 > > AM, > > > > > Jeff > > > > > > > > > Holoman > > > > > > > > > >> < > > > > > > > > > >> > > > > > > > >> >>>> jholo...@cloudera.com> > > > > > > > > > >> > > > > > > > >> >>>> > >> wrote: > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > >> > > > > > > > >> >>>> > >> > Jay / Joe > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > >> > > > > > > > >> >>>> > >> > We're happy to send out a > > Webex > > > > for > > > > > > > this > > > > > > > > > >> purpose. > > > > > > > > > >> > > We > > > > > > > > > >> > > > > > could > > > > > > > > > >> > > > > > > > >> record > > > > > > > > > >> > > > > > > > >> >>>> the > > > > > > > > > >> > > > > > > > >> >>>> > >> > sessions if there is > > interest and > > > > > > > publish > > > > > > > > > >> them > > > > > > > > > >> > > out. > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > >> > > > > > > > >> >>>> > >> > Thanks > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > >> > > > > > > > >> >>>> > >> > Jeff > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > >> > > > > > > > >> >>>> > >> > On Tue, Feb 24, 2015 at > > 11:28 AM, > > > > > Jay > > > > > > > > > Kreps < > > > > > > > > > >> > > > > > > > >> jay.kr...@gmail.com> > > > > > > > > > >> > > > > > > > >> >>>> wrote: > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > Let's try to get the > > technical > > > > > > > hang-ups > > > > > > > > > >> sorted > > > > > > > > > >> > > > out, > > > > > > > > > >> > > > > > > > though. > > > > > > > > > >> > > > > > > > >> I > > > > > > > > > >> > > > > > > > >> >>>> really > > > > > > > > > >> > > > > > > > >> >>>> > >> > think > > > > > > > > > >> > > > > > > > >> >>>> > >> > > there is some benefit to > > live > > > > > > > discussion > > > > > > > > > vs > > > > > > > > > >> > > > > writing. I > > > > > > > > > >> > > > > > > am > > > > > > > > > >> > > > > > > > >> >>>> hopeful that > > > > > > > > > >> > > > > > > > >> >>>> > >> if > > > > > > > > > >> > > > > > > > >> >>>> > >> > > we post instructions and > > give > > > > > > > ourselves a > > > > > > > > > >> few > > > > > > > > > >> > > > > attempts > > > > > > > > > >> > > > > > > we > > > > > > > > > >> > > > > > > > >> can > > > > > > > > > >> > > > > > > > >> >>>> get it > > > > > > > > > >> > > > > > > > >> >>>> > >> > > working. > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > Tuesday at that time > would > > work > > > > > for > > > > > > > > > >> me...any > > > > > > > > > >> > > > > > objections? > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > -Jay > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > On Tue, Feb 24, 2015 at > > 8:18 > > > > AM, > > > > > > Joe > > > > > > > > > Stein > > > > > > > > > >> < > > > > > > > > > >> > > > > > > > >> joe.st...@stealth.ly > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > >> > > > > > > > >> >>>> > >> wrote: > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > Weekly would be great > > maybe > > > > > like > > > > > > > every > > > > > > > > > >> > > Tuesday ~ > > > > > > > > > >> > > > > 1pm > > > > > > > > > >> > > > > > > ET > > > > > > > > > >> > > > > > > > / > > > > > > > > > >> > > > > > > > >> 10am > > > > > > > > > >> > > > > > > > >> >>>> PT > > > > > > > > > >> > > > > > > > >> >>>> > >> ???? > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > I don't mind google > > hangout > > > > but > > > > > > > there > > > > > > > > > is > > > > > > > > > >> > > always > > > > > > > > > >> > > > > some > > > > > > > > > >> > > > > > > > >> issue or > > > > > > > > > >> > > > > > > > >> >>>> > >> whatever > > > > > > > > > >> > > > > > > > >> >>>> > >> > so > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > we know the apache irc > > > > channel > > > > > > > works. > > > > > > > > > We > > > > > > > > > >> can > > > > > > > > > >> > > > start > > > > > > > > > >> > > > > > > there > > > > > > > > > >> > > > > > > > >> and > > > > > > > > > >> > > > > > > > >> >>>> see how > > > > > > > > > >> > > > > > > > >> >>>> > >> it > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > goes? We can pull > > transcripts > > > > > too > > > > > > > and > > > > > > > > > >> > > associate > > > > > > > > > >> > > > to > > > > > > > > > >> > > > > > > > >> tickets if > > > > > > > > > >> > > > > > > > >> >>>> need be > > > > > > > > > >> > > > > > > > >> >>>> > >> > > makes > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > it helpful for things. > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > ~ Joestein > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > On Tue, Feb 24, 2015 at > > 11:10 > > > > > AM, > > > > > > > Jay > > > > > > > > > >> Kreps < > > > > > > > > > >> > > > > > > > >> >>>> jay.kr...@gmail.com> > > > > > > > > > >> > > > > > > > >> >>>> > >> > wrote: > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > We'd talked about > > doing a > > > > > > Google > > > > > > > > > >> Hangout to > > > > > > > > > >> > > > chat > > > > > > > > > >> > > > > > > about > > > > > > > > > >> > > > > > > > >> this. > > > > > > > > > >> > > > > > > > >> >>>> What > > > > > > > > > >> > > > > > > > >> >>>> > >> > about > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > generalizing that a > > little > > > > > > > > > further...I > > > > > > > > > >> > > > actually > > > > > > > > > >> > > > > > > think > > > > > > > > > >> > > > > > > > it > > > > > > > > > >> > > > > > > > >> >>>> would be > > > > > > > > > >> > > > > > > > >> >>>> > >> > good > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > for > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > everyone spending a > > > > > reasonable > > > > > > > chunk > > > > > > > > > of > > > > > > > > > >> > > their > > > > > > > > > >> > > > > week > > > > > > > > > >> > > > > > > on > > > > > > > > > >> > > > > > > > >> Kafka > > > > > > > > > >> > > > > > > > >> >>>> stuff > > > > > > > > > >> > > > > > > > >> >>>> > >> to > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > maybe > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > sync up once a week. > I > > > > think > > > > > we > > > > > > > could > > > > > > > > > >> use > > > > > > > > > >> > > time > > > > > > > > > >> > > > > to > > > > > > > > > >> > > > > > > talk > > > > > > > > > >> > > > > > > > >> >>>> through > > > > > > > > > >> > > > > > > > >> >>>> > >> design > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > stuff, make sure we > > are on > > > > > top > > > > > > of > > > > > > > > > code > > > > > > > > > >> > > > reviews, > > > > > > > > > >> > > > > > talk > > > > > > > > > >> > > > > > > > >> through > > > > > > > > > >> > > > > > > > >> >>>> any > > > > > > > > > >> > > > > > > > >> >>>> > >> > tricky > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > issues, etc. > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > We can make it > publicly > > > > > > > available so > > > > > > > > > >> that > > > > > > > > > >> > > any > > > > > > > > > >> > > > > one > > > > > > > > > >> > > > > > > can > > > > > > > > > >> > > > > > > > >> follow > > > > > > > > > >> > > > > > > > >> >>>> along > > > > > > > > > >> > > > > > > > >> >>>> > >> > who > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > likes. > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > Any interest in doing > > this? > > > > > If > > > > > > so > > > > > > > > > I'll > > > > > > > > > >> try > > > > > > > > > >> > > to > > > > > > > > > >> > > > > set > > > > > > > > > >> > > > > > it > > > > > > > > > >> > > > > > > > up > > > > > > > > > >> > > > > > > > >> >>>> starting > > > > > > > > > >> > > > > > > > >> >>>> > >> next > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > week. > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > -Jay > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > On Tue, Feb 24, 2015 > at > > > > 3:57 > > > > > > AM, > > > > > > > > > Andrii > > > > > > > > > >> > > > > Biletskyi > > > > > > > > > >> > > > > > < > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > andrii.bilets...@stealth.ly> > > > > > > > wrote: > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > Hi all, > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > I've updated KIP > > page, > > > > > fixed > > > > > > / > > > > > > > > > >> aligned > > > > > > > > > >> > > > > document > > > > > > > > > >> > > > > > > > >> structure. > > > > > > > > > >> > > > > > > > >> >>>> Also I > > > > > > > > > >> > > > > > > > >> >>>> > >> > > added > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > some > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > very initial > > proposal for > > > > > > > > > >> AdminClient so > > > > > > > > > >> > > we > > > > > > > > > >> > > > > have > > > > > > > > > >> > > > > > > > >> something > > > > > > > > > >> > > > > > > > >> >>>> to > > > > > > > > > >> > > > > > > > >> >>>> > >> start > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > from > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > while > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > discussing the KIP. > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > >> > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > Thanks, > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > Andrii Biletskyi > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > On Wed, Feb 18, > 2015 > > at > > > > > 9:01 > > > > > > > PM, > > > > > > > > > >> Andrii > > > > > > > > > >> > > > > > Biletskyi > > > > > > > > > >> > > > > > > < > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > andrii.bilets...@stealth.ly> > > > > > > > > > wrote: > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Jay, > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Re error > messages: > > you > > > > > are > > > > > > > right, > > > > > > > > > >> in > > > > > > > > > >> > > most > > > > > > > > > >> > > > > > cases > > > > > > > > > >> > > > > > > > >> client > > > > > > > > > >> > > > > > > > >> >>>> will > > > > > > > > > >> > > > > > > > >> >>>> > >> have > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > enough > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > context to show > > > > > descriptive > > > > > > > error > > > > > > > > > >> > > message. > > > > > > > > > >> > > > > My > > > > > > > > > >> > > > > > > > >> concern is > > > > > > > > > >> > > > > > > > >> >>>> that > > > > > > > > > >> > > > > > > > >> >>>> > >> we > > > > > > > > > >> > > > > > > > >> >>>> > >> > > will > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > have > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > to > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > add lots of new > > error > > > > > codes > > > > > > > for > > > > > > > > > >> each > > > > > > > > > >> > > > > possible > > > > > > > > > >> > > > > > > > >> error. Of > > > > > > > > > >> > > > > > > > >> >>>> course, > > > > > > > > > >> > > > > > > > >> >>>> > >> > we > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > could > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > reuse > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > some of existing > > like > > > > > > > > > >> > > > > > > UknownTopicOrPartitionCode, > > > > > > > > > >> > > > > > > > >> but we > > > > > > > > > >> > > > > > > > >> >>>> will > > > > > > > > > >> > > > > > > > >> >>>> > >> > also > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > need > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > to > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > add smth like: > > > > > > > > > >> TopicAlreadyExistsCode, > > > > > > > > > >> > > > > > > > >> >>>> TopicConfigInvalid (both > > > > > > > > > >> > > > > > > > >> >>>> > >> > for > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > topic > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > name and config, > > and > > > > > > probably > > > > > > > > > user > > > > > > > > > >> would > > > > > > > > > >> > > > > like > > > > > > > > > >> > > > > > to > > > > > > > > > >> > > > > > > > >> know > > > > > > > > > >> > > > > > > > >> >>>> what > > > > > > > > > >> > > > > > > > >> >>>> > >> > exactly > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > is wrong in his > > > > config), > > > > > > > > > >> > > > > > > InvalidReplicaAssignment, > > > > > > > > > >> > > > > > > > >> >>>> > >> InternalError > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > (e.g. > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > zookeeper > failure) > > etc. > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > And this is only > > for > > > > > > > > > TopicCommand, > > > > > > > > > >> we > > > > > > > > > >> > > will > > > > > > > > > >> > > > > > also > > > > > > > > > >> > > > > > > > >> need to > > > > > > > > > >> > > > > > > > >> >>>> add > > > > > > > > > >> > > > > > > > >> >>>> > >> > similar > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > stuff > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > for > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > ReassignPartitions, > > > > > > > > > >> PreferredReplica. So > > > > > > > > > >> > > > > we'll > > > > > > > > > >> > > > > > > end > > > > > > > > > >> > > > > > > > >> up > > > > > > > > > >> > > > > > > > >> >>>> with a > > > > > > > > > >> > > > > > > > >> >>>> > >> > large > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > list > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > of > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > error codes, used > > only > > > > in > > > > > > > Admin > > > > > > > > > >> > > protocol. > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Having said > that, I > > > > agree > > > > > > my > > > > > > > > > >> proposal is > > > > > > > > > >> > > > not > > > > > > > > > >> > > > > > > > >> consistent > > > > > > > > > >> > > > > > > > >> >>>> with > > > > > > > > > >> > > > > > > > >> >>>> > >> > other > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > cases. > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Maybe we can find > > > > better > > > > > > > solution > > > > > > > > > >> or > > > > > > > > > >> > > > > something > > > > > > > > > >> > > > > > > > >> >>>> in-between. > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Re Hangout chat: > I > > > > think > > > > > it > > > > > > > is a > > > > > > > > > >> great > > > > > > > > > >> > > > idea. > > > > > > > > > >> > > > > > > This > > > > > > > > > >> > > > > > > > >> way we > > > > > > > > > >> > > > > > > > >> >>>> can > > > > > > > > > >> > > > > > > > >> >>>> > >> move > > > > > > > > > >> > > > > > > > >> >>>> > >> > > on > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > faster. > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Let's agree > > somehow on > > > > > > > date/time > > > > > > > > > so > > > > > > > > > >> > > people > > > > > > > > > >> > > > > can > > > > > > > > > >> > > > > > > > join. > > > > > > > > > >> > > > > > > > >> >>>> Will work > > > > > > > > > >> > > > > > > > >> >>>> > >> > for > > > > > > > > > >> > > > > > > > >> >>>> > >> > > me > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > this > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > and > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > next week almost > > > > anytime > > > > > if > > > > > > > > > agreed > > > > > > > > > >> in > > > > > > > > > >> > > > > advance. > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Thanks, > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Andrii > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > On Wed, Feb 18, > > 2015 at > > > > > > 7:09 > > > > > > > PM, > > > > > > > > > >> Jay > > > > > > > > > >> > > > Kreps < > > > > > > > > > >> > > > > > > > >> >>>> > >> jay.kr...@gmail.com> > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > wrote: > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> Hey Andrii, > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> Generally we can > > do > > > > good > > > > > > > error > > > > > > > > > >> handling > > > > > > > > > >> > > > > > without > > > > > > > > > >> > > > > > > > >> needing > > > > > > > > > >> > > > > > > > >> >>>> custom > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > server-side > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> messages. I.e. > > > > generally > > > > > > the > > > > > > > > > >> client has > > > > > > > > > >> > > > the > > > > > > > > > >> > > > > > > > >> context to > > > > > > > > > >> > > > > > > > >> >>>> know > > > > > > > > > >> > > > > > > > >> >>>> > >> that > > > > > > > > > >> > > > > > > > >> >>>> > >> > > if > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > it > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > got > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> an error that > the > > > > topic > > > > > > > doesn't > > > > > > > > > >> exist > > > > > > > > > >> > > to > > > > > > > > > >> > > > > say > > > > > > > > > >> > > > > > > > >> "Topic X > > > > > > > > > >> > > > > > > > >> >>>> doesn't > > > > > > > > > >> > > > > > > > >> >>>> > >> > > exist" > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> rather > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> than "error code > > 14" > > > > (or > > > > > > > > > >> whatever). > > > > > > > > > >> > > Maybe > > > > > > > > > >> > > > > > there > > > > > > > > > >> > > > > > > > are > > > > > > > > > >> > > > > > > > >> >>>> specific > > > > > > > > > >> > > > > > > > >> >>>> > >> > cases > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > where > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> this is hard? If > > we > > > > want > > > > > > to > > > > > > > add > > > > > > > > > >> > > > server-side > > > > > > > > > >> > > > > > > error > > > > > > > > > >> > > > > > > > >> >>>> messages we > > > > > > > > > >> > > > > > > > >> >>>> > >> > > really > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > do > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> need to do this > > in a > > > > > > > consistent > > > > > > > > > >> way > > > > > > > > > >> > > > across > > > > > > > > > >> > > > > > the > > > > > > > > > >> > > > > > > > >> protocol. > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> I still have a > > bunch > > > > of > > > > > > open > > > > > > > > > >> questions > > > > > > > > > >> > > > here > > > > > > > > > >> > > > > > > from > > > > > > > > > >> > > > > > > > my > > > > > > > > > >> > > > > > > > >> >>>> previous > > > > > > > > > >> > > > > > > > >> >>>> > >> > > list. I > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > will > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> be out for the > > next > > > > few > > > > > > > days for > > > > > > > > > >> Strata > > > > > > > > > >> > > > > > though. > > > > > > > > > >> > > > > > > > >> Maybe > > > > > > > > > >> > > > > > > > >> >>>> we could > > > > > > > > > >> > > > > > > > >> >>>> > >> > do > > > > > > > > > >> > > > > > > > >> >>>> > >> > > a > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > Google > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> Hangout chat on > > any > > > > open > > > > > > > issues > > > > > > > > > >> some > > > > > > > > > >> > > time > > > > > > > > > >> > > > > > > towards > > > > > > > > > >> > > > > > > > >> the > > > > > > > > > >> > > > > > > > >> >>>> end of > > > > > > > > > >> > > > > > > > >> >>>> > >> > next > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > week > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > for > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> anyone inte > ... > > [Message clipped]
