Guys,
Thanks for a great discussion!
Here are the actions points:
1. Q: Get rid of all scala requests objects, use java protocol definitions.
A: Gwen kindly took that (KAFKA-1927). It's important to speed up
review procedure
there since this ticket blocks other important changes.
2. Q: Generic re-reroute facility vs client maintaining cluster state.
A: Jay has added pseudo code to KAFKA-1912 - need to consider whether
this will be
easy to implement as a server-side feature (comments are welcomed!).
3. Q: Controller field in wire protocol.
A: This might be useful for clients, add this to TopicMetadataResponse
(already in KIP).
4. Q: Decoupling topic creation from TMR.
A: I will add proposed by Jun solution (using clientId for that) to the
KIP.
5. Q: Bumping new versions of TMR vs grabbing all protocol changes in one
version.
A: It was decided to try to gather all changes to protocol (before
release).
In case of TMR it worth checking: KAFKA-2020 and KIP-13 (quotas)
6. Q: JSON lib is needed to deserialize user's input in CLI tool.
A: Use jackson for that, /tools project is a separate jar so shouldn't
be a big deal.
7. Q: VerifyReassingPartitions vs generic status check command.
A: For long-running requests like reassign partitions *progress* check
request is useful,
it makes sense to introduce it.
Please add, correct me if I missed something.
Thanks,
Andrii Biletskyi
On Tue, Mar 17, 2015 at 6:20 PM, Andrii Biletskyi <
andrii.bilets...@stealth.ly> wrote:
> Joel,
>
> You are right, I removed ClusterMetadata because we have partially
> what we need in TopicMetadata. Also, as Jay pointed out earlier, we
> would like to have "orthogonal" API, but at the same time we need
> to be backward compatible.
>
> But I like your idea and even have some other arguments for this option:
> There is also DescribeTopicRequest which was proposed in this KIP,
> it returns topic configs, partitions, replication factor plus partition
> ISR, ASR,
> leader replica. The later part is really already there in
> TopicMetadataRequest.
> So again we'll have to add stuff to TMR, not to duplicate some info in
> newly added requests. However, this way we'll end up with "monster"
> request which returns cluster metadata, topic replication and config info
> plus partition replication data. Seems logical to split TMR to
> - ClusterMetadata (brokers + controller, maybe smth else)
> - TopicMetadata (topic info + partition details)
> But since current TMR is involved in lots of places (including network
> client,
> as I understand) this might be very serious change and it probably makes
> sense to stick with current approach.
>
> Thanks,
> Andrii Biletskyi
>
>
> On Tue, Mar 17, 2015 at 5:29 PM, Joel Koshy <jjkosh...@gmail.com> wrote:
>
>> I may be missing some context but hopefully this will also be covered
>> today: I thought the earlier proposal where there was an explicit
>> ClusterMetadata request was clearer and explicit. During the course of
>> this thread I think the conclusion was that the main need was for
>> controller information and that can be rolled into the topic metadata
>> response but that seems a bit irrelevant to topic metadata. FWIW I
>> think the full broker-list is also irrelevant to topic metadata, but
>> it is already there and in use. I think there is still room for an
>> explicit ClusterMetadata request since there may be other
>> cluster-level information that we may want to add over time (and that
>> have nothing to do with topic metadata).
>>
>> On Tue, Mar 17, 2015 at 02:45:30PM +0200, Andrii Biletskyi wrote:
>> > Jun,
>> >
>> > 101. Okay, if you say that such use case is important. I also think
>> > using clientId for these purposes is fine - if we already have this
>> field
>> > as part of all Wire protocol messages, why not use that.
>> > I will update KIP-4 page if nobody has other ideas (which may come up
>> > during the call today).
>> >
>> > 102.1 Agree, I'll update the KIP accordingly. I think we can add new,
>> > fine-grained error codes if some error code received in specific case
>> > won't give enough context to return a descriptive error message for
>> user.
>> >
>> > Look forward to discussing all outstanding issues in detail today during
>> > the call.
>> >
>> > Thanks,
>> > Andrii Biletskyi
>> >
>> >
>> >
>> > On Mon, Mar 16, 2015 at 10:59 PM, Jun Rao <j...@confluent.io> wrote:
>> >
>> > > 101. There may be a use case where you only want the topics to be
>> created
>> > > manually by admins. Currently, you can do that by disabling auto topic
>> > > creation and issue topic creation from the TopicCommand. If we
>> disable auto
>> > > topic creation completely on the broker and don't have a way to
>> distinguish
>> > > between topic creation requests from the regular clients and the
>> admin, we
>> > > can't support manual topic creation any more. I was thinking that
>> another
>> > > way of distinguishing the clients making the topic creation requests
>> is
>> > > using clientId. For example, the admin tool can set it to something
>> like
>> > > admin and the broker can treat that clientId specially.
>> > >
>> > > Also, there is a related discussion in KAFKA-2020. Currently, we do
>> the
>> > > following in TopicMetadataResponse:
>> > >
>> > > 1. If leader is not available, we set the partition level error code
>> to
>> > > LeaderNotAvailable.
>> > > 2. If a non-leader replica is not available, we take that replica out
>> of
>> > > the assigned replica list and isr in the response. As an indication
>> for
>> > > doing that, we set the partition level error code to
>> ReplicaNotAvailable.
>> > >
>> > > This has a few problems. First, ReplicaNotAvailable probably
>> shouldn't be
>> > > an error, at least for the normal producer/consumer clients that just
>> want
>> > > to find out the leader. Second, it can happen that both the leader and
>> > > another replica are not available at the same time. There is no error
>> code
>> > > to indicate both. Third, even if a replica is not available, it's
>> still
>> > > useful to return its replica id since some clients (e.g. admin tool)
>> may
>> > > still make use of it.
>> > >
>> > > One way to address this issue is to always return the replica id for
>> > > leader, assigned replicas, and isr regardless of whether the
>> corresponding
>> > > broker is live or not. Since we also return the list of live brokers,
>> the
>> > > client can figure out whether a leader or a replica is live or not
>> and act
>> > > accordingly. This way, we don't need to set the partition level error
>> code
>> > > when the leader or a replica is not available. This doesn't change
>> the wire
>> > > protocol, but does change the semantics. Since we are evolving the
>> protocol
>> > > of TopicMetadataRequest here, we can potentially piggyback the change.
>> > >
>> > > 102.1 For those types of errors due to invalid input, shouldn't we
>> just
>> > > guard it at parameter validation time and throw
>> InvalidArgumentException
>> > > without even sending the request to the broker?
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > >
>> > > On Mon, Mar 16, 2015 at 10:37 AM, Andrii Biletskyi <
>> > > andrii.bilets...@stealth.ly> wrote:
>> > >
>> > > > Jun,
>> > > >
>> > > > Answering your questions:
>> > > >
>> > > > 101. If I understand you correctly, you are saying future producer
>> > > versions
>> > > > (which
>> > > > will be ported to TMR_V1) won't be able to automatically create
>> topic (if
>> > > > we
>> > > > unconditionally remove topic creation from there). But we need to
>> this
>> > > > preserve logic.
>> > > > Ok, about your proposal: I'm not a big fan too, when it comes to
>> > > > differentiating
>> > > > clients directly in protocol schema. And also I'm not sure I
>> understand
>> > > at
>> > > > all why
>> > > > auto.create.topics.enable is a server side configuration. Can we
>> > > deprecate
>> > > > this setting
>> > > > in future versions, add this setting to producer and based on that
>> upon
>> > > > receiving
>> > > > UnknownTopic create topic explicitly by a separate producer call via
>> > > > adminClient?
>> > > >
>> > > > 102.1. Hm, yes. It's because we want to support batching and at the
>> same
>> > > > time we
>> > > > want to give descriptive error messages for clients. Since
>> AdminClient
>> > > > holds the context
>> > > > to construct such messages (e.g. AdminClient layer can know that
>> > > > InvalidArgumentsCode
>> > > > means two cases: either invalid number - e.g. -1; or
>> replication-factor
>> > > was
>> > > > provided while
>> > > > partitions argument wasn't) - I wrapped responses in Exceptions.
>> But I'm
>> > > > open to any
>> > > > other ideas, this was just initial version.
>> > > > 102.2. Yes, I agree. I'll change that to probably some other dto.
>> > > >
>> > > > Thanks,
>> > > > Andrii Biletskyi
>> > > >
>> > > > On Fri, Mar 13, 2015 at 7:16 PM, Jun Rao <j...@confluent.io> wrote:
>> > > >
>> > > > > Andrii,
>> > > > >
>> > > > > 101. That's what I was thinking too, but it may not be that
>> simple. In
>> > > > > TopicMetadataRequest_V1,
>> > > > > we can let it not trigger auto topic creation. Then, in the
>> producer
>> > > > side,
>> > > > > if it gets an UnknownTopicException, it can explicitly issue a
>> > > > > createTopicRequest for auto topic creation. On the consumer side,
>> it
>> > > will
>> > > > > never issue createTopicRequest. This works when auto topic
>> creation is
>> > > > > enabled on the broker side. However, I am not sure how things
>> will work
>> > > > > when auto topic creation is disabled on the broker side. In this
>> case,
>> > > we
>> > > > > want to have a way to manually create a topic, potentially through
>> > > admin
>> > > > > commands. However, then we need a way to distinguish
>> createTopicRequest
>> > > > > issued from the producer clients and the admin tools. May be we
>> can
>> > > add a
>> > > > > new field in createTopicRequest and set it differently in the
>> producer
>> > > > > client and the admin client. However, I am not sure if that's the
>> best
>> > > > > approach.
>> > > > >
>> > > > > 2. Yes, refactoring existing requests is a non-trivial amount of
>> work.
>> > > I
>> > > > > posted some comments in KAFKA-1927. We will probably have to fix
>> > > > KAFKA-1927
>> > > > > first, before adding the new logic in KAFKA-1694. Otherwise, the
>> > > changes
>> > > > > will be too big.
>> > > > >
>> > > > > 102. About the AdminClient:
>> > > > > 102.1. It's a bit weird that we return exception in the api. It
>> seems
>> > > > that
>> > > > > we should either return error code or throw an exception when
>> getting
>> > > the
>> > > > > response state.
>> > > > > 102.2. We probably shouldn't explicitly use the request object in
>> the
>> > > > api.
>> > > > > Not every request evolution requires an api change.
>> > > > >
>> > > > > Thanks,
>> > > > >
>> > > > > Jun
>> > > > >
>> > > > >
>> > > > > On Fri, Mar 13, 2015 at 4:08 AM, Andrii Biletskyi <
>> > > > > andrii.bilets...@stealth.ly> wrote:
>> > > > >
>> > > > > > Jun,
>> > > > > >
>> > > > > > Thanks for you comments. Answers inline:
>> > > > > >
>> > > > > > 100. There are a few fields such as ReplicaAssignment,
>> > > > > > > ReassignPartitionRequest,
>> > > > > > > and PartitionsSerialized that are represented as a string, but
>> > > > contain
>> > > > > > > composite structures in json. Could we flatten them out
>> directly in
>> > > > the
>> > > > > > > protocol definition as arrays/records?
>> > > > > >
>> > > > > >
>> > > > > > Yes, now with Admin Client this looks a bit weird. My initial
>> > > > motivation
>> > > > > > was:
>> > > > > > ReassignPartitionCommand accepts input in json, we want to
>> remain
>> > > > tools'
>> > > > > > interfaces unchanged, where possible.
>> > > > > > If we port it to deserialized format, in CLI (/tools project)
>> we will
>> > > > > have
>> > > > > > to add some
>> > > > > > json library since /tools is written in java and we'll need to
>> > > > > deserialize
>> > > > > > json file
>> > > > > > provided by a user. Can we quickly agree on what this library
>> should
>> > > be
>> > > > > > (Jackson, GSON, whatever)?
>> > > > > >
>> > > > > > 101. Does TopicMetadataRequest v1 still trigger auto topic
>> creation?
>> > > > This
>> > > > > > > will be a bit weird now that we have a separate topic
>> creation api.
>> > > > > Have
>> > > > > > > you thought about how the new createTopicRequest and
>> > > > > TopicMetadataRequest
>> > > > > > > v1 will be used in the producer/consumer client, in addition
>> to
>> > > admin
>> > > > > > > tools? For example, ideally, we don't want
>> TopicMetadataRequest
>> > > from
>> > > > > the
>> > > > > > > consumer to trigger auto topic creation.
>> > > > > >
>> > > > > >
>> > > > > > I agree, this strange logic should be fixed. I'm not confident
>> in
>> > > this
>> > > > > > Kafka part so
>> > > > > > correct me if I'm wrong, but it doesn't look like a hard thing
>> to
>> > > do, I
>> > > > > > think we can
>> > > > > > leverage AdminClient for that in Producer and unconditionally
>> remove
>> > > > > topic
>> > > > > > creation from the TopicMetadataRequest_V1.
>> > > > > >
>> > > > > > 2. I think Jay meant getting rid of scala classes
>> > > > > > > like HeartbeatRequestAndHeader and
>> HeartbeatResponseAndHeader. We
>> > > did
>> > > > > > that
>> > > > > > > as a stop-gap thing when adding the new requests for the
>> consumers.
>> > > > > > > However, the long term plan is to get rid of all those and
>> just
>> > > reuse
>> > > > > the
>> > > > > > > java request/response in the client. Since this KIP proposes
>> to
>> > > add a
>> > > > > > > significant number of new requests, perhaps we should bite the
>> > > bullet
>> > > > > to
>> > > > > > > clean up the existing scala requests first before adding new
>> ones?
>> > > > > > >
>> > > > > >
>> > > > > > Yes, looks like I misunderstood the point of
>> ...RequestAndHeader.
>> > > > Okay, I
>> > > > > > will
>> > > > > > rework that. The only thing is that I don't see any example how
>> it
>> > > was
>> > > > > done
>> > > > > > for at
>> > > > > > least one existing protocol message. Thus, as I understand, I
>> have to
>> > > > > think
>> > > > > > how we
>> > > > > > are going to do it.
>> > > > > > Re porting all existing RQ/RP in this patch. Sounds reasonable,
>> but
>> > > if
>> > > > > it's
>> > > > > > an *obligatory*
>> > > > > > requirement to have Admin KIP done, I'm afraid this can be a
>> serious
>> > > > > > blocker for us.
>> > > > > > There are 13 protocol messages and all that would require not
>> only
>> > > unit
>> > > > > > tests but quite
>> > > > > > intensive manual testing, no? I'm afraid I'm not the right guy
>> to
>> > > cover
>> > > > > > pretty much all
>> > > > > > Kafka core internals :). Let me know your thoughts on this
>> item. Btw
>> > > > > there
>> > > > > > is a ticket to
>> > > > > > follow-up this issue (
>> > > https://issues.apache.org/jira/browse/KAFKA-2006
>> > > > ).
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Andrii Biletskyi
>> > > > > >
>> > > > > >
>> > > > > > On Fri, Mar 13, 2015 at 6:40 AM, Jun Rao <j...@confluent.io>
>> wrote:
>> > > > > >
>> > > > > > > Andrii,
>> > > > > > >
>> > > > > > >
>> > > > > > > A few more comments.
>> > > > > > >
>> > > > > > > 100. There are a few fields such as ReplicaAssignment,
>> > > > > > > ReassignPartitionRequest,
>> > > > > > > and PartitionsSerialized that are represented as a string, but
>> > > > contain
>> > > > > > > composite structures in json. Could we flatten them out
>> directly in
>> > > > the
>> > > > > > > protocol definition as arrays/records?
>> > > > > > >
>> > > > > > > 101. Does TopicMetadataRequest v1 still trigger auto topic
>> > > creation?
>> > > > > This
>> > > > > > > will be a bit weird now that we have a separate topic
>> creation api.
>> > > > > Have
>> > > > > > > you thought about how the new createTopicRequest and
>> > > > > TopicMetadataRequest
>> > > > > > > v1 will be used in the producer/consumer client, in addition
>> to
>> > > admin
>> > > > > > > tools? For example, ideally, we don't want
>> TopicMetadataRequest
>> > > from
>> > > > > the
>> > > > > > > consumer to trigger auto topic creation.
>> > > > > > >
>> > > > > > > 2. I think Jay meant getting rid of scala classes
>> > > > > > > like HeartbeatRequestAndHeader and
>> HeartbeatResponseAndHeader. We
>> > > did
>> > > > > > that
>> > > > > > > as a stop-gap thing when adding the new requests for the
>> consumers.
>> > > > > > > However, the long term plan is to get rid of all those and
>> just
>> > > reuse
>> > > > > the
>> > > > > > > java request/response in the client. Since this KIP proposes
>> to
>> > > add a
>> > > > > > > significant number of new requests, perhaps we should bite the
>> > > bullet
>> > > > > to
>> > > > > > > clean up the existing scala requests first before adding new
>> ones?
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > >
>> > > > > > > Jun
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > On Thu, Mar 12, 2015 at 3:37 PM, Andrii Biletskyi <
>> > > > > > > andrii.bilets...@stealth.ly> wrote:
>> > > > > > >
>> > > > > > > > Hi,
>> > > > > > > >
>> > > > > > > > As said above - I list again all comments from this thread
>> so we
>> > > > > > > > can see what's left and finalize all pending issues.
>> > > > > > > >
>> > > > > > > > Comments from Jay:
>> > > > > > > > 1. This is much needed functionality, but there are a lot
>> of the
>> > > so
>> > > > > > let's
>> > > > > > > > really think these protocols through. We really want to end
>> up
>> > > > with a
>> > > > > > set
>> > > > > > > > of well thought-out, orthoganol apis. For this reason I
>> think it
>> > > is
>> > > > > > > really
>> > > > > > > > important to think through the end state even if that
>> includes
>> > > APIs
>> > > > > we
>> > > > > > > > won't implement in the first phase.
>> > > > > > > >
>> > > > > > > > A: Definitely behind this. Would appreciate if there are
>> concrete
>> > > > > > > comments
>> > > > > > > > how this can be improved.
>> > > > > > > >
>> > > > > > > > 2. Let's please please please wait until we have switched
>> the
>> > > > server
>> > > > > > over
>> > > > > > > > to the new java protocol definitions. If we add upteen more
>> ad
>> > > hoc
>> > > > > > scala
>> > > > > > > > objects that is just generating more work for the
>> conversion we
>> > > > know
>> > > > > we
>> > > > > > > > have to do.
>> > > > > > > >
>> > > > > > > > A: Fixed in the latest patch - removed scala protocol
>> classes.
>> > > > > > > >
>> > > > > > > > 3. This proposal introduces a new type of optional
>> parameter.
>> > > This
>> > > > is
>> > > > > > > > inconsistent with everything else in the protocol where we
>> use -1
>> > > > or
>> > > > > > some
>> > > > > > > > other marker value. You could argue either way but let's
>> stick
>> > > with
>> > > > > > that
>> > > > > > > > for consistency. For clients that implemented the protocol
>> in a
>> > > > > better
>> > > > > > > way
>> > > > > > > > than our scala code these basic primitives are hard to
>> change.
>> > > > > > > >
>> > > > > > > > A: Fixed in the latest patch - removed MaybeOf type and
>> changed
>> > > > > > protocol
>> > > > > > > > accordingly.
>> > > > > > > >
>> > > > > > > > 4. ClusterMetadata: This seems to duplicate
>> TopicMetadataRequest
>> > > > > which
>> > > > > > > has
>> > > > > > > > brokers, topics, and partitions. I think we should rename
>> that
>> > > > > request
>> > > > > > > > ClusterMetadataRequest (or just MetadataRequest) and
>> include the
>> > > id
>> > > > > of
>> > > > > > > the
>> > > > > > > > controller. Or are there other things we could add here?
>> > > > > > > >
>> > > > > > > > A: I agree. Updated the KIP. Let's extends TopicMetadata to
>> > > > version 2
>> > > > > > and
>> > > > > > > > include controller.
>> > > > > > > >
>> > > > > > > > 5. We have a tendency to try to make a lot of requests that
>> can
>> > > > only
>> > > > > go
>> > > > > > > to
>> > > > > > > > particular nodes. This adds a lot of burden for client
>> > > > > implementations
>> > > > > > > (it
>> > > > > > > > sounds easy but each discovery can fail in many parts so it
>> ends
>> > > up
>> > > > > > > being a
>> > > > > > > > full state machine to do right). I think we should consider
>> > > making
>> > > > > > admin
>> > > > > > > > commands and ideally as many of the other apis as possible
>> > > > available
>> > > > > on
>> > > > > > > all
>> > > > > > > > brokers and just redirect to the controller on the broker
>> side.
>> > > > > Perhaps
>> > > > > > > > there would be a general way to encapsulate this re-routing
>> > > > behavior.
>> > > > > > > >
>> > > > > > > > A: It's a very interesting idea, but seems there are some
>> > > concerns
>> > > > > > about
>> > > > > > > > this
>> > > > > > > > feature (like performance considerations, how this will
>> > > complicate
>> > > > > > server
>> > > > > > > > etc).
>> > > > > > > > I believe this shouldn't be a blocker. If this feature is
>> > > > implemented
>> > > > > > at
>> > > > > > > > some
>> > > > > > > > point it won't affect Admin changes - at least no changes to
>> > > public
>> > > > > API
>> > > > > > > > will be required.
>> > > > > > > >
>> > > > > > > > 6. We should probably normalize the key value pairs used for
>> > > > configs
>> > > > > > > rather
>> > > > > > > > than embedding a new formatting. So two strings rather than
>> one
>> > > > with
>> > > > > an
>> > > > > > > > internal equals sign.
>> > > > > > > >
>> > > > > > > > A: Fixed in the latest patch - normalized configs and
>> changed
>> > > > > protocol
>> > > > > > > > accordingly.
>> > > > > > > >
>> > > > > > > > 7. Is the postcondition of these APIs that the command has
>> begun
>> > > or
>> > > > > > that
>> > > > > > > > the command has been completed? It is a lot more usable if
>> the
>> > > > > command
>> > > > > > > has
>> > > > > > > > been completed so you know that if you create a topic and
>> then
>> > > > > publish
>> > > > > > to
>> > > > > > > > it you won't get an exception about there being no such
>> topic.
>> > > > > > > >
>> > > > > > > > A: For long running requests (like reassign partitions) -
>> the
>> > > post
>> > > > > > > > condition is
>> > > > > > > > command has begun - so we don't block the client. In case
>> of your
>> > > > > > > example -
>> > > > > > > > topic commands, this will be refactored and topic commands
>> will
>> > > be
>> > > > > > > executed
>> > > > > > > > immediately, since the Controller will serve Admin requests
>> > > > > > > > (follow-up ticket KAFKA-1777).
>> > > > > > > >
>> > > > > > > > 8. Describe topic and list topics duplicate a lot of stuff
>> in the
>> > > > > > > metadata
>> > > > > > > > request. Is there a reason to give back topics marked for
>> > > > deletion? I
>> > > > > > > feel
>> > > > > > > > like if we just make the post-condition of the delete
>> command be
>> > > > that
>> > > > > > the
>> > > > > > > > topic is deleted that will get rid of the need for this
>> right?
>> > > And
>> > > > it
>> > > > > > > will
>> > > > > > > > be much more intuitive.
>> > > > > > > >
>> > > > > > > > A: Fixed in the latest patch - removed topics marked for
>> deletion
>> > > > in
>> > > > > > > > ListTopicsRequest.
>> > > > > > > >
>> > > > > > > > 9. Should we consider batching these requests? We have
>> generally
>> > > > > tried
>> > > > > > to
>> > > > > > > > allow multiple operations to be batched. My suspicion is
>> that
>> > > > without
>> > > > > > > this
>> > > > > > > > we will get a lot of code that does something like
>> > > > > > > > for(topic: adminClient.listTopics())
>> > > > > > > > adminClient.describeTopic(topic)
>> > > > > > > > this code will work great when you test on 5 topics but not
>> do as
>> > > > > well
>> > > > > > if
>> > > > > > > > you have 50k.
>> > > > > > > >
>> > > > > > > > A: Updated the KIP - please check "Topic Admin Schema"
>> section.
>> > > > > > > >
>> > > > > > > > 10. I think we should also discuss how we want to expose a
>> > > > > programmatic
>> > > > > > > JVM
>> > > > > > > > client api for these operations. Currently people rely on
>> > > > AdminUtils
>> > > > > > > which
>> > > > > > > > is totally sketchy. I think we probably need another client
>> under
>> > > > > > > clients/
>> > > > > > > > that exposes administrative functionality. We will need
>> this just
>> > > > to
>> > > > > > > > properly test the new apis, I suspect. We should figure out
>> that
>> > > > API.
>> > > > > > > >
>> > > > > > > > A: Updated the KIP - please check "Admin Client" section
>> with an
>> > > > > > initial
>> > > > > > > > API proposal.
>> > > > > > > >
>> > > > > > > > 11. The other information that would be really useful to get
>> > > would
>> > > > be
>> > > > > > > > information about partitions--how much data is in the
>> partition,
>> > > > what
>> > > > > > are
>> > > > > > > > the segment offsets, what is the log-end offset (i.e. last
>> > > offset),
>> > > > > > what
>> > > > > > > is
>> > > > > > > > the compaction point, etc. I think that done right this
>> would be
>> > > > the
>> > > > > > > > successor to the very awkward OffsetRequest we have today.
>> > > > > > > >
>> > > > > > > > A: I removed ConsumerGroupOffsetsRequest in the latest
>> patch. I
>> > > > > believe
>> > > > > > > > this should
>> > > > > > > > be resolved in a separate KIP / jira ticket.
>> > > > > > > >
>> > > > > > > > 12. Generally we can do good error handling without needing
>> > > custom
>> > > > > > > > server-side
>> > > > > > > > messages. I.e. generally the client has the context to know
>> that
>> > > if
>> > > > > it
>> > > > > > > got
>> > > > > > > > an error that the topic doesn't exist to say "Topic X
>> doesn't
>> > > > exist"
>> > > > > > > rather
>> > > > > > > > than "error code 14" (or whatever). Maybe there are specific
>> > > cases
>> > > > > > where
>> > > > > > > > this is hard? If we want to add server-side error messages
>> we
>> > > > really
>> > > > > do
>> > > > > > > > need to do this in a consistent way across the protocol.
>> > > > > > > >
>> > > > > > > > A: Updated the KIP - please check "Protocol Errors"
>> section. I
>> > > > added
>> > > > > > the
>> > > > > > > > comprehensive, fine-grained list of error codes.
>> > > > > > > >
>> > > > > > > > Comments from Guozhang:
>> > > > > > > > 13. Describe topic request: it would be great to go beyond
>> just
>> > > > > > batching
>> > > > > > > on
>> > > > > > > > topic name regex for this request. For example, a very
>> common use
>> > > > > case
>> > > > > > of
>> > > > > > > > the topic command is to list all topics whose config A's
>> value is
>> > > > B.
>> > > > > > With
>> > > > > > > > topic name regex then we have to first retrieve __all__
>> topics's
>> > > > > > > > description info and then filter at the client end, which
>> will
>> > > be a
>> > > > > > huge
>> > > > > > > > burden on ZK.
>> > > > > > > > AND
>> > > > > > > > 14. Config K-Vs in create topic: this is related to the
>> previous
>> > > > > point;
>> > > > > > > > maybe we can add another metadata K-V or just a metadata
>> string
>> > > > along
>> > > > > > > side
>> > > > > > > > with config K-V in create topic like we did for offset
>> commit
>> > > > > request.
>> > > > > > > This
>> > > > > > > > field can be quite useful in storing information like
>> "owner" of
>> > > > the
>> > > > > > > topic
>> > > > > > > > who issue the create command, etc, which is quite important
>> for a
>> > > > > > > > multi-tenant setting. Then in the describe topic request we
>> can
>> > > > also
>> > > > > > > batch
>> > > > > > > > on regex of the metadata field.
>> > > > > > > >
>> > > > > > > > A: As discussed it is very interesting but can be
>> implemented
>> > > later
>> > > > > > after
>> > > > > > > > we have some basic functionality there.
>> > > > > > > >
>> > > > > > > > 15. Today all the admin operations are async in the sense
>> that
>> > > > > command
>> > > > > > > will
>> > > > > > > > return once it is written in ZK, and that is why we need
>> extra
>> > > > > > > verification
>> > > > > > > > like testUtil.waitForTopicCreated() / verify partition
>> > > reassignment
>> > > > > > > > request, etc. With admin requests we could add a flag to
>> enable /
>> > > > > > disable
>> > > > > > > > synchronous requests; when it is turned on, the response
>> will not
>> > > > > > return
>> > > > > > > > until the request has been completed. And for async
>> requests we
>> > > can
>> > > > > > add a
>> > > > > > > > "token" field in the response, and then only need a general
>> > > "admin
>> > > > > > > > verification request" with the given token to check if the
>> async
>> > > > > > request
>> > > > > > > > has been completed.
>> > > > > > > >
>> > > > > > > > A: I see your point. My idea was to provide specific
>> > > > Verify...Request
>> > > > > > per
>> > > > > > > > each
>> > > > > > > > long running request, where needed. We can do it the way you
>> > > > suggest.
>> > > > > > The
>> > > > > > > > only
>> > > > > > > > concern is that introducing a token we again will make
>> schema
>> > > > > > "dynamic".
>> > > > > > > We
>> > > > > > > > wanted
>> > > > > > > > to do similar thing introducing single AdminRequest for all
>> topic
>> > > > > > > commands
>> > > > > > > > but rejected
>> > > > > > > > this idea because we wanted to have schema defined. So this
>> is
>> > > > more a
>> > > > > > > > choice between:
>> > > > > > > > a) have fixed schema but introduce each time new
>> Verify...Request
>> > > > for
>> > > > > > > > long-running requests
>> > > > > > > > b) use one request for verification but generalize it with
>> token
>> > > > > > > > I'm fine with whatever decision community come to. Just let
>> me
>> > > know
>> > > > > > your
>> > > > > > > > thoughts.
>> > > > > > > >
>> > > > > > > > Comment from Gwen:
>> > > > > > > > 16. Specifically for ownership, I think the plan is to add
>> ACL
>> > > (it
>> > > > > > sounds
>> > > > > > > > like you are describing ACL) via an external system (Argus,
>> > > > Sentry).
>> > > > > > > > I remember KIP-11 described this, but I can't find the KIP
>> any
>> > > > > longer.
>> > > > > > > >
>> > > > > > > > A: Okay, no problem. Not sure though how we are going to
>> handle
>> > > it.
>> > > > > > Wait
>> > > > > > > > which KIP
>> > > > > > > > will be committed first and include changes to
>> TopicMetadata from
>> > > > the
>> > > > > > > later
>> > > > > > > > one?
>> > > > > > > > Anyway, I added this note to "Open Questions" section so we
>> don't
>> > > > > miss
>> > > > > > > this
>> > > > > > > > piece.
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > > Andrii Biletskyi
>> > > > > > > >
>> > > > > > > > On Fri, Mar 13, 2015 at 12:34 AM, Andrii Biletskyi <
>> > > > > > > > andrii.bilets...@stealth.ly> wrote:
>> > > > > > > >
>> > > > > > > > > Hi all,
>> > > > > > > > >
>> > > > > > > > > Today I uploaded the patch that covers some of the
>> discussed
>> > > and
>> > > > > > agreed
>> > > > > > > > > items:
>> > > > > > > > > - removed MaybeOf optional type
>> > > > > > > > > - switched to java protocol definitions
>> > > > > > > > > - simplified messages (normalized configs, removed topic
>> marked
>> > > > for
>> > > > > > > > > deletion)
>> > > > > > > > >
>> > > > > > > > > I also updated the KIP-4 with respective changes and
>> wrote down
>> > > > my
>> > > > > > > > > proposal for
>> > > > > > > > > pending items:
>> > > > > > > > > - Batch Admin Operations -> updated Wire Protocol schema
>> > > proposal
>> > > > > > > > > - Remove ClusterMetadata -> changed to extend
>> > > > TopicMetadataRequest
>> > > > > > > > > - Admin Client -> updated my initial proposal to reflect
>> > > batching
>> > > > > > > > > - Error codes -> proposed fine-grained error code instead
>> of
>> > > > > > > > > AdminRequestFailed
>> > > > > > > > >
>> > > > > > > > > I will also send a separate email to cover all comments
>> from
>> > > this
>> > > > > > > thread.
>> > > > > > > > >
>> > > > > > > > > Thanks,
>> > > > > > > > > Andrii Biletskyi
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Thu, Mar 12, 2015 at 9:26 PM, Gwen Shapira <
>> > > > > gshap...@cloudera.com
>> > > > > > >
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > >> Found KIP-11 (
>> > > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface
>> > > > > > > > >> )
>> > > > > > > > >> It actually specifies changes to the Metadata protocol,
>> so
>> > > > making
>> > > > > > sure
>> > > > > > > > >> both KIPs are consistent in this regard will be good.
>> > > > > > > > >>
>> > > > > > > > >> On Thu, Mar 12, 2015 at 12:21 PM, Gwen Shapira <
>> > > > > > gshap...@cloudera.com
>> > > > > > > >
>> > > > > > > > >> wrote:
>> > > > > > > > >> > Specifically for ownership, I think the plan is to add
>> ACL
>> > > (it
>> > > > > > > sounds
>> > > > > > > > >> > like you are describing ACL) via an external system
>> (Argus,
>> > > > > > Sentry).
>> > > > > > > > >> > I remember KIP-11 described this, but I can't find the
>> KIP
>> > > any
>> > > > > > > longer.
>> > > > > > > > >> >
>> > > > > > > > >> > Regardless, I think KIP-4 focuses on getting
>> information
>> > > that
>> > > > > > > already
>> > > > > > > > >> > exists from Kafka brokers, not on adding information
>> that
>> > > > > perhaps
>> > > > > > > > >> > should exist but doesn't yet?
>> > > > > > > > >> >
>> > > > > > > > >> > Gwen
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >> > On Thu, Mar 12, 2015 at 6:37 AM, Guozhang Wang <
>> > > > > > wangg...@gmail.com>
>> > > > > > > > >> wrote:
>> > > > > > > > >> >> Folks,
>> > > > > > > > >> >>
>> > > > > > > > >> >> Just want to elaborate a bit more on the create-topic
>> > > > metadata
>> > > > > > and
>> > > > > > > > >> batching
>> > > > > > > > >> >> describe-topic based on config / metadata in my
>> previous
>> > > > email
>> > > > > as
>> > > > > > > we
>> > > > > > > > >> work
>> > > > > > > > >> >> on KAFKA-1694. The main motivation is to have some
>> sort of
>> > > > > topic
>> > > > > > > > >> management
>> > > > > > > > >> >> mechanisms, which I think is quite important in a
>> > > > multi-tenant
>> > > > > /
>> > > > > > > > cloud
>> > > > > > > > >> >> architecture: today anyone can create topics in a
>> shared
>> > > > Kafka
>> > > > > > > > >> cluster, but
>> > > > > > > > >> >> there is no concept or "ownership" of topics that are
>> > > created
>> > > > > by
>> > > > > > > > >> different
>> > > > > > > > >> >> users. For example, at LinkedIn we basically
>> distinguish
>> > > > topic
>> > > > > > > owners
>> > > > > > > > >> via
>> > > > > > > > >> >> some casual topic name prefix, which is a bit awkward
>> and
>> > > > does
>> > > > > > not
>> > > > > > > > fly
>> > > > > > > > >> as
>> > > > > > > > >> >> we scale our customers. It would be great to use
>> > > > > describe-topics
>> > > > > > > such
>> > > > > > > > >> as:
>> > > > > > > > >> >>
>> > > > > > > > >> >> Describe all topics that is created by me.
>> > > > > > > > >> >>
>> > > > > > > > >> >> Describe all topics whose retention time is overriden
>> to X.
>> > > > > > > > >> >>
>> > > > > > > > >> >> Describe all topics whose writable group include user
>> Y
>> > > (this
>> > > > > is
>> > > > > > > > >> related to
>> > > > > > > > >> >> authorization), etc..
>> > > > > > > > >> >>
>> > > > > > > > >> >> One possible way to achieve this is to add a metadata
>> file
>> > > in
>> > > > > the
>> > > > > > > > >> >> create-topic request, whose value will also be
>> written ZK
>> > > as
>> > > > we
>> > > > > > > > create
>> > > > > > > > >> the
>> > > > > > > > >> >> topic; then describe-topics can choose to batch topics
>> > > based
>> > > > on
>> > > > > > 1)
>> > > > > > > > name
>> > > > > > > > >> >> regex, 2) config K-V matching, 3) metadata regex, etc.
>> > > > > > > > >> >>
>> > > > > > > > >> >> Thoughts?
>> > > > > > > > >> >>
>> > > > > > > > >> >> Guozhang
>> > > > > > > > >> >>
>> > > > > > > > >> >> On Thu, Mar 5, 2015 at 4:37 PM, Guozhang Wang <
>> > > > > > wangg...@gmail.com>
>> > > > > > > > >> wrote:
>> > > > > > > > >> >>
>> > > > > > > > >> >>> Thanks for the updated wiki. A few comments below:
>> > > > > > > > >> >>>
>> > > > > > > > >> >>> 1. Error description in response: I think if some
>> > > errorCode
>> > > > > > could
>> > > > > > > > >> indicate
>> > > > > > > > >> >>> several different error cases then we should really
>> change
>> > > > it
>> > > > > to
>> > > > > > > > >> multiple
>> > > > > > > > >> >>> codes. In general the errorCode itself would be
>> precise
>> > > and
>> > > > > > > > >> sufficient for
>> > > > > > > > >> >>> describing the server side errors.
>> > > > > > > > >> >>>
>> > > > > > > > >> >>> 2. Describe topic request: it would be great to go
>> beyond
>> > > > just
>> > > > > > > > >> batching on
>> > > > > > > > >> >>> topic name regex for this request. For example, a
>> very
>> > > > common
>> > > > > > use
>> > > > > > > > >> case of
>> > > > > > > > >> >>> the topic command is to list all topics whose config
>> A's
>> > > > value
>> > > > > > is
>> > > > > > > B.
>> > > > > > > > >> With
>> > > > > > > > >> >>> topic name regex then we have to first retrieve
>> __all__
>> > > > > topics's
>> > > > > > > > >> >>> description info and then filter at the client end,
>> which
>> > > > will
>> > > > > > be
>> > > > > > > a
>> > > > > > > > >> huge
>> > > > > > > > >> >>> burden on ZK.
>> > > > > > > > >> >>>
>> > > > > > > > >> >>> 3. Config K-Vs in create topic: this is related to
>> the
>> > > > > previous
>> > > > > > > > point;
>> > > > > > > > >> >>> maybe we can add another metadata K-V or just a
>> metadata
>> > > > > string
>> > > > > > > > along
>> > > > > > > > >> side
>> > > > > > > > >> >>> with config K-V in create topic like we did for
>> offset
>> > > > commit
>> > > > > > > > >> request. This
>> > > > > > > > >> >>> field can be quite useful in storing information like
>> > > > "owner"
>> > > > > of
>> > > > > > > the
>> > > > > > > > >> topic
>> > > > > > > > >> >>> who issue the create command, etc, which is quite
>> > > important
>> > > > > for
>> > > > > > a
>> > > > > > > > >> >>> multi-tenant setting. Then in the describe topic
>> request
>> > > we
>> > > > > can
>> > > > > > > also
>> > > > > > > > >> batch
>> > > > > > > > >> >>> on regex of the metadata field.
>> > > > > > > > >> >>>
>> > > > > > > > >> >>> 4. Today all the admin operations are async in the
>> sense
>> > > > that
>> > > > > > > > command
>> > > > > > > > >> will
>> > > > > > > > >> >>> return once it is written in ZK, and that is why we
>> need
>> > > > extra
>> > > > > > > > >> verification
>> > > > > > > > >> >>> like testUtil.waitForTopicCreated() / verify
>> partition
>> > > > > > > reassignment
>> > > > > > > > >> >>> request, etc. With admin requests we could add a
>> flag to
>> > > > > enable
>> > > > > > /
>> > > > > > > > >> disable
>> > > > > > > > >> >>> synchronous requests; when it is turned on, the
>> response
>> > > > will
>> > > > > > not
>> > > > > > > > >> return
>> > > > > > > > >> >>> until the request has been completed. And for async
>> > > requests
>> > > > > we
>> > > > > > > can
>> > > > > > > > >> add a
>> > > > > > > > >> >>> "token" field in the response, and then only need a
>> > > general
>> > > > > > "admin
>> > > > > > > > >> >>> verification request" with the given token to check
>> if the
>> > > > > async
>> > > > > > > > >> request
>> > > > > > > > >> >>> has been completed.
>> > > > > > > > >> >>>
>> > > > > > > > >> >>> 5. +1 for extending Metadata request to include
>> > > controller /
>> > > > > > > > >> coordinator
>> > > > > > > > >> >>> information, and then we can remove the
>> ConsumerMetadata /
>> > > > > > > > >> ClusterMetadata
>> > > > > > > > >> >>> requests.
>> > > > > > > > >> >>>
>> > > > > > > > >> >>> Guozhang
>> > > > > > > > >> >>>
>> > > > > > > > >> >>> On Tue, Mar 3, 2015 at 10:23 AM, Joel Koshy <
>> > > > > > jjkosh...@gmail.com>
>> > > > > > > > >> wrote:
>> > > > > > > > >> >>>
>> > > > > > > > >> >>>> Thanks for sending that out Joe - I don't think I
>> will be
>> > > > > able
>> > > > > > to
>> > > > > > > > >> make
>> > > > > > > > >> >>>> it today, so if notes can be sent out afterward that
>> > > would
>> > > > be
>> > > > > > > > great.
>> > > > > > > > >> >>>>
>> > > > > > > > >> >>>> On Mon, Mar 02, 2015 at 09:16:13AM -0800, Gwen
>> Shapira
>> > > > wrote:
>> > > > > > > > >> >>>> > Thanks for sending this out Joe. Looking forward
>> to
>> > > > > chatting
>> > > > > > > with
>> > > > > > > > >> >>>> everyone :)
>> > > > > > > > >> >>>> >
>> > > > > > > > >> >>>> > On Mon, Mar 2, 2015 at 6:46 AM, Joe Stein <
>> > > > > > > joe.st...@stealth.ly>
>> > > > > > > > >> wrote:
>> > > > > > > > >> >>>> > > Hey, I just sent out a google hangout invite to
>> all
>> > > > pmc,
>> > > > > > > > >> committers
>> > > > > > > > >> >>>> and
>> > > > > > > > >> >>>> > > everyone I found working on a KIP. If I missed
>> anyone
>> > > > in
>> > > > > > the
>> > > > > > > > >> invite
>> > > > > > > > >> >>>> please
>> > > > > > > > >> >>>> > > let me know and can update it, np.
>> > > > > > > > >> >>>> > >
>> > > > > > > > >> >>>> > > We should do this every Tuesday @ 2pm Eastern
>> Time.
>> > > > Maybe
>> > > > > > we
>> > > > > > > > can
>> > > > > > > > >> get
>> > > > > > > > >> >>>> INFRA
>> > > > > > > > >> >>>> > > help to make a google account so we can manage
>> > > better?
>> > > > > > > > >> >>>> > >
>> > > > > > > > >> >>>> > > To discuss
>> > > > > > > > >> >>>> > >
>> > > > > > > > >> >>>>
>> > > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>> > > > > > > > >> >>>> > > in progress and related JIRA that are
>> interdependent
>> > > > and
>> > > > > > > common
>> > > > > > > > >> work.
>> > > > > > > > >> >>>> > >
>> > > > > > > > >> >>>> > > ~ Joe Stein
>> > > > > > > > >> >>>> > >
>> > > > > > > > >> >>>> > > On Tue, Feb 24, 2015 at 2:59 PM, Jay Kreps <
>> > > > > > > > jay.kr...@gmail.com>
>> > > > > > > > >> >>>> wrote:
>> > > > > > > > >> >>>> > >
>> > > > > > > > >> >>>> > >> Let's stay on Google hangouts that will also
>> record
>> > > > and
>> > > > > > make
>> > > > > > > > the
>> > > > > > > > >> >>>> sessions
>> > > > > > > > >> >>>> > >> available on youtube.
>> > > > > > > > >> >>>> > >>
>> > > > > > > > >> >>>> > >> -Jay
>> > > > > > > > >> >>>> > >>
>> > > > > > > > >> >>>> > >> On Tue, Feb 24, 2015 at 11:49 AM, Jeff Holoman
>> <
>> > > > > > > > >> >>>> jholo...@cloudera.com>
>> > > > > > > > >> >>>> > >> wrote:
>> > > > > > > > >> >>>> > >>
>> > > > > > > > >> >>>> > >> > Jay / Joe
>> > > > > > > > >> >>>> > >> >
>> > > > > > > > >> >>>> > >> > We're happy to send out a Webex for this
>> purpose.
>> > > We
>> > > > > > could
>> > > > > > > > >> record
>> > > > > > > > >> >>>> the
>> > > > > > > > >> >>>> > >> > sessions if there is interest and publish
>> them
>> > > out.
>> > > > > > > > >> >>>> > >> >
>> > > > > > > > >> >>>> > >> > Thanks
>> > > > > > > > >> >>>> > >> >
>> > > > > > > > >> >>>> > >> > Jeff
>> > > > > > > > >> >>>> > >> >
>> > > > > > > > >> >>>> > >> > On Tue, Feb 24, 2015 at 11:28 AM, Jay Kreps <
>> > > > > > > > >> jay.kr...@gmail.com>
>> > > > > > > > >> >>>> wrote:
>> > > > > > > > >> >>>> > >> >
>> > > > > > > > >> >>>> > >> > > Let's try to get the technical hang-ups
>> sorted
>> > > > out,
>> > > > > > > > though.
>> > > > > > > > >> I
>> > > > > > > > >> >>>> really
>> > > > > > > > >> >>>> > >> > think
>> > > > > > > > >> >>>> > >> > > there is some benefit to live discussion vs
>> > > > > writing. I
>> > > > > > > am
>> > > > > > > > >> >>>> hopeful that
>> > > > > > > > >> >>>> > >> if
>> > > > > > > > >> >>>> > >> > > we post instructions and give ourselves a
>> few
>> > > > > attempts
>> > > > > > > we
>> > > > > > > > >> can
>> > > > > > > > >> >>>> get it
>> > > > > > > > >> >>>> > >> > > working.
>> > > > > > > > >> >>>> > >> > >
>> > > > > > > > >> >>>> > >> > > Tuesday at that time would work for
>> me...any
>> > > > > > objections?
>> > > > > > > > >> >>>> > >> > >
>> > > > > > > > >> >>>> > >> > > -Jay
>> > > > > > > > >> >>>> > >> > >
>> > > > > > > > >> >>>> > >> > > On Tue, Feb 24, 2015 at 8:18 AM, Joe Stein
>> <
>> > > > > > > > >> joe.st...@stealth.ly
>> > > > > > > > >> >>>> >
>> > > > > > > > >> >>>> > >> wrote:
>> > > > > > > > >> >>>> > >> > >
>> > > > > > > > >> >>>> > >> > > > Weekly would be great maybe like every
>> > > Tuesday ~
>> > > > > 1pm
>> > > > > > > ET
>> > > > > > > > /
>> > > > > > > > >> 10am
>> > > > > > > > >> >>>> PT
>> > > > > > > > >> >>>> > >> ????
>> > > > > > > > >> >>>> > >> > > >
>> > > > > > > > >> >>>> > >> > > > I don't mind google hangout but there is
>> > > always
>> > > > > some
>> > > > > > > > >> issue or
>> > > > > > > > >> >>>> > >> whatever
>> > > > > > > > >> >>>> > >> > so
>> > > > > > > > >> >>>> > >> > > > we know the apache irc channel works. We
>> can
>> > > > start
>> > > > > > > there
>> > > > > > > > >> and
>> > > > > > > > >> >>>> see how
>> > > > > > > > >> >>>> > >> it
>> > > > > > > > >> >>>> > >> > > > goes? We can pull transcripts too and
>> > > associate
>> > > > to
>> > > > > > > > >> tickets if
>> > > > > > > > >> >>>> need be
>> > > > > > > > >> >>>> > >> > > makes
>> > > > > > > > >> >>>> > >> > > > it helpful for things.
>> > > > > > > > >> >>>> > >> > > >
>> > > > > > > > >> >>>> > >> > > > ~ Joestein
>> > > > > > > > >> >>>> > >> > > >
>> > > > > > > > >> >>>> > >> > > > On Tue, Feb 24, 2015 at 11:10 AM, Jay
>> Kreps <
>> > > > > > > > >> >>>> jay.kr...@gmail.com>
>> > > > > > > > >> >>>> > >> > wrote:
>> > > > > > > > >> >>>> > >> > > >
>> > > > > > > > >> >>>> > >> > > > > We'd talked about doing a Google
>> Hangout to
>> > > > chat
>> > > > > > > about
>> > > > > > > > >> this.
>> > > > > > > > >> >>>> What
>> > > > > > > > >> >>>> > >> > about
>> > > > > > > > >> >>>> > >> > > > > generalizing that a little further...I
>> > > > actually
>> > > > > > > think
>> > > > > > > > it
>> > > > > > > > >> >>>> would be
>> > > > > > > > >> >>>> > >> > good
>> > > > > > > > >> >>>> > >> > > > for
>> > > > > > > > >> >>>> > >> > > > > everyone spending a reasonable chunk of
>> > > their
>> > > > > week
>> > > > > > > on
>> > > > > > > > >> Kafka
>> > > > > > > > >> >>>> stuff
>> > > > > > > > >> >>>> > >> to
>> > > > > > > > >> >>>> > >> > > > maybe
>> > > > > > > > >> >>>> > >> > > > > sync up once a week. I think we could
>> use
>> > > time
>> > > > > to
>> > > > > > > talk
>> > > > > > > > >> >>>> through
>> > > > > > > > >> >>>> > >> design
>> > > > > > > > >> >>>> > >> > > > > stuff, make sure we are on top of code
>> > > > reviews,
>> > > > > > talk
>> > > > > > > > >> through
>> > > > > > > > >> >>>> any
>> > > > > > > > >> >>>> > >> > tricky
>> > > > > > > > >> >>>> > >> > > > > issues, etc.
>> > > > > > > > >> >>>> > >> > > > >
>> > > > > > > > >> >>>> > >> > > > > We can make it publicly available so
>> that
>> > > any
>> > > > > one
>> > > > > > > can
>> > > > > > > > >> follow
>> > > > > > > > >> >>>> along
>> > > > > > > > >> >>>> > >> > who
>> > > > > > > > >> >>>> > >> > > > > likes.
>> > > > > > > > >> >>>> > >> > > > >
>> > > > > > > > >> >>>> > >> > > > > Any interest in doing this? If so I'll
>> try
>> > > to
>> > > > > set
>> > > > > > it
>> > > > > > > > up
>> > > > > > > > >> >>>> starting
>> > > > > > > > >> >>>> > >> next
>> > > > > > > > >> >>>> > >> > > > week.
>> > > > > > > > >> >>>> > >> > > > >
>> > > > > > > > >> >>>> > >> > > > > -Jay
>> > > > > > > > >> >>>> > >> > > > >
>> > > > > > > > >> >>>> > >> > > > > On Tue, Feb 24, 2015 at 3:57 AM, Andrii
>> > > > > Biletskyi
>> > > > > > <
>> > > > > > > > >> >>>> > >> > > > > andrii.bilets...@stealth.ly> wrote:
>> > > > > > > > >> >>>> > >> > > > >
>> > > > > > > > >> >>>> > >> > > > > > Hi all,
>> > > > > > > > >> >>>> > >> > > > > >
>> > > > > > > > >> >>>> > >> > > > > > I've updated KIP page, fixed /
>> aligned
>> > > > > document
>> > > > > > > > >> structure.
>> > > > > > > > >> >>>> Also I
>> > > > > > > > >> >>>> > >> > > added
>> > > > > > > > >> >>>> > >> > > > > > some
>> > > > > > > > >> >>>> > >> > > > > > very initial proposal for
>> AdminClient so
>> > > we
>> > > > > have
>> > > > > > > > >> something
>> > > > > > > > >> >>>> to
>> > > > > > > > >> >>>> > >> start
>> > > > > > > > >> >>>> > >> > > > from
>> > > > > > > > >> >>>> > >> > > > > > while
>> > > > > > > > >> >>>> > >> > > > > > discussing the KIP.
>> > > > > > > > >> >>>> > >> > > > > >
>> > > > > > > > >> >>>> > >> > > > > >
>> > > > > > > > >> >>>> > >> > > > > >
>> > > > > > > > >> >>>> > >> > > > >
>> > > > > > > > >> >>>> > >> > > >
>> > > > > > > > >> >>>> > >> > >
>> > > > > > > > >> >>>> > >> >
>> > > > > > > > >> >>>> > >>
>> > > > > > > > >> >>>>
>> > > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations
>> > > > > > > > >> >>>> > >> > > > > >
>> > > > > > > > >> >>>> > >> > > > > > Thanks,
>> > > > > > > > >> >>>> > >> > > > > > Andrii Biletskyi
>> > > > > > > > >> >>>> > >> > > > > >
>> > > > > > > > >> >>>> > >> > > > > > On Wed, Feb 18, 2015 at 9:01 PM,
>> Andrii
>> > > > > > Biletskyi
>> > > > > > > <
>> > > > > > > > >> >>>> > >> > > > > > andrii.bilets...@stealth.ly> wrote:
>> > > > > > > > >> >>>> > >> > > > > >
>> > > > > > > > >> >>>> > >> > > > > > > Jay,
>> > > > > > > > >> >>>> > >> > > > > > >
>> > > > > > > > >> >>>> > >> > > > > > > Re error messages: you are right,
>> in
>> > > most
>> > > > > > cases
>> > > > > > > > >> client
>> > > > > > > > >> >>>> will
>> > > > > > > > >> >>>> > >> have
>> > > > > > > > >> >>>> > >> > > > enough
>> > > > > > > > >> >>>> > >> > > > > > > context to show descriptive error
>> > > message.
>> > > > > My
>> > > > > > > > >> concern is
>> > > > > > > > >> >>>> that
>> > > > > > > > >> >>>> > >> we
>> > > > > > > > >> >>>> > >> > > will
>> > > > > > > > >> >>>> > >> > > > > > have
>> > > > > > > > >> >>>> > >> > > > > > > to
>> > > > > > > > >> >>>> > >> > > > > > > add lots of new error codes for
>> each
>> > > > > possible
>> > > > > > > > >> error. Of
>> > > > > > > > >> >>>> course,
>> > > > > > > > >> >>>> > >> > we
>> > > > > > > > >> >>>> > >> > > > > could
>> > > > > > > > >> >>>> > >> > > > > > > reuse
>> > > > > > > > >> >>>> > >> > > > > > > some of existing like
>> > > > > > > UknownTopicOrPartitionCode,
>> > > > > > > > >> but we
>> > > > > > > > >> >>>> will
>> > > > > > > > >> >>>> > >> > also
>> > > > > > > > >> >>>> > >> > > > need
>> > > > > > > > >> >>>> > >> > > > > > to
>> > > > > > > > >> >>>> > >> > > > > > > add smth like:
>> TopicAlreadyExistsCode,
>> > > > > > > > >> >>>> TopicConfigInvalid (both
>> > > > > > > > >> >>>> > >> > for
>> > > > > > > > >> >>>> > >> > > > > topic
>> > > > > > > > >> >>>> > >> > > > > > > name and config, and probably user
>> would
>> > > > > like
>> > > > > > to
>> > > > > > > > >> know
>> > > > > > > > >> >>>> what
>> > > > > > > > >> >>>> > >> > exactly
>> > > > > > > > >> >>>> > >> > > > > > > is wrong in his config),
>> > > > > > > InvalidReplicaAssignment,
>> > > > > > > > >> >>>> > >> InternalError
>> > > > > > > > >> >>>> > >> > > > (e.g.
>> > > > > > > > >> >>>> > >> > > > > > > zookeeper failure) etc.
>> > > > > > > > >> >>>> > >> > > > > > > And this is only for TopicCommand,
>> we
>> > > will
>> > > > > > also
>> > > > > > > > >> need to
>> > > > > > > > >> >>>> add
>> > > > > > > > >> >>>> > >> > similar
>> > > > > > > > >> >>>> > >> > > > > stuff
>> > > > > > > > >> >>>> > >> > > > > > > for
>> > > > > > > > >> >>>> > >> > > > > > > ReassignPartitions,
>> PreferredReplica. So
>> > > > > we'll
>> > > > > > > end
>> > > > > > > > >> up
>> > > > > > > > >> >>>> with a
>> > > > > > > > >> >>>> > >> > large
>> > > > > > > > >> >>>> > >> > > > list
>> > > > > > > > >> >>>> > >> > > > > > of
>> > > > > > > > >> >>>> > >> > > > > > > error codes, used only in Admin
>> > > protocol.
>> > > > > > > > >> >>>> > >> > > > > > > Having said that, I agree my
>> proposal is
>> > > > not
>> > > > > > > > >> consistent
>> > > > > > > > >> >>>> with
>> > > > > > > > >> >>>> > >> > other
>> > > > > > > > >> >>>> > >> > > > > cases.
>> > > > > > > > >> >>>> > >> > > > > > > Maybe we can find better solution
>> or
>> > > > > something
>> > > > > > > > >> >>>> in-between.
>> > > > > > > > >> >>>> > >> > > > > > >
>> > > > > > > > >> >>>> > >> > > > > > > Re Hangout chat: I think it is a
>> great
>> > > > idea.
>> > > > > > > This
>> > > > > > > > >> way we
>> > > > > > > > >> >>>> can
>> > > > > > > > >> >>>> > >> move
>> > > > > > > > >> >>>> > >> > > on
>> > > > > > > > >> >>>> > >> > > > > > > faster.
>> > > > > > > > >> >>>> > >> > > > > > > Let's agree somehow on date/time so
>> > > people
>> > > > > can
>> > > > > > > > join.
>> > > > > > > > >> >>>> Will work
>> > > > > > > > >> >>>> > >> > for
>> > > > > > > > >> >>>> > >> > > me
>> > > > > > > > >> >>>> > >> > > > > > this
>> > > > > > > > >> >>>> > >> > > > > > > and
>> > > > > > > > >> >>>> > >> > > > > > > next week almost anytime if agreed
>> in
>> > > > > advance.
>> > > > > > > > >> >>>> > >> > > > > > >
>> > > > > > > > >> >>>> > >> > > > > > > Thanks,
>> > > > > > > > >> >>>> > >> > > > > > > Andrii
>> > > > > > > > >> >>>> > >> > > > > > >
>> > > > > > > > >> >>>> > >> > > > > > > On Wed, Feb 18, 2015 at 7:09 PM,
>> Jay
>> > > > Kreps <
>> > > > > > > > >> >>>> > >> jay.kr...@gmail.com>
>> > > > > > > > >> >>>> > >> > > > > wrote:
>> > > > > > > > >> >>>> > >> > > > > > >
>> > > > > > > > >> >>>> > >> > > > > > >> Hey Andrii,
>> > > > > > > > >> >>>> > >> > > > > > >>
>> > > > > > > > >> >>>> > >> > > > > > >> Generally we can do good error
>> handling
>> > > > > > without
>> > > > > > > > >> needing
>> > > > > > > > >> >>>> custom
>> > > > > > > > >> >>>> > >> > > > > > server-side
>> > > > > > > > >> >>>> > >> > > > > > >> messages. I.e. generally the
>> client has
>> > > > the
>> > > > > > > > >> context to
>> > > > > > > > >> >>>> know
>> > > > > > > > >> >>>> > >> that
>> > > > > > > > >> >>>> > >> > > if
>> > > > > > > > >> >>>> > >> > > > it
>> > > > > > > > >> >>>> > >> > > > > > got
>> > > > > > > > >> >>>> > >> > > > > > >> an error that the topic doesn't
>> exist
>> > > to
>> > > > > say
>> > > > > > > > >> "Topic X
>> > > > > > > > >> >>>> doesn't
>> > > > > > > > >> >>>> > >> > > exist"
>> > > > > > > > >> >>>> > >> > > > > > >> rather
>> > > > > > > > >> >>>> > >> > > > > > >> than "error code 14" (or
>> whatever).
>> > > Maybe
>> > > > > > there
>> > > > > > > > are
>> > > > > > > > >> >>>> specific
>> > > > > > > > >> >>>> > >> > cases
>> > > > > > > > >> >>>> > >> > > > > where
>> > > > > > > > >> >>>> > >> > > > > > >> this is hard? If we want to add
>> > > > server-side
>> > > > > > > error
>> > > > > > > > >> >>>> messages we
>> > > > > > > > >> >>>> > >> > > really
>> > > > > > > > >> >>>> > >> > > > > do
>> > > > > > > > >> >>>> > >> > > > > > >> need to do this in a consistent
>> way
>> > > > across
>> > > > > > the
>> > > > > > > > >> protocol.
>> > > > > > > > >> >>>> > >> > > > > > >>
>> > > > > > > > >> >>>> > >> > > > > > >> I still have a bunch of open
>> questions
>> > > > here
>> > > > > > > from
>> > > > > > > > my
>> > > > > > > > >> >>>> previous
>> > > > > > > > >> >>>> > >> > > list. I
>> > > > > > > > >> >>>> > >> > > > > > will
>> > > > > > > > >> >>>> > >> > > > > > >> be out for the next few days for
>> Strata
>> > > > > > though.
>> > > > > > > > >> Maybe
>> > > > > > > > >> >>>> we could
>> > > > > > > > >> >>>> > >> > do
>> > > > > > > > >> >>>> > >> > > a
>> > > > > > > > >> >>>> > >> > > > > > Google
>> > > > > > > > >> >>>> > >> > > > > > >> Hangout chat on any open issues
>> some
>> > > time
>> > > > > > > towards
>> > > > > > > > >> the
>> > > > > > > > >> >>>> end of
>> > > > > > > > >> >>>> > >> > next
>> > > > > > > > >> >>>> > >> > > > week
>> > > > > > > > >> >>>> > >> > > > > > for
>> > > > > > > > >> >>>> > >> > > > > > >> anyone interested in this ticket?
>> I
>> > > have
>> > > > a
>> > > > > > > > feeling
>> > > > > > > > >> that
>> > > > > > > > >> >>>> might
>> > > > > > > > >> >>>> > >> > > > progress
>> > > > > > > > >> >>>> > >> > > > > > >> things a little faster than
>> email--I
>> > > > think
>> > > > > we
>> > > > > > > > >> could talk
>> > > > > > > > >> >>>> > >> through
>> > > > > > > > >> >>>> > >> > > > those
>> > > > > > > > >> >>>> > >> > > > > > >> issues I brought up fairly
>> quickly...
>> > > > > > > > >> >>>> > >> > > > > > >>
>> > > > > > > > >> >>>> > >> > > > > > >> -Jay
>> > > > > > > > >> >>>> > >> > > > > > >>
>> > > > > > > > >> >>>> > >> > > > > > >> On Wed, Feb 18, 2015 at 7:27 AM,
>> Andrii
>> > > > > > > > Biletskyi <
>> > > > > > > > >> >>>> > >> > > > > > >> andrii.bilets...@stealth.ly>
>> wrote:
>> > > > > > > > >> >>>> > >> > > > > > >>
>> > > > > > > > >> >>>> > >> > > > > > >> > Hi all,
>> > > > > > > > >> >>>> > >> > > > > > >> >
>> > > > > > > > >> >>>> > >> > > > > > >> > I'm trying to address some of
>> the
>> > > > issues
>> > > > > > > which
>> > > > > > > > >> were
>> > > > > > > > >> >>>> > >> mentioned
>> > > > > > > > >> >>>> > >> > > > > earlier
>> > > > > > > > >> >>>> > >> > > > > > >> about
>> > > > > > > > >> >>>> > >> > > > > > >> > Admin RQ/RP format. One of
>> those was
>> > > > > about
>> > > > > > > > >> batching
>> > > > > > > > >> >>>> > >> > operations.
>> > > > > > > > >> >>>> > >> > > > What
>> > > > > > > > >> >>>> > >> > > > > > if
>> > > > > > > > >> >>>> > >> > > > > > >> we
>> > > > > > > > >> >>>> > >> > > > > > >> > follow TopicCommand approach
>> and let
>> > > > > people
>> > > > > > > > >> specify
>> > > > > > > > >> >>>> > >> topic-name
>> > > > > > > > >> >>>> > >> > > by
>> > > > > > > > >> >>>> > >> > > > > > >> regexp -
>> > > > > > > > >> >>>> > >> > > > > > >> > would that cover most of the use
>> > > cases?
>> > > > > > > > >> >>>> > >> > > > > > >> >
>> > > > > > > > >> >>>> > >> > > > > > >> > Secondly, is what information
>> should
>> > > we
>> > > > > > > > generally
>> > > > > > > > >> >>>> provide in
>> > > > > > > > >> >>>> > >> > > Admin
>> > > > > > > > >> >>>> > >> > > > > > >> > responses.
>> > > > > > > > >> >>>> > >> > > > > > >> > I realize that Admin commands
>> don't
>> > > > imply
>> > > > > > > they
>> > > > > > > > >> will
>> > > > > > > > >> >>>> be used
>> > > > > > > > >> >>>> > >> > only
>> > > > > > > > >> >>>> > >> > > > in
>> > > > > > > > >> >>>> > >> > > > > > CLI
>> > > > > > > > >> >>>> > >> > > > > > >> > but,
>> > > > > > > > >> >>>> > >> > > > > > >> > it seems to me, CLI is a very
>> > > important
>> > > > > > > client
>> > > > > > > > >> of this
>> > > > > > > > >> >>>> > >> > feature.
>> > > > > > > > >> >>>> > >> > > In
>> > > > > > > > >> >>>> > >> > > > > > this
>> > > > > > > > >> >>>> > >> > > > > > >> > case,
>> > > > > > > > >> >>>> > >> > > > > > >> > seems logical, we would like to
>> > > provide
>> > > > > > users
>> > > > > > > > >> with
>> > > > > > > > >> >>>> rich
>> > > > > > > > >> >>>> > >> > > experience
>> > > > > > > > >> >>>> > >> > > > > in
>> > > > > > > > >> >>>> > >> > > > > > >> terms
>> > > > > > > > >> >>>> > >> > > > > > >> > of
>> > > > > > > > >> >>>> > >> > > > > > >> > getting results / errors of the
>> > > > executed
>> > > > > > > > >> commands.
>> > > > > > > > >> >>>> Usually
>> > > > > > > > >> >>>> > >> we
>> > > > > > > > >> >>>> > >> > > > supply
>> > > > > > > > >> >>>> > >> > > > > > >> with
>> > > > > > > > >> >>>> > >> > > > > > >> > responses only errorCode, which
>> looks
>> > > > > very
>> > > > > > > > >> limiting,
>> > > > > > > > >> >>>> in case
>> > > > > > > > >> >>>> > >> > of
>> > > > > > > > >> >>>> > >> > > > CLI
>> > > > > > > > >> >>>> > >> > > > > we
>> > > > > > > > >> >>>> > >> > > > > > >> may
>> > > > > > > > >> >>>> > >> > > > > > >> > want to print human readable
>> error
>> > > > > > > description.
>> > > > > > > > >> >>>> > >> > > > > > >> >
>> > > > > > > > >> >>>> > >> > > > > > >> > So, taking into account
>> previous item
>> > > > > about
>> > > > > > > > >> batching,
>> > > > > > > > >> >>>> what
>> > > > > > > > >> >>>> > >> do
>> > > > > > > > >> >>>> > >> > > you
>> > > > > > > > >> >>>> > >> > > > > > think
>> > > > > > > > >> >>>> > >> > > > > > >> > about
>> > > > > > > > >> >>>> > >> > > > > > >> > having smth like:
>> > > > > > > > >> >>>> > >> > > > > > >> >
>> > > > > > > > >> >>>> > >> > > > > > >> > ('create' doesn't support
>> regexp)
>> > > > > > > > >> >>>> > >> > > > > > >> > CreateTopicRequest => TopicName
>> > > > > Partitions
>> > > > > > > > >> Replicas
>> > > > > > > > >> >>>> > >> > > > > ReplicaAssignment
>> > > > > > > > >> >>>> > >> > > > > > >> > [Config]
>> > > > > > > > >> >>>> > >> > > > > > >> > CreateTopicResponse => ErrorCode
>> > > > > > > > ErrorDescription
>> > > > > > > > >> >>>> > >> > > > > > >> > ErrorCode => int16
>> > > > > > > > >> >>>> > >> > > > > > >> > ErrorDescription => string
>> (empty
>> > > if
>> > > > > > > > >> successful)
>> > > > > > > > >> >>>> > >> > > > > > >> >
>> > > > > > > > >> >>>> > >> > > > > > >> > AlterTopicRequest ->
>> TopicNameRegexp
>> > > >
>
> ...
>
> [Message clipped]
