Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

; From: Joel Koshy [jjkosh...@gmail.com] > Sent: Thursday, June 11, 2015 1:28 PM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative > operations (Thread 2) > > Discussion aside, was there any significant material change beside

RE: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

afka.apache.org Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2) Discussion aside, was there any significant material change besides the additions below? If so, then we can avoid the overhead of another vote unless someone wants to down-vote these

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

> To: dev@kafka.apache.org > Subject: RE: [DISCUSS] KIP-4 - Command line and centralized administrative > operations (Thread 2) > > I've made two changes to the document: > - Removed the TMR evolution piece since we agreed to retain this. > - Added two new API's to

RE: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

rg Subject: RE: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2) I've made two changes to the document: - Removed the TMR evolution piece since we agreed to retain this. - Added two new API's to the admin client spec. (Alter and Describe config). Pl

RE: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

.com] Sent: Friday, May 29, 2015 8:36 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2) +1 on discussing this on next KIP hangout. I will update KIP-24 before that. On Fri, May 29, 2015 at 3:40 AM, Andrii Biletskyi <

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

anks. Perhaps we should leave TMR unchanged for now. Should we discuss > > this during the next hangout? > > > > Aditya > > > > > > From: Jun Rao [j...@confluent.io] > > Sent: Thursday, May 28, 2015 5:32 PM > > To: d

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

x27;ve altered the KIP with the > > assumption that this is a good enough reason by itself to evolve the > > request/response protocol. Any concerns there? > > > > Thanks, > > Aditya > > > > ____________ > > From: Mayuresh Gharat [gharatmayures.

RE: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

___ > From: Mayuresh Gharat [gharatmayures...@gmail.com] > Sent: Thursday, May 21, 2015 8:29 PM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative > operations (Thread 2) > > Hi Jun, > > Thanks a lo

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

arat [gharatmayures...@gmail.com] > Sent: Thursday, May 21, 2015 8:29 PM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative > operations (Thread 2) > > Hi Jun, > > Thanks a lot. I get it now. > Point 4) will actually en

RE: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

015 8:29 PM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2) Hi Jun, Thanks a lot. I get it now. Point 4) will actually enable clients to who don't want to create a topic with default partitions, if it does not exist an

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Hi Jun, Thanks a lot. I get it now. Point 4) will actually enable clients to who don't want to create a topic with default partitions, if it does not exist and then can manually create the topic with their own configs(#partitions). Thanks, Mayuresh On Thu, May 21, 2015 at 6:16 PM, Jun Rao wro

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Mayuresh, The current plan is the following. 1. Add TMR v1, which still triggers auto topic creation. 2. Change the consumer client to TMR v1. Change the producer client to use TMR v1 and on UnknownTopicException, issue TopicCreateRequest to explicitly create the topic with the default server sid

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

> Here are some ideas to address this : > > 1) The way this can be addressed is that TopicMetadata request should have > a way to specify whether it should only check if the topic exist or check > and create a topic with given number of partitions. If the number of > partitions is not specified u

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Hi, I had a question about TopicMetadata Request. Currently the way it works is : 1) Suppose a topic T1 does not exist. 2) Client wants to produce data to T1 using producer P1. 3) Since T1 does not exist, P1 issues a TopicMetadata request to kafka. This in turn creates the default number of partit

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

For ListTopics, we decided not to add a ListTopics request for now and just rely on passing in an empty list to TMR. We can revisit this in the future if it becomes an issue. Thanks, Jun On Wed, May 13, 2015 at 3:31 PM, Joel Koshy wrote: > Just had a few minor questions before I join the vote

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Joel, - DecreasePartitionNotAllowed. Yeah, that's kind of subcase of InvalidPartitions... But since client can always request topic metadata and check what exactly is was wrong with Partitions argument, I think, yes, we can remove DecreasePartitionNotAllowed and use InvalidPartitions instead. I'll

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Just had a few minor questions before I join the vote thread. Apologies if these have been discussed: - Do we need DecreasePartitionsNotAllowed? i.e., can we just return InvalidPartitions instead? - AdminClient.listTopics: should we allow listing all partitions? Or do you intend for the client

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Guys, I've updated the wiki to reflect all previously discussed items (regarding the schema only - this is included to phase 1). https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations I think we can have the final discussion today (for pha

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

The following is a description of some of my concerns on allowing the same topic multiple times in AlterTopicRequest. ATP has an array of entries, each corresponding to a topic. We allow multiple changes to a topic in a single entry. Those changes may fail to apply independently (e.g., the config

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Guys, A quick summary of our today's meeting. There were no additional issues/questions. The only item about which we are not 100% sure is "multiple instructions for one topic in one request" case. It was proposed by Jun to explain reasons behind not allowing users doing that again here in mailin

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Guys, It seems that there are no open questions left so prior to our weekly call let me summarize what I'm going to implement as part of phase one for KIP-4. 1. Add 3 new Wire Protocol requests - Create-, Alter- and DeleteTopicRequest 2. Topic requests are batch requests, errors are returned per

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Yes, to verify if a partition reassignment completes or not, we just need to make sure AR == RAR. So, we don't need ISR for this. It's probably still useful to know ISR for monitoring in general though. Thanks, Jun On Mon, Apr 27, 2015 at 4:15 AM, Andrii Biletskyi < andrii.bilets...@stealth.ly>

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Okay, I had some doubts in terms of reassign-partitions case. I was not sure whether we need ISR to check post condition of partition reassignment. But I think we can rely on assigned replicas - the workflow in reassignPartitions is the following: 1. Update AR in ZK with OAR + RAR. ... 10. Update A

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Andrii, Another thing. We decided not to add the lag info in TMR. To be consistent, we probably also want to remove ISR from TMR since only the leader knows it. We can punt on adding any new request from getting ISR. ISR is mostly useful for monitoring. Currently, one can determine if a replica is

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Jun, I like your approach to AlterTopicReques semantics! Sounds like we linearize all request fields to ReplicaAssignment - I will definitely try this out to ensure there are no other pitfalls. With regards to multiple instructions in one batch per topic. For me this sounds reasonable too. We dis

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Andrii, Thanks for the update. For your second point, I agree that if a single AlterTopicRequest can make multiple changes, there is no need to support the same topic included more than once in the request. Now about the semantics in your first question. I was thinking that we can do the followi

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Guys, Can we come to some agreement in terms of the second item from the email above? This blocks me from updating and uploading the patch. Also the new schedule for the weekly calls doesn't work very well for me - it's 1 am in my timezone :) - so I'd rather we confirm everything that is possible

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

As said above, I spent some time thinking about AlterTopicRequest semantics and batching. Firstly, about AlterTopicRequest. Our goal here is to see whether we can suggest some simple semantics and at the same time let users change different things in one instruction (hereinafter instruction - is o

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Guys, Thank you for your time. A short summary of our discussion. Answering previous items: 1. 2. I will double check existing error codes to align the list of errors that needs to be added. 3. We agreed to think again about the batch requests semantics. The main concern is that users would expe

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Hey Andrii, thanks for all the hard work on this, it has come a long way. A couple questions and comments on this. For the errors, can we do the following: 1. Remove IllegalArgument from the name, we haven't used that convention for other errors. 2. Normalize this list with the existing errors. F

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Hi all, I've updated KIP-4 page to include all previously discussed items such as: new error codes, merged alter-topic and reassign-partitions requests, added TMR_V1. It'd be great if we concentrate on the Errors+Wire Protocol schema and discuss any remaining issues today, since first patch will

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

1. Yes, this will be much easier. Okay, let's add it. 2, Okay. This will differ a little bit from the way currently kafka-topics.sh handles alter-topic command, but I think it's a reasonable restriction. I'll update KIP acordingly to our weekly call. Thanks, Andrii Biletskyi On Mon, Apr 20, 201

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

1. Yes, lag is probably only going to be useful for the admin client. However, so is isr. It seems to me that we should get lag and isr from the same request. I was thinking that we can just extend TMR by changing replicas from an array of int to an array of (int, lag) pairs. Is that too complicate

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Jun, 1. Yes, seems we can add lag info to the TMR. But before that I wonder whether there are other reasons we need this info except for reassign partition command? As we discussed earlier the problem with poor monitoring capabilities for reassign-partitions (as currently we only inform users Comp

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

1. For the lags, we can add a new field "lags" per partition. It will return for each replica that's not in isr, the replica id and the lag in messages. Also, if TMR is sent to a non-leader, the response can just include an empty array for isr and lags. 2. So, we will just return a topic level err

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

1. agreed 2. agree new error 3. having discrete operations for tasks makes sense, combining them is confusing for users I think. + 1 for "let user change only one thing at a time" 4. lets be consistent both to the new code and existing code. lets not confuse the user but give them the right erro

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Guys, Thanks for the discussion! Summary: 1. Q: How KAFKA-1367 (isr is inconsistent in brokers' metadata cache) can affect implementation? A: We can fix this issue for the leading broker - ReplicaManager (or Partition) component should have accurate isr list, then with leadin

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Andrii, 500. I think what you suggested also sounds reasonable. Since ISR is only maintained accurately at the leader, TMR can return ISR if the broker is the leader of a partition. Otherwise, we can return an empty ISR. For partition reassignment, it would be useful to know the lag of each follow

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Jun, 404. Great, thanks! 500. If I understand correctly KAFKA-1367 says ISR part of TMR may be inconsistent. If so, then I believe all admin commands but describeTopic are not affected. Let me emphasize that it's about AdminClient operations, not about Wire Protocol requests. What I mean: To veri

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Andrii, 404. Jay and I chatted a bit. We agreed to leave createTopicRequest as async for now. There is another thing. 500. Currently, we have this issue (KAFKA-1367) that the ISR in the metadata cache can be out of sync. The reason is that ISR is really maintained at the leader. We can potential

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Hi all, A summary of our discussion: 201. Q: Cluster updates in backward compatible way. A: Add topicConfigs map property and change constructor, this shouldn't break Consumer/Producer since TMR is used in NetworkClient, not directly by Consumer/Producer. 300. Q: Can we merge AlterTopic

[DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Hi all, I wasn't able to send email to our thread (it says we exceeded message size limit :)). So I'm starting the new one. Jun, Thanks again for the review. Answering your comments: 201. I'm not sure I understand how can we evolve Cluster in backward compatible way. In my understanding topic

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Andrii, A few points. 1. Create/Alter can typically complete quickly. So, it's possible to make the request block until it's completed. However, currently, doing this at the broker is a bit involved. To make Create block, we will need to add some callbacks in KafkaController. This is possible. Ho

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Jun, I see your point. But wouldn't that lead to a "fat" client implementations? Suppose someone would like to implement client for Admin Wire protocol. Not only people will have to code quite complicated logic like "send describe request to each broker" (again state machin?) but it will also mean

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

For 1), 2) and 3), blocking would probably mean that the new metadata is propagated to every broker. To achieve that, the client can keep issuing the describe topic request to every broker until it sees the new metadata in the response. Thanks, Jun On Fri, Mar 20, 2015 at 12:16 PM, Andrii Bilets

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Hm, actually the ticket you linked, Guozhang, brings as back to the problem what should be considered a post-condition for each of the admin commands. In my understanding: 1) CreateTopic - broker created /brokers/topics/ (Not the controller picked up changes from zk and broadcasted LeaderAndIsr an

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

I think while loop is fine for supporting blocking, just that we need to add back off to avoid bombarding brokers with DescribeTopic requests. Also I have linked KAFKA-1125 to your proposal, and when KAFKA-1694 is done this ticket can also be clos

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Great. I want to elaborate this a bit more, to see we are on the same page concerning the client code. So with all topic commands being async a client (AdminClient in our case or any other other client people would like to implement) to support a blocking operation (which seems to be a natural use

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Andrii, I think you are right. It seems that only ReassignPartitions needs a separate verification request. Thanks, Jun On Thu, Mar 19, 2015 at 9:22 AM, Andrii Biletskyi < andrii.bilets...@stealth.ly> wrote: > Guys, > I like this idea too. Let's stick with that. I'll update KIP accordingly. >

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Guys, I like this idea too. Let's stick with that. I'll update KIP accordingly. I was also thinking we can avoid adding dedicated status check requests for topic commands. - We have everything in DescribeTopic for that! E.g.: User issued CreateTopic - to check the status client sends DescribeTopic

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

+1 on broker writing to ZK for async handling. I was thinking that in the end state the admin requests would be eventually sent to controller either through re-routing or clients discovering them, instead of letting controller listen on ZK admin path. But thinking about it a second time, I think it

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

+1 as well. I think it helps to keep the rerouting approach orthogonal to this KIP. On Wed, Mar 18, 2015 at 03:40:48PM -0700, Jay Kreps wrote: > I'm +1 on Jun's suggestion as long as it can work for all the requests. > > On Wed, Mar 18, 2015 at 3:35 PM, Jun Rao wrote: > > > Andrii, > > > > I th

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

I'm +1 on Jun's suggestion as long as it can work for all the requests. On Wed, Mar 18, 2015 at 3:35 PM, Jun Rao wrote: > Andrii, > > I think we agreed on the following. > > (a) Admin requests can be sent to and handled by any broker. > (b) Admin requests are processed asynchronously, at least f

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Andrii, I think we agreed on the following. (a) Admin requests can be sent to and handled by any broker. (b) Admin requests are processed asynchronously, at least for now. That is, when the client gets a response, it just means that the request is initiated, but not necessarily completed. Then, i

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Yes that is what I was alluding to when I said that if we finally do request rerouting in Kafka then the field would add little to no value. I wasn't sure if we agreed that we _will_ do rerouting or whether we agreed to evaluate it (KAFKA-1912). Andrii can you update the KIP with this? Thanks, Jo

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Jun, I might be wrong but didn't we agree we will let any broker from the cluster handle *long-running* admin requests (at this time preferredReplica and reassignPartitions), via zk admin path. Thus CreateTopics etc should be sent only to the controller. Thanks, Andrii Biletskyi On Wed, Mar 18,

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Joel, Andril, I think we agreed that those admin requests can be issued to any broker. Because of that, there doesn't seem to be a strong need to know the controller. So, perhaps we can proceed by not making any change to the format of TMR right now. When we start using create topic request in the

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Joel, I'm totally behind your arguments concerning adding irrelevant stuff to TopicMetadataRequest. And also about having a bloated request. Personally I'd go with a separate ClusterMetadataRequest (CMR), actually this was our initial proposal. But since the second part of the request - brokers i

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Got it. Thanks for clarifying! On Wed, Mar 18, 2015 at 11:54 AM, Andrii Biletskyi wrote: > Gwen, > > Yes, looks like KAFKA-1927 will leave TopicMetadataRequest/Response. > But I believe, KIP is still tightly related with KAFKA-1927 since we are > not only > going to update TopicMetadataRequest th

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

(Thanks Andrii for the summary) For (1) yes we will circle back on that shortly after syncing up in person. I think it is close to getting committed although development for KAFKA-1927 can probably begin without it. There is one more item we covered at the hangout. i.e., whether we want to add th

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Gwen, Yes, looks like KAFKA-1927 will leave TopicMetadataRequest/Response. But I believe, KIP is still tightly related with KAFKA-1927 since we are not only going to update TopicMetadataRequest there but we will introduce a bunch of new requests too. And it probably makes sense to do those correct

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

On Wed, Mar 18, 2015 at 9:34 AM, Jun Rao wrote: > Andri, > > Thanks for the summary. > > 1. I just realized that in order to start working on KAFKA-1927, we will > need to merge the changes to OffsetCommitRequest (from 0.8.2) to trunk. > This is planned to be done as part of KAFKA-1634. So, we wil

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Andri, Thanks for the summary. 1. I just realized that in order to start working on KAFKA-1927, we will need to merge the changes to OffsetCommitRequest (from 0.8.2) to trunk. This is planned to be done as part of KAFKA-1634. So, we will need Guozhang and Joel's help to wrap this up. 2. Thinking

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Guys, Thanks for a great discussion! Here are the actions points: 1. Q: Get rid of all scala requests objects, use java protocol definitions. A: Gwen kindly took that (KAFKA-1927). It's important to speed up review procedure there since this ticket blocks other important changes. 2.

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Joel, You are right, I removed ClusterMetadata because we have partially what we need in TopicMetadata. Also, as Jay pointed out earlier, we would like to have "orthogonal" API, but at the same time we need to be backward compatible. But I like your idea and even have some other arguments for thi

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

I may be missing some context but hopefully this will also be covered today: I thought the earlier proposal where there was an explicit ClusterMetadata request was clearer and explicit. During the course of this thread I think the conclusion was that the main need was for controller information and

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Jun, 101. Okay, if you say that such use case is important. I also think using clientId for these purposes is fine - if we already have this field as part of all Wire protocol messages, why not use that. I will update KIP-4 page if nobody has other ideas (which may come up during the call today).

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

101. There may be a use case where you only want the topics to be created manually by admins. Currently, you can do that by disabling auto topic creation and issue topic creation from the TopicCommand. If we disable auto topic creation completely on the broker and don't have a way to distinguish be

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Jun, Answering your questions: 101. If I understand you correctly, you are saying future producer versions (which will be ported to TMR_V1) won't be able to automatically create topic (if we unconditionally remove topic creation from there). But we need to this preserve logic. Ok, about your prop

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Andrii, 101. That's what I was thinking too, but it may not be that simple. In TopicMetadataRequest_V1, we can let it not trigger auto topic creation. Then, in the producer side, if it gets an UnknownTopicException, it can explicitly issue a createTopicRequest for auto topic creation. On the consu

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Jun, Thanks for you comments. Answers inline: 100. There are a few fields such as ReplicaAssignment, > ReassignPartitionRequest, > and PartitionsSerialized that are represented as a string, but contain > composite structures in json. Could we flatten them out directly in the > protocol definition

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Andrii, A few more comments. 100. There are a few fields such as ReplicaAssignment, ReassignPartitionRequest, and PartitionsSerialized that are represented as a string, but contain composite structures in json. Could we flatten them out directly in the protocol definition as arrays/records? 101

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Hi, As said above - I list again all comments from this thread so we can see what's left and finalize all pending issues. Comments from Jay: 1. This is much needed functionality, but there are a lot of the so let's really think these protocols through. We really want to end up with a set of well

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Hi all, Today I uploaded the patch that covers some of the discussed and agreed items: - removed MaybeOf optional type - switched to java protocol definitions - simplified messages (normalized configs, removed topic marked for deletion) I also updated the KIP-4 with respective changes and wrote d

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Gwen, My main motivation is not "authenticate via ownership", but rather "query topics via ownership", and more generally "query topics via patterns", where a pattern could be a config value, metadata k-v pair, etc. Does it make sense? Guozhang On Thu, Mar 12, 2015 at 12:26 PM, Gwen Shapira wro

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Found KIP-11 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface) It actually specifies changes to the Metadata protocol, so making sure both KIPs are consistent in this regard will be good. On Thu, Mar 12, 2015 at 12:21 PM, Gwen Shapira wrote: > Specifically for

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Specifically for ownership, I think the plan is to add ACL (it sounds like you are describing ACL) via an external system (Argus, Sentry). I remember KIP-11 described this, but I can't find the KIP any longer. Regardless, I think KIP-4 focuses on getting information that already exists from Kafka

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

eal with any of these. > > > > > > > > Thanks. > > > > > > > > Tong Li > > > > OpenStack & Kafka Community Development > > > > Building 501/B205 > > > > liton...@us.ibm.com > > > > > > > > [imag

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

; Kafka Community Development > > > Building 501/B205 > > > liton...@us.ibm.com > > > > > > [image: Inactive hide details for Guozhang Wang ---03/12/2015 09:39:50 > > > AM---Folks, Just want to elaborate a bit more on the > create-topi]Guozhang > &

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

lks, Just want to elaborate a bit more on the create-topi]Guozhang > > Wang ---03/12/2015 09:39:50 AM---Folks, Just want to elaborate a bit more > > on the create-topic metadata and batching > > > > From: Guozhang Wang > > To: "dev@kafka.apache.org" > > Date

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

--Folks, Just want to elaborate a bit more on the create-topi]Guozhang > Wang ---03/12/2015 09:39:50 AM---Folks, Just want to elaborate a bit more > on the create-topic metadata and batching > > From: Guozhang Wang > To: "dev@kafka.apache.org" > Date: 03/12/2015

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Wang To: "dev@kafka.apache.org" Date: 03/12/2015 09:39 AM Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations Folks, Just want to elaborate a bit more on the create-topic metadata and batching describe-topic based on config

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Folks, Just want to elaborate a bit more on the create-topic metadata and batching describe-topic based on config / metadata in my previous email as we work on KAFKA-1694. The main motivation is to have some sort of topic management mechanisms, which I think is quite important in a multi-tenant /

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Thanks for the updated wiki. A few comments below: 1. Error description in response: I think if some errorCode could indicate several different error cases then we should really change it to multiple codes. In general the errorCode itself would be precise and sufficient for describing the server s

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Thanks for sending that out Joe - I don't think I will be able to make it today, so if notes can be sent out afterward that would be great. On Mon, Mar 02, 2015 at 09:16:13AM -0800, Gwen Shapira wrote: > Thanks for sending this out Joe. Looking forward to chatting with everyone :) > > On Mon, Mar

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

It would also be good to think through how we can use those new admin requests in the producer/consumer client as well. Currently, both the producer and the consumer use TopicMetadataRequest to obtain the metadata, which will trigger a topic creation if auto topic creation is enabled. This is a bit

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Thanks for sending this out Joe. Looking forward to chatting with everyone :) On Mon, Mar 2, 2015 at 6:46 AM, Joe Stein wrote: > Hey, I just sent out a google hangout invite to all pmc, committers and > everyone I found working on a KIP. If I missed anyone in the invite please > let me know and c

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Hey, I just sent out a google hangout invite to all pmc, committers and everyone I found working on a KIP. If I missed anyone in the invite please let me know and can update it, np. We should do this every Tuesday @ 2pm Eastern Time. Maybe we can get INFRA help to make a google account so we can m

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Let's stay on Google hangouts that will also record and make the sessions available on youtube. -Jay On Tue, Feb 24, 2015 at 11:49 AM, Jeff Holoman wrote: > Jay / Joe > > We're happy to send out a Webex for this purpose. We could record the > sessions if there is interest and publish them out.

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Jay / Joe We're happy to send out a Webex for this purpose. We could record the sessions if there is interest and publish them out. Thanks Jeff On Tue, Feb 24, 2015 at 11:28 AM, Jay Kreps wrote: > Let's try to get the technical hang-ups sorted out, though. I really think > there is some benef

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Let's try to get the technical hang-ups sorted out, though. I really think there is some benefit to live discussion vs writing. I am hopeful that if we post instructions and give ourselves a few attempts we can get it working. Tuesday at that time would work for me...any objections? -Jay On Tue,

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Weekly would be great maybe like every Tuesday ~ 1pm ET / 10am PT I don't mind google hangout but there is always some issue or whatever so we know the apache irc channel works. We can start there and see how it goes? We can pull transcripts too and associate to tickets if need be makes it he

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

We'd talked about doing a Google Hangout to chat about this. What about generalizing that a little further...I actually think it would be good for everyone spending a reasonable chunk of their week on Kafka stuff to maybe sync up once a week. I think we could use time to talk through design stuff,

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Hi all, I've updated KIP page, fixed / aligned document structure. Also I added some very initial proposal for AdminClient so we have something to start from while discussing the KIP. https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Jay, Re error messages: you are right, in most cases client will have enough context to show descriptive error message. My concern is that we will have to add lots of new error codes for each possible error. Of course, we could reuse some of existing like UknownTopicOrPartitionCode, but we will al

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Hey Andrii, Generally we can do good error handling without needing custom server-side messages. I.e. generally the client has the context to know that if it got an error that the topic doesn't exist to say "Topic X doesn't exist" rather than "error code 14" (or whatever). Maybe there are specific

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Hi all, I'm trying to address some of the issues which were mentioned earlier about Admin RQ/RP format. One of those was about batching operations. What if we follow TopicCommand approach and let people specify topic-name by regexp - would that cover most of the use cases? Secondly, is what infor

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

@Guozhang: Thanks for your comments, I've answered some of those. The main thing is having merged request for create-alter-delete-describe - I have some concerns about this approach. @*Jay*: I see that introduced ClusterMetadaRequest is also one of the concerns. We can solve it if we implement re-

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

I think for the topics commands we can actually merge create/alter/delete/describe as one request type since their formats are very much similar, and keep list-topics and others like partition-reassignment / preferred-leader-election as separate request types, I also left some other comments on the

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

Yeah I totally agree that we don't want to just have one "do admin stuff" command that has the union of all parameters. What I am saying is that command line tools are one client of the administrative apis, but these will be used in a number of scenarios so they should make logical sense even in t

  1   2   >