It seems that the links to images in the KIP are broken. Liquan
On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt < brahmbhatt.pa...@gmail.com> wrote: > 110. What does getDelegationTokenAs mean? > In the current proposal we only allow a user to get delegation token for > the identity that it authenticated as using another mechanism, i.e. A user > that authenticate using a keytab for principal us...@example.com will get > delegation tokens for that user only. In future I think we will have to > extend support such that we allow some set of users ( > kafka-rest-u...@example.com, storm-nim...@example.com) to acquire > delegation tokens on behalf of other users whose identity they have > verified independently. Kafka brokers will have ACLs to control which > users are allowed to impersonate other users and get tokens on behalf of > them. Overall Impersonation is a whole different problem in my opinion and > I think we can tackle it in separate KIP. > > 111. What's the typical rate of getting and renewing delegation tokens? > Typically this should be very very low, 1 request per minute is a > relatively high estimate. However it depends on the token expiration. I am > less worried about the extra load it puts on controller vs the added > complexity and the value it offers. > > Thanks > Parth > > > > On Tue, May 24, 2016 at 7:30 AM, Ismael Juma <ism...@juma.me.uk> wrote: > > > Thanks Rajini. It would probably require a separate KIP as it will > > introduce user visible changes. We could also update KIP-48 to have this > > information, but it seems cleaner to do it separately. We can discuss > that > > in the KIP call today. > > > > Ismael > > > > On Tue, May 24, 2016 at 3:19 PM, Rajini Sivaram < > > rajinisiva...@googlemail.com> wrote: > > > > > Ismael, > > > > > > I have created a JIRA ( > https://issues.apache.org/jira/browse/KAFKA-3751) > > > for adding SCRAM as a SASL mechanism. Would that need another KIP? If > > > KIP-48 will use this mechanism, can this just be a JIRA that gets > > reviewed > > > when the PR is ready? > > > > > > Thank you, > > > > > > Rajini > > > > > > On Tue, May 24, 2016 at 2:46 PM, Ismael Juma <ism...@juma.me.uk> > wrote: > > > > > > > Thanks Rajini, SCRAM seems like a good candidate. > > > > > > > > Gwen had independently mentioned this as a SASL mechanism that might > be > > > > useful for Kafka and I have been meaning to check it in more detail. > > Good > > > > to know that you are willing to contribute an implementation. Maybe > we > > > > should file a separate JIRA for this? > > > > > > > > Ismael > > > > > > > > On Tue, May 24, 2016 at 2:12 PM, Rajini Sivaram < > > > > rajinisiva...@googlemail.com> wrote: > > > > > > > > > SCRAM (Salted Challenge Response Authentication Mechanism) is a > > better > > > > > mechanism than Digest-MD5. Java doesn't come with a built-in SCRAM > > > > > SaslServer or SaslClient, but I will be happy to add support in > Kafka > > > > since > > > > > it would be a useful mechanism to support anyway. > > > > > https://tools.ietf.org/html/rfc7677 describes the protocol for > > > > > SCRAM-SHA-256. > > > > > > > > > > On Tue, May 24, 2016 at 2:37 AM, Jun Rao <j...@confluent.io> wrote: > > > > > > > > > > > Parth, > > > > > > > > > > > > Thanks for the explanation. A couple of more questions. > > > > > > > > > > > > 110. What does getDelegationTokenAs mean? > > > > > > > > > > > > 111. What's the typical rate of getting and renewing delegation > > > tokens? > > > > > > That may have an impact on whether they should be directed to the > > > > > > controller. > > > > > > > > > > > > Jun > > > > > > > > > > > > On Mon, May 23, 2016 at 1:19 PM, parth brahmbhatt < > > > > > > brahmbhatt.pa...@gmail.com> wrote: > > > > > > > > > > > > > Hi Jun, > > > > > > > > > > > > > > Thanks for reviewing. > > > > > > > > > > > > > > * We could add a Cluster action to add acls on who can request > > > > > delegation > > > > > > > tokens. I don't see the use case for that yet but down the line > > > when > > > > we > > > > > > > start supporting getDelegationTokenAs it will be necessary. > > > > > > > * Yes we recommend tokens to be only used/distributed over > secure > > > > > > channels. > > > > > > > * Depending on what design we end up choosing Invalidation will > > be > > > > > > > responsibility of every broker or controller. > > > > > > > * I am not sure if I documented somewhere that invalidation > will > > > > > directly > > > > > > > go through zookeeper but that is not the intent. Invalidation > > will > > > > > either > > > > > > > be request based or due to expiration. No direct zookeeper > > > > interaction > > > > > > from > > > > > > > any client. > > > > > > > * "Broker also stores the DelegationToken without the hmac in > the > > > > > > > zookeeper." : Sorry about the confusion. The sole purpose of > > > > zookeeper > > > > > in > > > > > > > this design is as distribution channel for tokens between all > > > brokers > > > > > > and a > > > > > > > layer that ensures only tokens that were generated by making a > > > > request > > > > > > to a > > > > > > > broker will be accepted (more on this in second paragraph). The > > > token > > > > > > > consists of few elements (owner, renewer, uuid , expiration, > > hmac) > > > , > > > > > one > > > > > > of > > > > > > > which is the finally generated hmac but hmac it self is > derivable > > > if > > > > > you > > > > > > > have all the other elements of the token + secret key to > generate > > > > hmac. > > > > > > > Given zookeeper does not provide SSL support we do not want the > > > > entire > > > > > > > token to be wire transferred to zookeeper as that will be an > > > insecure > > > > > > wire > > > > > > > transfer. Instead we only store all the other elements of a > > > > delegation > > > > > > > tokens. Brokers can read these elements and because they also > > have > > > > > access > > > > > > > to secret key they will be able to generate hmac on their end. > > > > > > > > > > > > > > One of the alternative proposed is to avoid zookeeper > > altogether. A > > > > > > Client > > > > > > > will call broker with required information (owner, renwer, > > > > expiration) > > > > > > and > > > > > > > get back (signed hmac, uuid). Broker won't store this in > > zookeeper. > > > > > From > > > > > > > this point a client can contact any broker with all the > > delegation > > > > > token > > > > > > > info (owner, rewner, expiration, hmac, uuid) the borker will > > > > regenerate > > > > > > the > > > > > > > hmac and as long as it matches with hmac presented by client , > > > broker > > > > > > will > > > > > > > allow the request to authenticate. Only problem with this > > approach > > > > is > > > > > if > > > > > > > the secret key is compromised any client can now generate > random > > > > tokens > > > > > > and > > > > > > > they will still be able to authenticate as any user they like. > > with > > > > > > > zookeeper we guarantee that only tokens acquired via a broker > > > (using > > > > > some > > > > > > > auth scheme other than delegation token) will be accepted. We > > need > > > to > > > > > > > discuss which proposal makes more sense and we can go over it > in > > > > > > tomorrow's > > > > > > > meeting. > > > > > > > > > > > > > > Also, can you forward the invite to me? > > > > > > > > > > > > > > Thanks > > > > > > > Parth > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, May 23, 2016 at 10:35 AM, Jun Rao <j...@confluent.io> > > > wrote: > > > > > > > > > > > > > > > Thanks for the KIP. A few comments. > > > > > > > > > > > > > > > > 100. This potentially can be useful for Kafka Connect and > Kafka > > > > rest > > > > > > > proxy > > > > > > > > where a worker agent will need to run a task on behalf of a > > > client. > > > > > We > > > > > > > will > > > > > > > > likely need to change how those services use Kafka clients > > > > > > > > (producer/consumer). Instead of a shared client per worker, > we > > > will > > > > > > need > > > > > > > a > > > > > > > > client per user task since the authentication happens at the > > > > > connection > > > > > > > > level. For Kafka Connect, the renewer will be the workers. > So, > > we > > > > > > > probably > > > > > > > > need to allow multiple renewers. For Kafka rest proxy, the > > > renewer > > > > > can > > > > > > > > probably just be the creator of the token. > > > > > > > > > > > > > > > > 101. Do we need new acl on who can request delegation tokens? > > > > > > > > > > > > > > > > 102. Do we recommend people to send delegation tokens in an > > > > encrypted > > > > > > > > channel? > > > > > > > > > > > > > > > > 103. Who is responsible for expiring tokens, every broker? > > > > > > > > > > > > > > > > 104. For invalidating tokens, would it be better to do it in > a > > > > > request > > > > > > > > instead of going to ZK directly? > > > > > > > > > > > > > > > > 105. The terminology of client in the wiki sometimes refers > to > > > the > > > > > end > > > > > > > > client and some other times refers to the client using the > > > > delegation > > > > > > > > tokens. It would be useful to distinguish between the two. > > > > > > > > > > > > > > > > 106. Could you explain the sentence "Broker also stores the > > > > > > > DelegationToken > > > > > > > > without the hmac in the zookeeper." a bit more? I thought the > > > > > > delegation > > > > > > > > token is the hmac. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > On Mon, May 23, 2016 at 9:22 AM, Jun Rao <j...@confluent.io> > > > wrote: > > > > > > > > > > > > > > > > > Hi, Harsha, > > > > > > > > > > > > > > > > > > Just sent out a KIP meeting invite. We can discuss this in > > the > > > > > > meeting > > > > > > > > > tomorrow. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > On Thu, May 19, 2016 at 8:47 AM, Harsha <ka...@harsha.io> > > > wrote: > > > > > > > > > > > > > > > > > >> Hi All, > > > > > > > > >> Can we have a KIP meeting around this. The KIP > is > > > up > > > > > for > > > > > > > > >> sometime and if there are any questions lets > > > quickly > > > > > hash > > > > > > > out > > > > > > > > >> details. > > > > > > > > >> > > > > > > > > >> Thanks, > > > > > > > > >> Harsha > > > > > > > > >> > > > > > > > > >> On Thu, May 19, 2016, at 08:40 AM, parth brahmbhatt wrote: > > > > > > > > >> > That is what the hadoop echo system uses so no good > reason > > > > > really. > > > > > > > We > > > > > > > > >> > could > > > > > > > > >> > change it to whatever is the newest recommended standard > > is. > > > > > > > > >> > > > > > > > > > >> > Thanks > > > > > > > > >> > Parth > > > > > > > > >> > > > > > > > > > >> > On Thu, May 19, 2016 at 3:33 AM, Ismael Juma < > > > > ism...@juma.me.uk > > > > > > > > > > > > > > wrote: > > > > > > > > >> > > > > > > > > > >> > > Hi Parth, > > > > > > > > >> > > > > > > > > > > >> > > Thanks for the KIP. I only started reviewing this and > > may > > > > have > > > > > > > > >> additional > > > > > > > > >> > > questions later. The immediate question that came to > > mind > > > is > > > > > our > > > > > > > > >> choice of > > > > > > > > >> > > "DIGEST-MD5" even though it's marked as OBSOLETE in > the > > > IANA > > > > > > > > Registry > > > > > > > > >> of > > > > > > > > >> > > SASL mechanisms and the original RFC (2831) has been > > moved > > > > to > > > > > > > > Historic > > > > > > > > >> > > status: > > > > > > > > >> > > > > > > > > > > >> > > https://tools.ietf.org/html/rfc6331 > > > > > > > > >> > > > > > > > > > > > > > > > > > http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml > > > > > > > > >> > > > > > > > > > > >> > > What is the reasoning behind that choice? > > > > > > > > >> > > > > > > > > > > >> > > Thanks, > > > > > > > > >> > > Ismael > > > > > > > > >> > > > > > > > > > > >> > > On Fri, May 13, 2016 at 11:29 PM, Gwen Shapira < > > > > > > g...@confluent.io > > > > > > > > > > > > > > > > >> wrote: > > > > > > > > >> > > > > > > > > > > >> > > > Also comments inline :) > > > > > > > > >> > > > > > > > > > > > >> > > > > * I want to emphasize that even though delegation > > > tokens > > > > > > are a > > > > > > > > >> Hadoop > > > > > > > > >> > > > > innovation, I feel very strongly about not adding > > > > > dependency > > > > > > > on > > > > > > > > >> Hadoop > > > > > > > > >> > > > > when implementing delegation tokens for Kafka. The > > KIP > > > > > > doesn't > > > > > > > > >> imply > > > > > > > > >> > > > > such dependency, but if you can clarify... > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > *No hadoop dependency.* > > > > > > > > >> > > > > > > > > > > > >> > > > Yay! Just add this to the KIP so no one will read > the > > > KIP > > > > > and > > > > > > > > panic > > > > > > > > >> > > > three weeks before the next release... > > > > > > > > >> > > > > > > > > > > > >> > > > > * Can we get delegation token at any time after > > > > > > > authenticating? > > > > > > > > >> only > > > > > > > > >> > > > > immediately after? > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > *As long as you are authenticated you can get > > > delegation > > > > > > > tokens. > > > > > > > > >> We > > > > > > > > >> > > need > > > > > > > > >> > > > to > > > > > > > > >> > > > > discuss if a client authenticated using delegation > > > > token, > > > > > > can > > > > > > > > also > > > > > > > > >> > > > acquire > > > > > > > > >> > > > > delegation token again or not. Also there is the > > > > question > > > > > of > > > > > > > do > > > > > > > > we > > > > > > > > >> > > allow > > > > > > > > >> > > > > anyone to acquire delegation token or we want > > specific > > > > > ACLs > > > > > > (I > > > > > > > > >> think > > > > > > > > >> > > its > > > > > > > > >> > > > an > > > > > > > > >> > > > > overkill.)* > > > > > > > > >> > > > > > > > > > > > >> > > > I agree that ACLs is an overkill. > > > > > > > > >> > > > > > > > > > > > >> > > > I think we are debating two options: Either require > > > > Kerberos > > > > > > > auth > > > > > > > > >> for > > > > > > > > >> > > > renewal or require non-owners to renew. > > > > > > > > >> > > > I *think* the latter is simpler (it basically > require > > a > > > > "job > > > > > > > > master" > > > > > > > > >> > > > to take responsibility for the renewal, it will have > > its > > > > own > > > > > > > > >> identity > > > > > > > > >> > > > anyway and I think this is the correct design > pattern > > > > > anyway. > > > > > > > For > > > > > > > > >> > > > storm, I'd expect Nimbus to coordinate renewals?), > but > > > it > > > > is > > > > > > > hard > > > > > > > > to > > > > > > > > >> > > > debate simplicity without looking at the code > changes > > > > > > required. > > > > > > > If > > > > > > > > >> you > > > > > > > > >> > > > have a draft of how the "require Kerberos" will look > > in > > > > > Kafka > > > > > > > > code, > > > > > > > > >> > > > I'll be happy to take a look. > > > > > > > > >> > > > > > > > > > > > >> > > > > * My understanding is that tokens will propagate > via > > > ZK > > > > > but > > > > > > > > >> without > > > > > > > > >> > > > > additional changes to UpdateMetadata protocol, > > > correct? > > > > > > > Clients > > > > > > > > >> > > > > currently don't retry on SASL auth failure (IIRC), > > but > > > > > since > > > > > > > the > > > > > > > > >> > > > > tokens propagate between brokers asynch, we will > > need > > > to > > > > > > > retry a > > > > > > > > >> bit > > > > > > > > >> > > > > to avoid clients failing auth due to timing > issues. > > > > > > > > >> > > > > > > > > > > > > >> > > > > *I am considering 2 alternatives right now. The > > > current > > > > > > > > documented > > > > > > > > >> > > > approach > > > > > > > > >> > > > > is zookeeper based and it does not require any > > changes > > > > to > > > > > > > > >> > > UpdateMetadata > > > > > > > > >> > > > > protocol. An alternative approach can remove > > zookeeper > > > > > > > > dependency > > > > > > > > >> as > > > > > > > > >> > > well > > > > > > > > >> > > > > but we can discuss that in KIP discussion call.* > > > > > > > > >> > > > > > > > > > > > >> > > > Oooh! Sounds interesting. Do you want to ping Jun to > > > > > arrange a > > > > > > > > call? > > > > > > > > >> > > > > > > > > > > > >> > > > > * I liked Ashish's suggestion of having just the > > > > > controller > > > > > > > > issue > > > > > > > > >> the > > > > > > > > >> > > > > delegation tokens, to avoid syncing a shared > secret. > > > Not > > > > > > sure > > > > > > > if > > > > > > > > >> we > > > > > > > > >> > > > > want to continue the discussion here or on the > > wiki. I > > > > > think > > > > > > > > that > > > > > > > > >> we > > > > > > > > >> > > > > can decouple the problem of "token distribution" > > from > > > > > > "shared > > > > > > > > >> secret > > > > > > > > >> > > > > distribution" and use the controller as the only > > token > > > > > > > generator > > > > > > > > >> to > > > > > > > > >> > > > > solve the second issue, while still using ZK async > > to > > > > > > > distribute > > > > > > > > >> > > > > tokens. > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > *As mentioned in the previous Email I am fine with > > > that > > > > > > > approach > > > > > > > > >> as > > > > > > > > >> > > long > > > > > > > > >> > > > as > > > > > > > > >> > > > > we agree that the extra complexity of > > adding/updating > > > > APIS > > > > > > > adds > > > > > > > > >> enough > > > > > > > > >> > > > > value. The advantage with the controller approach > is > > > > > secret > > > > > > > > >> rotation > > > > > > > > >> > > can > > > > > > > > >> > > > be > > > > > > > > >> > > > > automated,frequent and would not require > > deployment. * > > > > > > > > >> > > > > > > > > > > > >> > > > Can you detail the extra complexity (or point me to > > the > > > > > email > > > > > > I > > > > > > > > >> > > > missed?) - which Apis are required? > > > > > > > > >> > > > As far as I can tell, clients can already find the > > > > > controller > > > > > > > from > > > > > > > > >> > > > metadata. I'm a bit more concerned about controller > > > load. > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > * While I like the idea of forcing kerberos auth > for > > > > > > renwal, I > > > > > > > > >> think > > > > > > > > >> > > > > it mixes the transport layer the the request > content > > > in > > > > a > > > > > > > pretty > > > > > > > > >> ugly > > > > > > > > >> > > > > way. Perhaps limiting renewer to non-owner is > > better. > > > > > > > > >> > > > > > > > > > > > > >> > > > > *I feel this is a necessary evil. While this will > > make > > > > the > > > > > > > kafka > > > > > > > > >> code > > > > > > > > >> > > > > pretty straight forward , forcing renewer to > > > non-owner > > > > > > pushes > > > > > > > > >> the code > > > > > > > > >> > > > > ugliness to client and makes it even harder to > > > > > integrate. * > > > > > > > > >> > > > > > > > > > > > >> > > > As mentioned before, I don't think the "renewal by > > > other" > > > > > > > approach > > > > > > > > >> is > > > > > > > > >> > > > that ugly for the clients we expect to use > delegation > > > > tokens > > > > > > > since > > > > > > > > >> > > > they will have an app-master of some sort who > > requested > > > > the > > > > > > > token > > > > > > > > to > > > > > > > > >> > > > begin with. > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > The response for my question on how multiple > > > identities > > > > > will > > > > > > > be > > > > > > > > >> > > > > handled wasn't super clear to me - AFAIK, we don't > > > > > > > authenticate > > > > > > > > >> each > > > > > > > > >> > > > > request, we authenticate connections. > > > > > > > > >> > > > > > > > > > > > > >> > > > > *We authenticate connections, and only when they > are > > > > being > > > > > > > > >> established. > > > > > > > > >> > > > Let > > > > > > > > >> > > > > me try to phrase this as a question, in absence of > > > > > > delegation > > > > > > > > >> tokens if > > > > > > > > >> > > > we > > > > > > > > >> > > > > had to support the use case using user TGT's how > > would > > > > we > > > > > do > > > > > > > it? > > > > > > > > >> My > > > > > > > > >> > > point > > > > > > > > >> > > > > was it would be no different with delegation > tokens. > > > The > > > > > use > > > > > > > > case > > > > > > > > >> you > > > > > > > > >> > > are > > > > > > > > >> > > > > describing seems more like impersonation.* > > > > > > > > >> > > > > > > > > > > > >> > > > Yeah, I thought that one of the things that > delegation > > > > > tokens > > > > > > > > >> handled. > > > > > > > > >> > > > Maybe I got it wrong :) > > > > > > > > >> > > > > > > > > > > > >> > > > Thanks for the detailed answers. > > > > > > > > >> > > > > > > > > > > > >> > > > Gwen > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > Thanks > > > > > > > > >> > > > > Parth > > > > > > > > >> > > > > > > > > > > > > >> > > > > On Fri, May 13, 2016 at 12:19 AM, Gwen Shapira < > > > > > > > > g...@confluent.io > > > > > > > > >> > > > > > > > > > >> > > > wrote: > > > > > > > > >> > > > > > > > > > > > > >> > > > >> Hi Parth and Harsha, > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> Few more comments: > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> * The API / RequestResponse section doesn't seem > to > > > > have > > > > > > good > > > > > > > > >> > > > >> description of the changes to the Kafka Protocol. > > > > Sounds > > > > > > like > > > > > > > > >> you are > > > > > > > > >> > > > >> proposing new DelegationTokenRequest and > > > > > RenewTokenRequest > > > > > > > (and > > > > > > > > >> > > > >> matching responses), without detailing the > contents > > > of > > > > > the > > > > > > > > >> requests > > > > > > > > >> > > > >> and responses? Or rather, you show the class > > > interface, > > > > > but > > > > > > > not > > > > > > > > >> the > > > > > > > > >> > > > >> underlying protocol. This could be seen as an > > > > > > implementation > > > > > > > > >> detail, > > > > > > > > >> > > > >> but since the binary protocol is what we provide > to > > > > > > non-Java > > > > > > > > >> clients, > > > > > > > > >> > > > >> we need to show the changes there. > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> * getDelegationToken sounds like > > > > > > > delegationTokenRequestHandler? > > > > > > > > >> Is it > > > > > > > > >> > > > >> planned to be part of KafkaApi? or Client? Its > > > > unclear... > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> * I want to emphasize that even though delegation > > > > tokens > > > > > > are > > > > > > > a > > > > > > > > >> Hadoop > > > > > > > > >> > > > >> innovation, I feel very strongly about not adding > > > > > > dependency > > > > > > > on > > > > > > > > >> Hadoop > > > > > > > > >> > > > >> when implementing delegation tokens for Kafka. > The > > > KIP > > > > > > > doesn't > > > > > > > > >> imply > > > > > > > > >> > > > >> such dependency, but if you can clarify... > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> * Can we get delegation token at any time after > > > > > > > authenticating? > > > > > > > > >> only > > > > > > > > >> > > > >> immediately after? > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> * My understanding is that tokens will propagate > > via > > > ZK > > > > > but > > > > > > > > >> without > > > > > > > > >> > > > >> additional changes to UpdateMetadata protocol, > > > correct? > > > > > > > Clients > > > > > > > > >> > > > >> currently don't retry on SASL auth failure > (IIRC), > > > but > > > > > > since > > > > > > > > the > > > > > > > > >> > > > >> tokens propagate between brokers asynch, we will > > need > > > > to > > > > > > > retry > > > > > > > > a > > > > > > > > >> bit > > > > > > > > >> > > > >> to avoid clients failing auth due to timing > issues. > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> * Strongly agreeing on clients not touching ZK > > > directly > > > > > :) > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> * I liked Ashish's suggestion of having just the > > > > > controller > > > > > > > > >> issue the > > > > > > > > >> > > > >> delegation tokens, to avoid syncing a shared > > secret. > > > > Not > > > > > > sure > > > > > > > > if > > > > > > > > >> we > > > > > > > > >> > > > >> want to continue the discussion here or on the > > wiki. > > > I > > > > > > think > > > > > > > > >> that we > > > > > > > > >> > > > >> can decouple the problem of "token distribution" > > from > > > > > > "shared > > > > > > > > >> secret > > > > > > > > >> > > > >> distribution" and use the controller as the only > > > token > > > > > > > > generator > > > > > > > > >> to > > > > > > > > >> > > > >> solve the second issue, while still using ZK > async > > to > > > > > > > > distribute > > > > > > > > >> > > > >> tokens. > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> * I am also uncomfortable with infinite lifetime > of > > > > > tokens > > > > > > > (and > > > > > > > > >> hoped > > > > > > > > >> > > > >> to hear from others in the community) - but > having > > > the > > > > > > option > > > > > > > > and > > > > > > > > >> > > > >> documenting it as less secure, allows users to > > > > configure > > > > > > > their > > > > > > > > >> system > > > > > > > > >> > > > >> as the wish. > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> * While I like the idea of forcing kerberos auth > > for > > > > > > renwal, > > > > > > > I > > > > > > > > >> think > > > > > > > > >> > > > >> it mixes the transport layer the the request > > content > > > > in a > > > > > > > > pretty > > > > > > > > >> ugly > > > > > > > > >> > > > >> way. Perhaps limiting renewer to non-owner is > > better. > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> Things I'd still like to see: > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> * More detailed explanation on what we plan to do > > for > > > > the > > > > > > > java > > > > > > > > >> clients > > > > > > > > >> > > > >> specifically - new configuration? new APIs? > > > > > > > > >> > > > >> The response for my question on how multiple > > > identities > > > > > > will > > > > > > > be > > > > > > > > >> > > > >> handled wasn't super clear to me - AFAIK, we > don't > > > > > > > authenticate > > > > > > > > >> each > > > > > > > > >> > > > >> request, we authenticate connections. > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> * Alternatives: Delegation tokens are only used > in > > > the > > > > > > Hadoop > > > > > > > > >> > > > >> ecosystem. I'm wondering if there are > alternatives > > in > > > > > other > > > > > > > > >> ecosystems > > > > > > > > >> > > > >> (Mesos? Tachyon? Cassandra?) and whether there > are > > > some > > > > > > > > >> advantages > > > > > > > > >> > > > >> there. > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> Gwen > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> On Thu, May 12, 2016 at 1:05 PM, Harsha < > > > > ka...@harsha.io > > > > > > > > > > > > > > wrote: > > > > > > > > >> > > > >> > Hi Gwen, > > > > > > > > >> > > > >> > Can you look at Parth's last reply. > > Does > > > > it > > > > > > > answer > > > > > > > > >> your > > > > > > > > >> > > > >> > concerns. > > > > > > > > >> > > > >> > > > > > > > > > >> > > > >> > Thanks, > > > > > > > > >> > > > >> > Harsha > > > > > > > > >> > > > >> > > > > > > > > > >> > > > >> > On Wed, May 4, 2016, at 09:25 AM, parth > > brahmbhatt > > > > > wrote: > > > > > > > > >> > > > >> >> Thanks for reviewing Gwen. The wiki already > has > > > > > details > > > > > > on > > > > > > > > >> token > > > > > > > > >> > > > >> >> expiration > > > > > > > > >> > > > >> >> under token acquisition process > > > > > > > > >> > > > >> >> < > > > > > > > > >> > > > >> > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition > > > > > > > > >> > > > >> >. > > > > > > > > >> > > > >> >> Current proposal is that tokens will expire > > based > > > > on a > > > > > > > > server > > > > > > > > >> side > > > > > > > > >> > > > >> >> configuration (default 24 hours) unless > renewed. > > > > > Renewal > > > > > > > is > > > > > > > > >> only > > > > > > > > >> > > > allowed > > > > > > > > >> > > > >> >> until the max life time of token. > Alternatively > > we > > > > > could > > > > > > > > also > > > > > > > > >> make > > > > > > > > >> > > > that > > > > > > > > >> > > > >> >> an > > > > > > > > >> > > > >> >> optional param and the server side default can > > > serve > > > > > as > > > > > > > the > > > > > > > > >> upper > > > > > > > > >> > > > bound. > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > >> >> To your second point it will be done exactly > the > > > > same > > > > > > way > > > > > > > we > > > > > > > > >> would > > > > > > > > >> > > > >> >> support > > > > > > > > >> > > > >> >> multiple keytabs. The calling client will have > > to > > > > put > > > > > > the > > > > > > > > >> tokens it > > > > > > > > >> > > > >> wants > > > > > > > > >> > > > >> >> to use in the subject instance and call > > > > > produce/consume > > > > > > > > inside > > > > > > > > >> > > > >> >> subject.doas. Each caller will have to keep > > track > > > of > > > > > its > > > > > > > own > > > > > > > > >> > > > subject. I > > > > > > > > >> > > > >> >> will have to look at the code to see if we > > support > > > > > this > > > > > > > > >> feature > > > > > > > > >> > > right > > > > > > > > >> > > > >> now > > > > > > > > >> > > > >> >> but my understanding is delegation token > > shouldn't > > > > > need > > > > > > > any > > > > > > > > >> special > > > > > > > > >> > > > >> >> treatment as its just another type of > Credential > > > in > > > > > the > > > > > > > > >> subject. > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > >> >> I would also like to know what is your opinion > > > about > > > > > > > > infinite > > > > > > > > >> > > renewal > > > > > > > > >> > > > >> (my > > > > > > > > >> > > > >> >> recommendation is to not support this), tokens > > > > > renewing > > > > > > > them > > > > > > > > >> > > self(my > > > > > > > > >> > > > >> >> recommendation is to not support this) and > most > > > > > > > importantly > > > > > > > > >> your > > > > > > > > >> > > > choice > > > > > > > > >> > > > >> >> between the alternatives listed on this thread > > > > > > > > >> > > > >> >> < > > > > > > > > >> > > > >> > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism > > > > > > > > >> > > > >> > > > > > > > > > >> > > > >> >> ( I am leaning towards the alternative-2 minus > > > > > > controller > > > > > > > > >> > > > distributing > > > > > > > > >> > > > >> >> secret). Thanks again for reviewing. > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > >> >> Thanks > > > > > > > > >> > > > >> >> Parth > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira < > > > > > > > > >> g...@confluent.io> > > > > > > > > >> > > > wrote: > > > > > > > > >> > > > >> >> > > > > > > > > >> > > > >> >> > Harsha, > > > > > > > > >> > > > >> >> > > > > > > > > > >> > > > >> >> > I was thinking of the Rest Proxy. I didn't > see > > > > your > > > > > > > design > > > > > > > > >> yet, > > > > > > > > >> > > > but in > > > > > > > > >> > > > >> >> > our proxy, we have a set of producers, which > > > will > > > > > > serve > > > > > > > > >> multiple > > > > > > > > >> > > > users > > > > > > > > >> > > > >> >> > going through the proxy. Since these users > > will > > > > have > > > > > > > > >> different > > > > > > > > >> > > > >> >> > privileges, they'll need to authenticate > > > > separately, > > > > > > and > > > > > > > > >> can't > > > > > > > > >> > > > share a > > > > > > > > >> > > > >> >> > token. > > > > > > > > >> > > > >> >> > > > > > > > > > >> > > > >> >> > Am I missing anything? > > > > > > > > >> > > > >> >> > > > > > > > > > >> > > > >> >> > Gwen > > > > > > > > >> > > > >> >> > > > > > > > > > >> > > > >> >> > On Tue, May 3, 2016 at 2:11 PM, Harsha < > > > > > > ka...@harsha.io > > > > > > > > > > > > > > > > >> wrote: > > > > > > > > >> > > > >> >> > > Gwen, > > > > > > > > >> > > > >> >> > > On your second point. Can you > > > > describe > > > > > a > > > > > > > > >> usecase > > > > > > > > >> > > where > > > > > > > > >> > > > >> >> > > mutliple clients ended up > > sharing a > > > > > > > producer > > > > > > > > >> and > > > > > > > > >> > > even > > > > > > > > >> > > > if > > > > > > > > >> > > > >> they > > > > > > > > >> > > > >> >> > > do why can't they not use > single > > > > token > > > > > > that > > > > > > > > >> producer > > > > > > > > >> > > > >> >> > > captures. Why would we need > > > multiple > > > > > > > clients > > > > > > > > >> with > > > > > > > > >> > > > >> different > > > > > > > > >> > > > >> >> > > tokens sharing a single > instance > > of > > > > > > > producer. > > > > > > > > >> Also > > > > > > > > >> > > in > > > > > > > > >> > > > >> this > > > > > > > > >> > > > >> >> > > case other clients have access > > all > > > > the > > > > > > > tokens > > > > > > > > >> no? > > > > > > > > >> > > > >> >> > > > > > > > > > > >> > > > >> >> > > Thanks, > > > > > > > > >> > > > >> >> > > Harsha > > > > > > > > >> > > > >> >> > > > > > > > > > > >> > > > >> >> > > > > > > > > > > >> > > > >> >> > > On Tue, May 3, 2016, at 11:49 AM, Gwen > > Shapira > > > > > > wrote: > > > > > > > > >> > > > >> >> > >> Sorry for the delay: > > > > > > > > >> > > > >> >> > >> > > > > > > > > >> > > > >> >> > >> Two questions that we didn't see in the > > wiki: > > > > > > > > >> > > > >> >> > >> 1. Is there an expiration for delegation > > > > tokens? > > > > > > > > >> Renewal? How > > > > > > > > >> > > > do we > > > > > > > > >> > > > >> >> > >> revoke them? > > > > > > > > >> > > > >> >> > >> 2. If we want to use delegation tokens > for > > > > > "do-as" > > > > > > > > (say, > > > > > > > > >> > > submit > > > > > > > > >> > > > >> Storm > > > > > > > > >> > > > >> >> > >> job as my user), we will need a producer > > for > > > > > every > > > > > > > job > > > > > > > > >> (we > > > > > > > > >> > > can't > > > > > > > > >> > > > >> share > > > > > > > > >> > > > >> >> > >> them between multiple jobs running on > same > > > > node), > > > > > > > since > > > > > > > > >> we > > > > > > > > >> > > only > > > > > > > > >> > > > >> >> > >> authenticate when connecting. Is there a > > plan > > > > to > > > > > > > change > > > > > > > > >> this > > > > > > > > >> > > for > > > > > > > > >> > > > >> >> > >> delegation tokens, in order to allow > > multiple > > > > > users > > > > > > > > with > > > > > > > > >> > > > different > > > > > > > > >> > > > >> >> > >> tokens to share a client? > > > > > > > > >> > > > >> >> > >> > > > > > > > > >> > > > >> >> > >> Gwen > > > > > > > > >> > > > >> >> > >> > > > > > > > > >> > > > >> >> > >> On Tue, May 3, 2016 at 9:12 AM, parth > > > > brahmbhatt > > > > > > > > >> > > > >> >> > >> <brahmbhatt.pa...@gmail.com> wrote: > > > > > > > > >> > > > >> >> > >> > Bumping this up one more time, can > other > > > > > > committers > > > > > > > > >> review? > > > > > > > > >> > > > >> >> > >> > > > > > > > > > >> > > > >> >> > >> > Thanks > > > > > > > > >> > > > >> >> > >> > Parth > > > > > > > > >> > > > >> >> > >> > > > > > > > > > >> > > > >> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM, > Harsha < > > > > > > > > >> ka...@harsha.io> > > > > > > > > >> > > > wrote: > > > > > > > > >> > > > >> >> > >> > > > > > > > > > >> > > > >> >> > >> >> Parth, > > > > > > > > >> > > > >> >> > >> >> Overall current design looks > > > good > > > > to > > > > > > > me. I > > > > > > > > >> am +1 > > > > > > > > >> > > on > > > > > > > > >> > > > >> the > > > > > > > > >> > > > >> >> > KIP. > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > >> > > > >> >> > >> >> Gwen , Jun can you review this as > well. > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > >> > > > >> >> > >> >> -Harsha > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > >> > > > >> >> > >> >> On Tue, Apr 19, 2016, at 09:57 AM, > parth > > > > > > > brahmbhatt > > > > > > > > >> wrote: > > > > > > > > >> > > > >> >> > >> >> > Thanks for review Jitendra. > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > >> > > > >> >> > >> >> > I don't like the idea of infinite > > > lifetime > > > > > > but I > > > > > > > > >> see the > > > > > > > > >> > > > >> Streaming > > > > > > > > >> > > > >> >> > use > > > > > > > > >> > > > >> >> > >> >> > case. Even for Streaming use case I > > was > > > > > hoping > > > > > > > > >> there will > > > > > > > > >> > > > be > > > > > > > > >> > > > >> some > > > > > > > > >> > > > >> >> > notion > > > > > > > > >> > > > >> >> > >> >> > of > > > > > > > > >> > > > >> >> > >> >> > master/driver that can get new > > > delegation > > > > > > tokens > > > > > > > > at > > > > > > > > >> fixed > > > > > > > > >> > > > >> interval > > > > > > > > >> > > > >> >> > and > > > > > > > > >> > > > >> >> > >> >> > distribute to workers. If that is > not > > > the > > > > > case > > > > > > > for > > > > > > > > >> we can > > > > > > > > >> > > > >> discuss > > > > > > > > >> > > > >> >> > >> >> > delegation tokens renewing them self > > and > > > > the > > > > > > > > >> security > > > > > > > > >> > > > >> implications > > > > > > > > >> > > > >> >> > of the > > > > > > > > >> > > > >> >> > >> >> > same. > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > >> > > > >> >> > >> >> > I did not want clients to fetch > tokens > > > > from > > > > > > > > >> zookeeper, > > > > > > > > >> > > > >> overall I > > > > > > > > >> > > > >> >> > think > > > > > > > > >> > > > >> >> > >> >> > its > > > > > > > > >> > > > >> >> > >> >> > better if clients don't rely on our > > > > metadata > > > > > > > store > > > > > > > > >> and I > > > > > > > > >> > > > >> think we > > > > > > > > >> > > > >> >> > are > > > > > > > > >> > > > >> >> > >> >> > moving in that direction with all > the > > > > KIP-4 > > > > > > > > >> improvements. > > > > > > > > >> > > > I > > > > > > > > >> > > > >> chose > > > > > > > > >> > > > >> >> > >> >> > zookeeper as in this case the client > > > will > > > > > > still > > > > > > > > >> just talk > > > > > > > > >> > > > to > > > > > > > > >> > > > >> >> > broker , its > > > > > > > > >> > > > >> >> > >> >> > the brokers that will use zookeeper > > > which > > > > we > > > > > > > > >> already do > > > > > > > > >> > > > for a > > > > > > > > >> > > > >> lot > > > > > > > > >> > > > >> >> > of > > > > > > > > >> > > > >> >> > >> >> > other > > > > > > > > >> > > > >> >> > >> >> > usecases + ease of development + and > > the > > > > > > ability > > > > > > > > so > > > > > > > > >> > > tokens > > > > > > > > >> > > > >> will > > > > > > > > >> > > > >> >> > survive > > > > > > > > >> > > > >> >> > >> >> > even a rolling restart/cluster > > failure. > > > > if a > > > > > > > > >> majority > > > > > > > > >> > > > agrees > > > > > > > > >> > > > >> the > > > > > > > > >> > > > >> >> > added > > > > > > > > >> > > > >> >> > >> >> > complexity to have controller > > forwarding > > > > > keys > > > > > > to > > > > > > > > all > > > > > > > > >> > > > broker is > > > > > > > > >> > > > >> >> > justified > > > > > > > > >> > > > >> >> > >> >> > as > > > > > > > > >> > > > >> >> > >> >> > it provides tighter security , I am > > fine > > > > > with > > > > > > > that > > > > > > > > >> option > > > > > > > > >> > > > too. > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > >> > > > >> >> > >> >> > Given zookeeper does not support SSL > > we > > > > can > > > > > > not > > > > > > > > >> store > > > > > > > > >> > > > master > > > > > > > > >> > > > >> keys > > > > > > > > >> > > > >> >> > in > > > > > > > > >> > > > >> >> > >> >> > zookeeper as master keys will be > > exposed > > > > on > > > > > > > wire. > > > > > > > > To > > > > > > > > >> > > > support > > > > > > > > >> > > > >> >> > rotation > > > > > > > > >> > > > >> >> > >> >> > without affecting current clients is > > > > > > something I > > > > > > > > >> need to > > > > > > > > >> > > > put > > > > > > > > >> > > > >> more > > > > > > > > >> > > > >> >> > thought > > > > > > > > >> > > > >> >> > >> >> > in. My current proposal assumes the > > > > rotation > > > > > > > will > > > > > > > > >> > > > invalidate > > > > > > > > >> > > > >> all > > > > > > > > >> > > > >> >> > current > > > > > > > > >> > > > >> >> > >> >> > tokens. > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > >> > > > >> >> > >> >> > I request committers to also review > > and > > > > post > > > > > > > their > > > > > > > > >> > > comments > > > > > > > > >> > > > >> so we > > > > > > > > >> > > > >> >> > can > > > > > > > > >> > > > >> >> > >> >> > make > > > > > > > > >> > > > >> >> > >> >> > progress on this KIP. > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > >> > > > >> >> > >> >> > Thanks > > > > > > > > >> > > > >> >> > >> >> > Parth > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > >> > > > >> >> > >> >> > On Tue, Apr 19, 2016 at 8:39 AM, > > Ashish > > > > > Singh > > > > > > < > > > > > > > > >> > > > >> asi...@cloudera.com > > > > > > > > >> > > > >> >> > > > > > > > > > > >> > > > >> >> > >> >> > wrote: > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > >> > > > >> >> > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM, > > > > Harsha < > > > > > > > > >> > > > ka...@harsha.io> > > > > > > > > >> > > > >> >> > wrote: > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > >> > > > >> >> > >> >> > > > Unifying the two discussion > > threads > > > on > > > > > > this > > > > > > > > KIP. > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > > >> > > > >> >> > >> >> > > > Here is the response from > Jitendra > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > > >> > > > >> >> > >> >> > > > "The need for a large number of > > > > clients > > > > > > that > > > > > > > > are > > > > > > > > >> > > > running > > > > > > > > >> > > > >> all > > > > > > > > >> > > > >> >> > over the > > > > > > > > >> > > > >> >> > >> >> > > > cluster that authenticate with > > Kafka > > > > > > > brokers, > > > > > > > > >> is very > > > > > > > > >> > > > >> similar > > > > > > > > >> > > > >> >> > to the > > > > > > > > >> > > > >> >> > >> >> > > > Hadoop use case of large number > of > > > > tasks > > > > > > > > running > > > > > > > > >> > > across > > > > > > > > >> > > > >> the > > > > > > > > >> > > > >> >> > cluster > > > > > > > > >> > > > >> >> > >> >> that > > > > > > > > >> > > > >> >> > >> >> > > > need authentication to Hdfs > > > Namenode. > > > > > > > > >> Therefore, the > > > > > > > > >> > > > >> >> > delegation token > > > > > > > > >> > > > >> >> > >> >> > > > approach does seem like a good > fit > > > for > > > > > > this > > > > > > > > use > > > > > > > > >> case > > > > > > > > >> > > > as we > > > > > > > > >> > > > >> >> > have seen > > > > > > > > >> > > > >> >> > >> >> it > > > > > > > > >> > > > >> >> > >> >> > > > working at large scale in HDFS > and > > > > YARN. > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > > >> > > > >> >> > >> >> > > > The proposed design is very > much > > > > > inline > > > > > > > with > > > > > > > > >> Hadoop > > > > > > > > >> > > > >> >> > approach. A few > > > > > > > > >> > > > >> >> > >> >> > > > comments: > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > > >> > > > >> >> > >> >> > > > 1) Why do you guys want to allow > > > > > infinite > > > > > > > > >> renewable > > > > > > > > >> > > > >> lifetime > > > > > > > > >> > > > >> >> > for a > > > > > > > > >> > > > >> >> > >> >> > > > token? HDFS restricts a token > to a > > > max > > > > > > life > > > > > > > > time > > > > > > > > >> > > > (default > > > > > > > > >> > > > >> 7 > > > > > > > > >> > > > >> >> > days). A > > > > > > > > >> > > > >> >> > >> >> > > > token's vulnerability is > believed > > to > > > > > > > increase > > > > > > > > >> with > > > > > > > > >> > > > time. > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > > >> > > > >> >> > >> >> > > I agree that having infinite > > lifetime > > > > > might > > > > > > > not > > > > > > > > >> be the > > > > > > > > >> > > > best > > > > > > > > >> > > > >> idea. > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > > >> > > > >> >> > >> >> > > > 2) As I understand the tokens > are > > > > stored > > > > > > in > > > > > > > > >> zookeeper > > > > > > > > >> > > > as > > > > > > > > >> > > > >> well, > > > > > > > > >> > > > >> >> > and > > > > > > > > >> > > > >> >> > >> >> can > > > > > > > > >> > > > >> >> > >> >> > > > be updated there. This is clever > > as > > > it > > > > > can > > > > > > > > allow > > > > > > > > >> > > > >> replacing the > > > > > > > > >> > > > >> >> > tokens > > > > > > > > >> > > > >> >> > >> >> > > > once they run out of max life > > time, > > > > and > > > > > > > > clients > > > > > > > > >> can > > > > > > > > >> > > > >> download > > > > > > > > >> > > > >> >> > new > > > > > > > > >> > > > >> >> > >> >> tokens > > > > > > > > >> > > > >> >> > >> >> > > > from zookeeper. It shouldn't be > a > > > big > > > > > load > > > > > > > on > > > > > > > > >> > > zookeeper > > > > > > > > >> > > > >> as a > > > > > > > > >> > > > >> >> > client > > > > > > > > >> > > > >> >> > >> >> will > > > > > > > > >> > > > >> >> > >> >> > > > need to get a new token once in > > > > several > > > > > > > days. > > > > > > > > >> In this > > > > > > > > >> > > > >> approach > > > > > > > > >> > > > >> >> > you > > > > > > > > >> > > > >> >> > >> >> don't > > > > > > > > >> > > > >> >> > >> >> > > > need infinite lifetime on the > > token > > > > even > > > > > > for > > > > > > > > >> long > > > > > > > > >> > > > running > > > > > > > > >> > > > >> >> > clients. > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > > >> > > > >> >> > >> >> > > > 3) The token password are > > generated > > > > > using > > > > > > a > > > > > > > > >> master > > > > > > > > >> > > key. > > > > > > > > >> > > > >> The > > > > > > > > >> > > > >> >> > master > > > > > > > > >> > > > >> >> > >> >> key > > > > > > > > >> > > > >> >> > >> >> > > > should also be periodically > > changed. > > > > In > > > > > > > > Hadoop, > > > > > > > > >> the > > > > > > > > >> > > > >> default > > > > > > > > >> > > > >> >> > renewal > > > > > > > > >> > > > >> >> > >> >> > > > period is 1 day.? > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > > >> > > > >> >> > >> >> > > IIUC, this will require brokers > > > > > maintaining > > > > > > a > > > > > > > > >> list of X > > > > > > > > >> > > > most > > > > > > > > >> > > > >> >> > recent > > > > > > > > >> > > > >> >> > >> >> master > > > > > > > > >> > > > >> >> > >> >> > > keys. This list will have to be > > > > persisted > > > > > > > > >> somewhere, as > > > > > > > > >> > > > if a > > > > > > > > >> > > > >> >> > broker > > > > > > > > >> > > > >> >> > >> >> goes > > > > > > > > >> > > > >> >> > >> >> > > down it will have to get that list > > > again > > > > > and > > > > > > > > >> storing > > > > > > > > >> > > > master > > > > > > > > >> > > > >> keys > > > > > > > > >> > > > >> >> > on ZK > > > > > > > > >> > > > >> >> > >> >> is > > > > > > > > >> > > > >> >> > >> >> > > not the best idea. However, if a > > > broker > > > > > goes > > > > > > > > down > > > > > > > > >> then > > > > > > > > >> > > we > > > > > > > > >> > > > >> have > > > > > > > > >> > > > >> >> > much > > > > > > > > >> > > > >> >> > >> >> bigger > > > > > > > > >> > > > >> >> > >> >> > > issue to deal with and client can > > > always > > > > > > > > >> > > re-authenticate > > > > > > > > >> > > > is > > > > > > > > >> > > > >> such > > > > > > > > >> > > > >> >> > >> >> events. > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > >> > > > >> >> > >> >> > > Did you happen to take a look at > > other > > > > > > > > >> alternatives > > > > > > > > >> > > this > > > > > > > > >> > > > >> list has > > > > > > > > >> > > > >> >> > >> >> > > suggested? > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > > >> > > > >> >> > >> >> > > > Thanks for a thorough proposal, > > > great > > > > > > work!" > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > > >> > > > >> >> > >> >> > > > On Mon, Mar 7, 2016, at 10:28 > PM, > > > Gwen > > > > > > > Shapira > > > > > > > > >> wrote: > > > > > > > > >> > > > >> >> > >> >> > > > > Makes sense to me. Thanks! > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > > > >> > > > >> >> > >> >> > > > > On Mon, Mar 7, 2016 at 9:25 > PM, > > > > > Harsha < > > > > > > > > >> > > > ka...@harsha.io > > > > > > > > >> > > > >> > > > > > > > > > >> > > > >> >> > wrote: > > > > > > > > >> > > > >> >> > >> >> > > > > > It doesn't need any release > > > > vehicle > > > > > > but > > > > > > > > >> still the > > > > > > > > >> > > > >> work can > > > > > > > > >> > > > >> >> > move > > > > > > > > >> > > > >> >> > >> >> > > > forward. > > > > > > > > >> > > > >> >> > >> >> > > > > > If anyone is interested in > the > > > KIP > > > > > > > please > > > > > > > > >> do the > > > > > > > > >> > > > >> review and > > > > > > > > >> > > > >> >> > >> >> provide > > > > > > > > >> > > > >> >> > >> >> > > the > > > > > > > > >> > > > >> >> > >> >> > > > > > comments. > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > > > > >> > > > >> >> > >> >> > > > > > -Harsha > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > > > > >> > > > >> >> > >> >> > > > > > On Mon, Mar 7, 2016, at > 04:59 > > > PM, > > > > > > Ismael > > > > > > > > >> Juma > > > > > > > > >> > > > wrote: > > > > > > > > >> > > > >> >> > >> >> > > > > >> I agree that it would be > good > > > to > > > > > have > > > > > > > > more > > > > > > > > >> time > > > > > > > > >> > > to > > > > > > > > >> > > > >> review > > > > > > > > >> > > > >> >> > and > > > > > > > > >> > > > >> >> > >> >> > > discuss > > > > > > > > >> > > > >> >> > >> >> > > > > >> KIP-48. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> Ismael > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> On Tue, Mar 8, 2016 at > 12:55 > > > AM, > > > > > Gwen > > > > > > > > >> Shapira < > > > > > > > > >> > > > >> >> > >> >> g...@confluent.io> > > > > > > > > >> > > > >> >> > >> >> > > > wrote: > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Hi Team, > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Since KIP-48 depends on > > > KIP-43, > > > > > > which > > > > > > > > is > > > > > > > > >> > > > already a > > > > > > > > >> > > > >> bit > > > > > > > > >> > > > >> >> > of a > > > > > > > > >> > > > >> >> > >> >> risk > > > > > > > > >> > > > >> >> > >> >> > > for > > > > > > > > >> > > > >> >> > >> >> > > > > >> > the next release - any > > chance > > > > we > > > > > > can > > > > > > > > >> delay > > > > > > > > >> > > > >> delegation > > > > > > > > >> > > > >> >> > tokens > > > > > > > > >> > > > >> >> > >> >> to > > > > > > > > >> > > > >> >> > >> >> > > > Kafka > > > > > > > > >> > > > >> >> > >> >> > > > > >> > 0.10.1? > > > > > > > > >> > > > >> >> > >> >> > > > > >> > With the community > working > > > on a > > > > > > > release > > > > > > > > >> every > > > > > > > > >> > > 3 > > > > > > > > >> > > > >> month, > > > > > > > > >> > > > >> >> > this > > > > > > > > >> > > > >> >> > >> >> is not > > > > > > > > >> > > > >> >> > >> >> > > > a huge > > > > > > > > >> > > > >> >> > >> >> > > > > >> > delay. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > >> > > > >> >> > >> >> > > > > >> > Gwen > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > >> > > > >> >> > >> >> > > > > >> > On Fri, Feb 26, 2016 at > > 5:11 > > > > PM, > > > > > > > Ashish > > > > > > > > >> Singh > > > > > > > > >> > > < > > > > > > > > >> > > > >> >> > >> >> > > asi...@cloudera.com> > > > > > > > > >> > > > >> >> > >> >> > > > wrote: > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Parth, > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Thanks again for the > > > awesome > > > > > > write > > > > > > > > up. > > > > > > > > >> > > > Following > > > > > > > > >> > > > >> our > > > > > > > > >> > > > >> >> > >> >> discussion > > > > > > > > >> > > > >> >> > >> >> > > > from the > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > JIRA, I think it will > be > > > > easier > > > > > > to > > > > > > > > >> compare > > > > > > > > >> > > > >> various > > > > > > > > >> > > > >> >> > >> >> alternatives > > > > > > > > >> > > > >> >> > >> >> > > > if they > > > > > > > > >> > > > >> >> > >> >> > > > > >> > are > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > listed together. I am > > > stating > > > > > > > below a > > > > > > > > >> few > > > > > > > > >> > > > >> >> > alternatives along > > > > > > > > >> > > > >> >> > >> >> > > with > > > > > > > > >> > > > >> >> > >> >> > > > a the > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > current proposal. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Current proposal) > Store > > > > > > Delegation > > > > > > > > >> Token, > > > > > > > > >> > > DT, > > > > > > > > >> > > > >> on ZK. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 1. Client > > authenticates > > > > > with a > > > > > > > > >> broker. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 2. Once a client is > > > > > > > authenticated, > > > > > > > > >> it > > > > > > > > >> > > will > > > > > > > > >> > > > >> make a > > > > > > > > >> > > > >> >> > broker > > > > > > > > >> > > > >> >> > >> >> side > > > > > > > > >> > > > >> >> > >> >> > > > call to > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > issue a delegation > > > token. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 3. The broker > > generates > > > a > > > > > > shared > > > > > > > > >> secret > > > > > > > > >> > > > based > > > > > > > > >> > > > >> on > > > > > > > > >> > > > >> >> > >> >> > > HMAC-SHA256(a > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Password/Secret > shared > > > > > between > > > > > > > all > > > > > > > > >> > > brokers, > > > > > > > > >> > > > >> >> > randomly > > > > > > > > >> > > > >> >> > >> >> > > generated > > > > > > > > >> > > > >> >> > >> >> > > > > >> > tokenId). > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 4. Broker stores > this > > > > token > > > > > in > > > > > > > its > > > > > > > > >> in > > > > > > > > >> > > > memory > > > > > > > > >> > > > >> cache. > > > > > > > > >> > > > >> >> > >> >> Broker > > > > > > > > >> > > > >> >> > >> >> > > > also stores > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > the DelegationToken > > > > without > > > > > > the > > > > > > > > >> hmac in > > > > > > > > >> > > the > > > > > > > > >> > > > >> >> > zookeeper. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 5. All brokers will > > > have a > > > > > > cache > > > > > > > > >> backed > > > > > > > > >> > > by > > > > > > > > >> > > > >> >> > zookeeper so > > > > > > > > >> > > > >> >> > >> >> they > > > > > > > > >> > > > >> >> > >> >> > > > will all > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > get notified > whenever > > a > > > > new > > > > > > > token > > > > > > > > is > > > > > > > > >> > > > >> generated and > > > > > > > > >> > > > >> >> > they > > > > > > > > >> > > > >> >> > >> >> will > > > > > > > > >> > > > >> >> > >> >> > > > update > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > local cache whenever > > > token > > > > > > state > > > > > > > > >> changes. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 6. Broker returns > the > > > > token > > > > > to > > > > > > > > >> Client. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and > fixes > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 1. Probable race > > > > condition, > > > > > > > client > > > > > > > > >> tries > > > > > > > > >> > > to > > > > > > > > >> > > > >> >> > authenticate > > > > > > > > >> > > > >> >> > >> >> with > > > > > > > > >> > > > >> >> > >> >> > > > a broker > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > that is yet to be > > > updated > > > > > with > > > > > > > the > > > > > > > > >> newly > > > > > > > > >> > > > >> generated > > > > > > > > >> > > > >> >> > DT. > > > > > > > > >> > > > >> >> > >> >> This > > > > > > > > >> > > > >> >> > >> >> > > can > > > > > > > > >> > > > >> >> > >> >> > > > > >> > probably be > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > dealt with making > > > > dtRequest > > > > > > > block > > > > > > > > >> until > > > > > > > > >> > > all > > > > > > > > >> > > > >> >> > brokers have > > > > > > > > >> > > > >> >> > >> >> > > > updated > > > > > > > > >> > > > >> >> > >> >> > > > > >> > their DT > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > cache. Zk barrier or > > > > similar > > > > > > > > >> mechanism > > > > > > > > >> > > can > > > > > > > > >> > > > be > > > > > > > > >> > > > >> used. > > > > > > > > >> > > > >> >> > >> >> However, > > > > > > > > >> > > > >> >> > >> >> > > > all such > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > mechanisms will > > increase > > > > > > > > complexity. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 2. Using a static > > secret > > > > key > > > > > > > from > > > > > > > > >> config > > > > > > > > >> > > > >> file. Will > > > > > > > > >> > > > >> >> > >> >> require > > > > > > > > >> > > > >> >> > >> >> > > yet > > > > > > > > >> > > > >> >> > >> >> > > > > >> > another > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > config and uses a > > static > > > > > > secret > > > > > > > > >> key. It > > > > > > > > >> > > is > > > > > > > > >> > > > >> advised > > > > > > > > >> > > > >> >> > to > > > > > > > > >> > > > >> >> > >> >> rotate > > > > > > > > >> > > > >> >> > >> >> > > > secret > > > > > > > > >> > > > >> >> > >> >> > > > > >> > keys > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > periodically. This > can > > > be > > > > > > > avoided > > > > > > > > >> with > > > > > > > > >> > > > >> controller > > > > > > > > >> > > > >> >> > >> >> generating > > > > > > > > >> > > > >> >> > >> >> > > > > >> > secretKey and > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > passing to brokers > > > > > > periodically. > > > > > > > > >> However, > > > > > > > > >> > > > >> this will > > > > > > > > >> > > > >> >> > >> >> require > > > > > > > > >> > > > >> >> > >> >> > > > brokers to > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > maintain certain > > counts > > > of > > > > > > > > >> secretKeys. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 1) Have > > > > controller > > > > > > > > >> generate > > > > > > > > >> > > > >> delegation > > > > > > > > >> > > > >> >> > token. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 1. Client > > authenticates > > > > > with a > > > > > > > > >> broker. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 2. Once a client is > > > > > > > authenticated, > > > > > > > > >> it > > > > > > > > >> > > will > > > > > > > > >> > > > >> make a > > > > > > > > >> > > > >> >> > broker > > > > > > > > >> > > > >> >> > >> >> side > > > > > > > > >> > > > >> >> > >> >> > > > call to > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > issue a delegation > > > token. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 3. Broker forwards > the > > > > > request > > > > > > > to > > > > > > > > >> > > > controller. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 4. Controller > > generates > > > a > > > > DT > > > > > > and > > > > > > > > >> > > broadcasts > > > > > > > > >> > > > >> to all > > > > > > > > >> > > > >> >> > >> >> brokers. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 5. Broker stores > this > > > > token > > > > > in > > > > > > > its > > > > > > > > >> memory > > > > > > > > >> > > > >> cache. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 6. Controller > responds > > > to > > > > > > > broker’s > > > > > > > > >> DT > > > > > > > > >> > > req. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 7. Broker returns > the > > > > token > > > > > to > > > > > > > > >> Client. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and > fixes > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 1. We will have to > add > > > new > > > > > > APIs > > > > > > > to > > > > > > > > >> > > support > > > > > > > > >> > > > >> >> > controller > > > > > > > > >> > > > >> >> > >> >> pushing > > > > > > > > >> > > > >> >> > >> >> > > > tokens > > > > > > > > >> > > > >> >> > >> >> > > > > >> > to > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > brokers on top of > the > > > > > minimal > > > > > > > APIs > > > > > > > > >> that > > > > > > > > >> > > are > > > > > > > > >> > > > >> >> > currently > > > > > > > > >> > > > >> >> > >> >> > > proposed. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 2. We will also have > > to > > > > add > > > > > > APIs > > > > > > > > to > > > > > > > > >> > > support > > > > > > > > >> > > > >> the > > > > > > > > >> > > > >> >> > >> >> bootstrapping > > > > > > > > >> > > > >> >> > >> >> > > > case, > > > > > > > > >> > > > >> >> > >> >> > > > > >> > i.e, > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > when a new broker > > comes > > > up > > > > > it > > > > > > > will > > > > > > > > >> have > > > > > > > > >> > > to > > > > > > > > >> > > > >> get all > > > > > > > > >> > > > >> >> > >> >> delegation > > > > > > > > >> > > > >> >> > >> >> > > > tokens > > > > > > > > >> > > > >> >> > >> >> > > > > >> > from > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > the controller. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 3. In catastrophic > > > > failures > > > > > > > where > > > > > > > > >> all > > > > > > > > >> > > > brokers > > > > > > > > >> > > > >> go > > > > > > > > >> > > > >> >> > down, > > > > > > > > >> > > > >> >> > >> >> the > > > > > > > > >> > > > >> >> > >> >> > > > tokens will > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > be lost even if > > servers > > > > are > > > > > > > > >> restarted as > > > > > > > > >> > > > >> tokens > > > > > > > > >> > > > >> >> > are not > > > > > > > > >> > > > >> >> > >> >> > > > persisted > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > If this happens, > then > > > > there > > > > > > are > > > > > > > > more > > > > > > > > >> > > > important > > > > > > > > >> > > > >> >> > things to > > > > > > > > >> > > > >> >> > >> >> > > worry > > > > > > > > >> > > > >> >> > >> >> > > > about > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > maybe it is better > to > > > > > > > > >> re-authenticate. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > (Alternative 2) Do not > > > > > distribute > > > > > > > DT > > > > > > > > to > > > > > > > > >> > > other > > > > > > > > >> > > > >> brokers > > > > > > > > >> > > > >> >> > at > > > > > > > > >> > > > >> >> > >> >> all. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 1. Client > > authenticates > > > > > with a > > > > > > > > >> broker. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 2. Once a client is > > > > > > > authenticated, > > > > > > > > >> it > > > > > > > > >> > > will > > > > > > > > >> > > > >> make a > > > > > > > > >> > > > >> >> > broker > > > > > > > > >> > > > >> >> > >> >> side > > > > > > > > >> > > > >> >> > >> >> > > > call to > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > issue a delegation > > > token. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 3. The broker > > generates > > > DT > > > > > of > > > > > > > > form, > > > > > > > > >> > > [hmac + > > > > > > > > >> > > > >> (owner, > > > > > > > > >> > > > >> >> > >> >> renewer, > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > maxLifeTime, id, > hmac, > > > > > > > > >> expirationTime)] > > > > > > > > >> > > and > > > > > > > > >> > > > >> passes > > > > > > > > >> > > > >> >> > back > > > > > > > > >> > > > >> >> > >> >> this > > > > > > > > >> > > > >> >> > >> >> > > > DT to > > > > > > > > >> > > > >> >> > >> >> > > > > >> > client. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > hmac is generated > via > > > > > > > > >> {HMAC-SHA256(owner, > > > > > > > > >> > > > >> renewer, > > > > > > > > >> > > > >> >> > >> >> > > > maxLifeTime, id, > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac, > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > expirationTime) > using > > > > > > > SecretKey}. > > > > > > > > >> Note > > > > > > > > >> > > that > > > > > > > > >> > > > >> all > > > > > > > > >> > > > >> >> > brokers > > > > > > > > >> > > > >> >> > >> >> have > > > > > > > > >> > > > >> >> > >> >> > > > this > > > > > > > > >> > > > >> >> > >> >> > > > > >> > SecretKey. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 4. Client then goes > to > > > any > > > > > > > broker > > > > > > > > >> and to > > > > > > > > >> > > > >> >> > authenticate > > > > > > > > >> > > > >> >> > >> >> sends > > > > > > > > >> > > > >> >> > >> >> > > > the DT. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Broker recalculates > > hmac > > > > > using > > > > > > > > >> (owner, > > > > > > > > >> > > > >> renewer, > > > > > > > > >> > > > >> >> > >> >> maxLifeTime, > > > > > > > > >> > > > >> >> > >> >> > > > id, hmac, > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > expirationTime) info > > > from > > > > DT > > > > > > and > > > > > > > > its > > > > > > > > >> > > > >> SecretKey. If > > > > > > > > >> > > > >> >> > it > > > > > > > > >> > > > >> >> > >> >> matches > > > > > > > > >> > > > >> >> > >> >> > > > with > > > > > > > > >> > > > >> >> > >> >> > > > > >> > hmac of > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > DT, client is > > > > authenticated. > > > > > > > Yes, > > > > > > > > >> it will > > > > > > > > >> > > > do > > > > > > > > >> > > > >> other > > > > > > > > >> > > > >> >> > >> >> obvious > > > > > > > > >> > > > >> >> > >> >> > > > checks of > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > timestamp expiry and > > > such. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Note that secret key > will > > > be > > > > > > > > generated > > > > > > > > >> by > > > > > > > > >> > > > >> controller > > > > > > > > >> > > > >> >> > and > > > > > > > > >> > > > >> >> > >> >> passed > > > > > > > > >> > > > >> >> > >> >> > > to > > > > > > > > >> > > > >> >> > >> >> > > > > >> > brokers > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > periodically. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Probable issues and > fixes > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 1. How to delete a > DT? > > > > Yes, > > > > > > that > > > > > > > > is > > > > > > > > >> a > > > > > > > > >> > > > downside > > > > > > > > >> > > > >> >> > here. > > > > > > > > >> > > > >> >> > >> >> However, > > > > > > > > >> > > > >> >> > >> >> > > > this can > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > be handled with > > brokers > > > > > > > > maintaining > > > > > > > > >> a > > > > > > > > >> > > > >> blacklist of > > > > > > > > >> > > > >> >> > DTs, > > > > > > > > >> > > > >> >> > >> >> DTs > > > > > > > > >> > > > >> >> > >> >> > > > from this > > > > > > > > >> > > > >> >> > >> >> > > > > >> > list > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > can be removed after > > > > expiry. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > 2. In catastrophic > > > > failures > > > > > > > where > > > > > > > > >> all > > > > > > > > >> > > > brokers > > > > > > > > >> > > > >> go > > > > > > > > >> > > > >> >> > down, > > > > > > > > >> > > > >> >> > >> >> the > > > > > > > > >> > > > >> >> > >> >> > > > tokens will > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > be lost even if > > servers > > > > are > > > > > > > > >> restarted as > > > > > > > > >> > > > >> tokens > > > > > > > > >> > > > >> >> > are not > > > > > > > > >> > > > >> >> > >> >> > > > persisted > > > > > > > > >> > > > >> >> > >> >> > > > > >> > anywhere. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > If this happens, > then > > > > there > > > > > > are > > > > > > > > more > > > > > > > > >> > > > important > > > > > > > > >> > > > >> >> > things to > > > > > > > > >> > > > >> >> > >> >> > > worry > > > > > > > > >> > > > >> >> > >> >> > > > about > > > > > > > > >> > > > >> >> > >> >> > > > > >> > and > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > maybe it is better > to > > > > > > > > >> re-authenticate. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > On Fri, Feb 26, 2016 at > > > 1:58 > > > > > PM, > > > > > > > > Parth > > > > > > > > >> > > > >> Brahmbhatt < > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > pbrahmbh...@hortonworks.com> > > > > > > > wrote: > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Hi, > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> I have filed KIP-48 so > > we > > > > can > > > > > > > offer > > > > > > > > >> hadoop > > > > > > > > >> > > > like > > > > > > > > >> > > > >> >> > delegation > > > > > > > > >> > > > >> >> > >> >> > > > tokens in > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> kafka. You can review > > the > > > > > design > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > >> > > > >> >> > > > > > > > > > >> > > > >> > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka > > > > > > > > >> > > > >> >> > >> >> > > > > >> > . > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> This KIP depends on > > KIP-43 > > > > and > > > > > > we > > > > > > > > >> have also > > > > > > > > >> > > > >> >> > discussed an > > > > > > > > >> > > > >> >> > >> >> > > > alternative to > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> proposed design here< > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > >> > > > >> >> > > > > > > > > > >> > > > >> > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800 > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> >. > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Thanks > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> Parth > > > > > > > > >> > > > >> >> > >> >> > > > > >> > >> > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > -- > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Regards, > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > Ashish > > > > > > > > >> > > > >> >> > >> >> > > > > >> > > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > >> > > > >> >> > >> >> > > -- > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > >> > > > >> >> > >> >> > > Regards, > > > > > > > > >> > > > >> >> > >> >> > > Ashish > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > > > >> > > > >> >> > >> >> > > > > > > > > >> > > > >> >> > > > > > > > > > >> > > > >> > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Regards, > > > > > > > > > > Rajini > > > > > > > > > > > > > > > > > > > > > -- > > > Regards, > > > > > > Rajini > > > > > > -- Liquan Pei Software Engineer, Confluent Inc
