Hi All,
Can we have a KIP meeting around this. The KIP is up for
sometime and if there are any questions lets quickly hash out
details.
Thanks,
Harsha
On Thu, May 19, 2016, at 08:40 AM, parth brahmbhatt wrote:
> That is what the hadoop echo system uses so no good reason really. We
> could
> change it to whatever is the newest recommended standard is.
>
> Thanks
> Parth
>
> On Thu, May 19, 2016 at 3:33 AM, Ismael Juma <ism...@juma.me.uk> wrote:
>
> > Hi Parth,
> >
> > Thanks for the KIP. I only started reviewing this and may have additional
> > questions later. The immediate question that came to mind is our choice of
> > "DIGEST-MD5" even though it's marked as OBSOLETE in the IANA Registry of
> > SASL mechanisms and the original RFC (2831) has been moved to Historic
> > status:
> >
> > https://tools.ietf.org/html/rfc6331
> > http://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml
> >
> > What is the reasoning behind that choice?
> >
> > Thanks,
> > Ismael
> >
> > On Fri, May 13, 2016 at 11:29 PM, Gwen Shapira <g...@confluent.io> wrote:
> >
> > > Also comments inline :)
> > >
> > > > * I want to emphasize that even though delegation tokens are a Hadoop
> > > > innovation, I feel very strongly about not adding dependency on Hadoop
> > > > when implementing delegation tokens for Kafka. The KIP doesn't imply
> > > > such dependency, but if you can clarify...
> > > >
> > > >
> > > > *No hadoop dependency.*
> > >
> > > Yay! Just add this to the KIP so no one will read the KIP and panic
> > > three weeks before the next release...
> > >
> > > > * Can we get delegation token at any time after authenticating? only
> > > > immediately after?
> > > >
> > > >
> > > > *As long as you are authenticated you can get delegation tokens. We
> > need
> > > to
> > > > discuss if a client authenticated using delegation token, can also
> > > acquire
> > > > delegation token again or not. Also there is the question of do we
> > allow
> > > > anyone to acquire delegation token or we want specific ACLs (I think
> > its
> > > an
> > > > overkill.)*
> > >
> > > I agree that ACLs is an overkill.
> > >
> > > I think we are debating two options: Either require Kerberos auth for
> > > renewal or require non-owners to renew.
> > > I *think* the latter is simpler (it basically require a "job master"
> > > to take responsibility for the renewal, it will have its own identity
> > > anyway and I think this is the correct design pattern anyway. For
> > > storm, I'd expect Nimbus to coordinate renewals?), but it is hard to
> > > debate simplicity without looking at the code changes required. If you
> > > have a draft of how the "require Kerberos" will look in Kafka code,
> > > I'll be happy to take a look.
> > >
> > > > * My understanding is that tokens will propagate via ZK but without
> > > > additional changes to UpdateMetadata protocol, correct? Clients
> > > > currently don't retry on SASL auth failure (IIRC), but since the
> > > > tokens propagate between brokers asynch, we will need to retry a bit
> > > > to avoid clients failing auth due to timing issues.
> > > >
> > > > *I am considering 2 alternatives right now. The current documented
> > > approach
> > > > is zookeeper based and it does not require any changes to
> > UpdateMetadata
> > > > protocol. An alternative approach can remove zookeeper dependency as
> > well
> > > > but we can discuss that in KIP discussion call.*
> > >
> > > Oooh! Sounds interesting. Do you want to ping Jun to arrange a call?
> > >
> > > > * I liked Ashish's suggestion of having just the controller issue the
> > > > delegation tokens, to avoid syncing a shared secret. Not sure if we
> > > > want to continue the discussion here or on the wiki. I think that we
> > > > can decouple the problem of "token distribution" from "shared secret
> > > > distribution" and use the controller as the only token generator to
> > > > solve the second issue, while still using ZK async to distribute
> > > > tokens.
> > > >
> > > >
> > > > *As mentioned in the previous Email I am fine with that approach as
> > long
> > > as
> > > > we agree that the extra complexity of adding/updating APIS adds enough
> > > > value. The advantage with the controller approach is secret rotation
> > can
> > > be
> > > > automated,frequent and would not require deployment. *
> > >
> > > Can you detail the extra complexity (or point me to the email I
> > > missed?) - which Apis are required?
> > > As far as I can tell, clients can already find the controller from
> > > metadata. I'm a bit more concerned about controller load.
> > >
> > > >
> > > > * While I like the idea of forcing kerberos auth for renwal, I think
> > > > it mixes the transport layer the the request content in a pretty ugly
> > > > way. Perhaps limiting renewer to non-owner is better.
> > > >
> > > > *I feel this is a necessary evil. While this will make the kafka code
> > > > pretty straight forward , forcing renewer to non-owner pushes the code
> > > > ugliness to client and makes it even harder to integrate. *
> > >
> > > As mentioned before, I don't think the "renewal by other" approach is
> > > that ugly for the clients we expect to use delegation tokens since
> > > they will have an app-master of some sort who requested the token to
> > > begin with.
> > >
> > > >
> > > > The response for my question on how multiple identities will be
> > > > handled wasn't super clear to me - AFAIK, we don't authenticate each
> > > > request, we authenticate connections.
> > > >
> > > > *We authenticate connections, and only when they are being established.
> > > Let
> > > > me try to phrase this as a question, in absence of delegation tokens if
> > > we
> > > > had to support the use case using user TGT's how would we do it? My
> > point
> > > > was it would be no different with delegation tokens. The use case you
> > are
> > > > describing seems more like impersonation.*
> > >
> > > Yeah, I thought that one of the things that delegation tokens handled.
> > > Maybe I got it wrong :)
> > >
> > > Thanks for the detailed answers.
> > >
> > > Gwen
> > >
> > >
> > > > Thanks
> > > > Parth
> > > >
> > > > On Fri, May 13, 2016 at 12:19 AM, Gwen Shapira <g...@confluent.io>
> > > wrote:
> > > >
> > > >> Hi Parth and Harsha,
> > > >>
> > > >> Few more comments:
> > > >>
> > > >> * The API / RequestResponse section doesn't seem to have good
> > > >> description of the changes to the Kafka Protocol. Sounds like you are
> > > >> proposing new DelegationTokenRequest and RenewTokenRequest (and
> > > >> matching responses), without detailing the contents of the requests
> > > >> and responses? Or rather, you show the class interface, but not the
> > > >> underlying protocol. This could be seen as an implementation detail,
> > > >> but since the binary protocol is what we provide to non-Java clients,
> > > >> we need to show the changes there.
> > > >>
> > > >> * getDelegationToken sounds like delegationTokenRequestHandler? Is it
> > > >> planned to be part of KafkaApi? or Client? Its unclear...
> > > >>
> > > >> * I want to emphasize that even though delegation tokens are a Hadoop
> > > >> innovation, I feel very strongly about not adding dependency on Hadoop
> > > >> when implementing delegation tokens for Kafka. The KIP doesn't imply
> > > >> such dependency, but if you can clarify...
> > > >>
> > > >> * Can we get delegation token at any time after authenticating? only
> > > >> immediately after?
> > > >>
> > > >> * My understanding is that tokens will propagate via ZK but without
> > > >> additional changes to UpdateMetadata protocol, correct? Clients
> > > >> currently don't retry on SASL auth failure (IIRC), but since the
> > > >> tokens propagate between brokers asynch, we will need to retry a bit
> > > >> to avoid clients failing auth due to timing issues.
> > > >>
> > > >> * Strongly agreeing on clients not touching ZK directly :)
> > > >>
> > > >> * I liked Ashish's suggestion of having just the controller issue the
> > > >> delegation tokens, to avoid syncing a shared secret. Not sure if we
> > > >> want to continue the discussion here or on the wiki. I think that we
> > > >> can decouple the problem of "token distribution" from "shared secret
> > > >> distribution" and use the controller as the only token generator to
> > > >> solve the second issue, while still using ZK async to distribute
> > > >> tokens.
> > > >>
> > > >> * I am also uncomfortable with infinite lifetime of tokens (and hoped
> > > >> to hear from others in the community) - but having the option and
> > > >> documenting it as less secure, allows users to configure their system
> > > >> as the wish.
> > > >>
> > > >> * While I like the idea of forcing kerberos auth for renwal, I think
> > > >> it mixes the transport layer the the request content in a pretty ugly
> > > >> way. Perhaps limiting renewer to non-owner is better.
> > > >>
> > > >> Things I'd still like to see:
> > > >>
> > > >> * More detailed explanation on what we plan to do for the java clients
> > > >> specifically - new configuration? new APIs?
> > > >> The response for my question on how multiple identities will be
> > > >> handled wasn't super clear to me - AFAIK, we don't authenticate each
> > > >> request, we authenticate connections.
> > > >>
> > > >> * Alternatives: Delegation tokens are only used in the Hadoop
> > > >> ecosystem. I'm wondering if there are alternatives in other ecosystems
> > > >> (Mesos? Tachyon? Cassandra?) and whether there are some advantages
> > > >> there.
> > > >>
> > > >> Gwen
> > > >>
> > > >> On Thu, May 12, 2016 at 1:05 PM, Harsha <ka...@harsha.io> wrote:
> > > >> > Hi Gwen,
> > > >> > Can you look at Parth's last reply. Does it answer your
> > > >> > concerns.
> > > >> >
> > > >> > Thanks,
> > > >> > Harsha
> > > >> >
> > > >> > On Wed, May 4, 2016, at 09:25 AM, parth brahmbhatt wrote:
> > > >> >> Thanks for reviewing Gwen. The wiki already has details on token
> > > >> >> expiration
> > > >> >> under token acquisition process
> > > >> >> <
> > > >>
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka#KIP-48DelegationtokensupportforKafka-Tokenacquisition
> > > >> >.
> > > >> >> Current proposal is that tokens will expire based on a server side
> > > >> >> configuration (default 24 hours) unless renewed. Renewal is only
> > > allowed
> > > >> >> until the max life time of token. Alternatively we could also make
> > > that
> > > >> >> an
> > > >> >> optional param and the server side default can serve as the upper
> > > bound.
> > > >> >>
> > > >> >> To your second point it will be done exactly the same way we would
> > > >> >> support
> > > >> >> multiple keytabs. The calling client will have to put the tokens it
> > > >> wants
> > > >> >> to use in the subject instance and call produce/consume inside
> > > >> >> subject.doas. Each caller will have to keep track of its own
> > > subject. I
> > > >> >> will have to look at the code to see if we support this feature
> > right
> > > >> now
> > > >> >> but my understanding is delegation token shouldn't need any special
> > > >> >> treatment as its just another type of Credential in the subject.
> > > >> >>
> > > >> >> I would also like to know what is your opinion about infinite
> > renewal
> > > >> (my
> > > >> >> recommendation is to not support this), tokens renewing them
> > self(my
> > > >> >> recommendation is to not support this) and most importantly your
> > > choice
> > > >> >> between the alternatives listed on this thread
> > > >> >> <
> > > >>
> > >
> > http://apache.markmail.org/message/ca3iakt3m6c4yygp?q=KIP-48+Support+for+delegation+tokens+as+an+authentication+mechanism
> > > >> >
> > > >> >> ( I am leaning towards the alternative-2 minus controller
> > > distributing
> > > >> >> secret). Thanks again for reviewing.
> > > >> >>
> > > >> >> Thanks
> > > >> >> Parth
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> On Wed, May 4, 2016 at 6:17 AM, Gwen Shapira <g...@confluent.io>
> > > wrote:
> > > >> >>
> > > >> >> > Harsha,
> > > >> >> >
> > > >> >> > I was thinking of the Rest Proxy. I didn't see your design yet,
> > > but in
> > > >> >> > our proxy, we have a set of producers, which will serve multiple
> > > users
> > > >> >> > going through the proxy. Since these users will have different
> > > >> >> > privileges, they'll need to authenticate separately, and can't
> > > share a
> > > >> >> > token.
> > > >> >> >
> > > >> >> > Am I missing anything?
> > > >> >> >
> > > >> >> > Gwen
> > > >> >> >
> > > >> >> > On Tue, May 3, 2016 at 2:11 PM, Harsha <ka...@harsha.io> wrote:
> > > >> >> > > Gwen,
> > > >> >> > > On your second point. Can you describe a usecase
> > where
> > > >> >> > > mutliple clients ended up sharing a producer and
> > even
> > > if
> > > >> they
> > > >> >> > > do why can't they not use single token that producer
> > > >> >> > > captures. Why would we need multiple clients with
> > > >> different
> > > >> >> > > tokens sharing a single instance of producer. Also
> > in
> > > >> this
> > > >> >> > > case other clients have access all the tokens no?
> > > >> >> > >
> > > >> >> > > Thanks,
> > > >> >> > > Harsha
> > > >> >> > >
> > > >> >> > >
> > > >> >> > > On Tue, May 3, 2016, at 11:49 AM, Gwen Shapira wrote:
> > > >> >> > >> Sorry for the delay:
> > > >> >> > >>
> > > >> >> > >> Two questions that we didn't see in the wiki:
> > > >> >> > >> 1. Is there an expiration for delegation tokens? Renewal? How
> > > do we
> > > >> >> > >> revoke them?
> > > >> >> > >> 2. If we want to use delegation tokens for "do-as" (say,
> > submit
> > > >> Storm
> > > >> >> > >> job as my user), we will need a producer for every job (we
> > can't
> > > >> share
> > > >> >> > >> them between multiple jobs running on same node), since we
> > only
> > > >> >> > >> authenticate when connecting. Is there a plan to change this
> > for
> > > >> >> > >> delegation tokens, in order to allow multiple users with
> > > different
> > > >> >> > >> tokens to share a client?
> > > >> >> > >>
> > > >> >> > >> Gwen
> > > >> >> > >>
> > > >> >> > >> On Tue, May 3, 2016 at 9:12 AM, parth brahmbhatt
> > > >> >> > >> <brahmbhatt.pa...@gmail.com> wrote:
> > > >> >> > >> > Bumping this up one more time, can other committers review?
> > > >> >> > >> >
> > > >> >> > >> > Thanks
> > > >> >> > >> > Parth
> > > >> >> > >> >
> > > >> >> > >> > On Tue, Apr 26, 2016 at 9:07 AM, Harsha <ka...@harsha.io>
> > > wrote:
> > > >> >> > >> >
> > > >> >> > >> >> Parth,
> > > >> >> > >> >> Overall current design looks good to me. I am +1
> > on
> > > >> the
> > > >> >> > KIP.
> > > >> >> > >> >>
> > > >> >> > >> >> Gwen , Jun can you review this as well.
> > > >> >> > >> >>
> > > >> >> > >> >> -Harsha
> > > >> >> > >> >>
> > > >> >> > >> >> On Tue, Apr 19, 2016, at 09:57 AM, parth brahmbhatt wrote:
> > > >> >> > >> >> > Thanks for review Jitendra.
> > > >> >> > >> >> >
> > > >> >> > >> >> > I don't like the idea of infinite lifetime but I see the
> > > >> Streaming
> > > >> >> > use
> > > >> >> > >> >> > case. Even for Streaming use case I was hoping there will
> > > be
> > > >> some
> > > >> >> > notion
> > > >> >> > >> >> > of
> > > >> >> > >> >> > master/driver that can get new delegation tokens at fixed
> > > >> interval
> > > >> >> > and
> > > >> >> > >> >> > distribute to workers. If that is not the case for we can
> > > >> discuss
> > > >> >> > >> >> > delegation tokens renewing them self and the security
> > > >> implications
> > > >> >> > of the
> > > >> >> > >> >> > same.
> > > >> >> > >> >> >
> > > >> >> > >> >> > I did not want clients to fetch tokens from zookeeper,
> > > >> overall I
> > > >> >> > think
> > > >> >> > >> >> > its
> > > >> >> > >> >> > better if clients don't rely on our metadata store and I
> > > >> think we
> > > >> >> > are
> > > >> >> > >> >> > moving in that direction with all the KIP-4 improvements.
> > > I
> > > >> chose
> > > >> >> > >> >> > zookeeper as in this case the client will still just talk
> > > to
> > > >> >> > broker , its
> > > >> >> > >> >> > the brokers that will use zookeeper which we already do
> > > for a
> > > >> lot
> > > >> >> > of
> > > >> >> > >> >> > other
> > > >> >> > >> >> > usecases + ease of development + and the ability so
> > tokens
> > > >> will
> > > >> >> > survive
> > > >> >> > >> >> > even a rolling restart/cluster failure. if a majority
> > > agrees
> > > >> the
> > > >> >> > added
> > > >> >> > >> >> > complexity to have controller forwarding keys to all
> > > broker is
> > > >> >> > justified
> > > >> >> > >> >> > as
> > > >> >> > >> >> > it provides tighter security , I am fine with that option
> > > too.
> > > >> >> > >> >> >
> > > >> >> > >> >> > Given zookeeper does not support SSL we can not store
> > > master
> > > >> keys
> > > >> >> > in
> > > >> >> > >> >> > zookeeper as master keys will be exposed on wire. To
> > > support
> > > >> >> > rotation
> > > >> >> > >> >> > without affecting current clients is something I need to
> > > put
> > > >> more
> > > >> >> > thought
> > > >> >> > >> >> > in. My current proposal assumes the rotation will
> > > invalidate
> > > >> all
> > > >> >> > current
> > > >> >> > >> >> > tokens.
> > > >> >> > >> >> >
> > > >> >> > >> >> > I request committers to also review and post their
> > comments
> > > >> so we
> > > >> >> > can
> > > >> >> > >> >> > make
> > > >> >> > >> >> > progress on this KIP.
> > > >> >> > >> >> >
> > > >> >> > >> >> > Thanks
> > > >> >> > >> >> > Parth
> > > >> >> > >> >> >
> > > >> >> > >> >> > On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh <
> > > >> asi...@cloudera.com
> > > >> >> > >
> > > >> >> > >> >> > wrote:
> > > >> >> > >> >> >
> > > >> >> > >> >> > > On Mon, Apr 18, 2016 at 11:26 AM, Harsha <
> > > ka...@harsha.io>
> > > >> >> > wrote:
> > > >> >> > >> >> > >
> > > >> >> > >> >> > > > Unifying the two discussion threads on this KIP.
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > > > Here is the response from Jitendra
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > > > "The need for a large number of clients that are
> > > running
> > > >> all
> > > >> >> > over the
> > > >> >> > >> >> > > > cluster that authenticate with Kafka brokers, is very
> > > >> similar
> > > >> >> > to the
> > > >> >> > >> >> > > > Hadoop use case of large number of tasks running
> > across
> > > >> the
> > > >> >> > cluster
> > > >> >> > >> >> that
> > > >> >> > >> >> > > > need authentication to Hdfs Namenode. Therefore, the
> > > >> >> > delegation token
> > > >> >> > >> >> > > > approach does seem like a good fit for this use case
> > > as we
> > > >> >> > have seen
> > > >> >> > >> >> it
> > > >> >> > >> >> > > > working at large scale in HDFS and YARN.
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > > > The proposed design is very much inline with Hadoop
> > > >> >> > approach. A few
> > > >> >> > >> >> > > > comments:
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > > > 1) Why do you guys want to allow infinite renewable
> > > >> lifetime
> > > >> >> > for a
> > > >> >> > >> >> > > > token? HDFS restricts a token to a max life time
> > > (default
> > > >> 7
> > > >> >> > days). A
> > > >> >> > >> >> > > > token's vulnerability is believed to increase with
> > > time.
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > > I agree that having infinite lifetime might not be the
> > > best
> > > >> idea.
> > > >> >> > >> >> > >
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > > > 2) As I understand the tokens are stored in zookeeper
> > > as
> > > >> well,
> > > >> >> > and
> > > >> >> > >> >> can
> > > >> >> > >> >> > > > be updated there. This is clever as it can allow
> > > >> replacing the
> > > >> >> > tokens
> > > >> >> > >> >> > > > once they run out of max life time, and clients can
> > > >> download
> > > >> >> > new
> > > >> >> > >> >> tokens
> > > >> >> > >> >> > > > from zookeeper. It shouldn't be a big load on
> > zookeeper
> > > >> as a
> > > >> >> > client
> > > >> >> > >> >> will
> > > >> >> > >> >> > > > need to get a new token once in several days. In this
> > > >> approach
> > > >> >> > you
> > > >> >> > >> >> don't
> > > >> >> > >> >> > > > need infinite lifetime on the token even for long
> > > running
> > > >> >> > clients.
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > > > 3) The token password are generated using a master
> > key.
> > > >> The
> > > >> >> > master
> > > >> >> > >> >> key
> > > >> >> > >> >> > > > should also be periodically changed. In Hadoop, the
> > > >> default
> > > >> >> > renewal
> > > >> >> > >> >> > > > period is 1 day.?
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > > IIUC, this will require brokers maintaining a list of X
> > > most
> > > >> >> > recent
> > > >> >> > >> >> master
> > > >> >> > >> >> > > keys. This list will have to be persisted somewhere, as
> > > if a
> > > >> >> > broker
> > > >> >> > >> >> goes
> > > >> >> > >> >> > > down it will have to get that list again and storing
> > > master
> > > >> keys
> > > >> >> > on ZK
> > > >> >> > >> >> is
> > > >> >> > >> >> > > not the best idea. However, if a broker goes down then
> > we
> > > >> have
> > > >> >> > much
> > > >> >> > >> >> bigger
> > > >> >> > >> >> > > issue to deal with and client can always
> > re-authenticate
> > > is
> > > >> such
> > > >> >> > >> >> events.
> > > >> >> > >> >> > >
> > > >> >> > >> >> > > Did you happen to take a look at other alternatives
> > this
> > > >> list has
> > > >> >> > >> >> > > suggested?
> > > >> >> > >> >> > >
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > > > Thanks for a thorough proposal, great work!"
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > > > On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira wrote:
> > > >> >> > >> >> > > > > Makes sense to me. Thanks!
> > > >> >> > >> >> > > > >
> > > >> >> > >> >> > > > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <
> > > ka...@harsha.io
> > > >> >
> > > >> >> > wrote:
> > > >> >> > >> >> > > > > > It doesn't need any release vehicle but still the
> > > >> work can
> > > >> >> > move
> > > >> >> > >> >> > > > forward.
> > > >> >> > >> >> > > > > > If anyone is interested in the KIP please do the
> > > >> review and
> > > >> >> > >> >> provide
> > > >> >> > >> >> > > the
> > > >> >> > >> >> > > > > > comments.
> > > >> >> > >> >> > > > > >
> > > >> >> > >> >> > > > > > -Harsha
> > > >> >> > >> >> > > > > >
> > > >> >> > >> >> > > > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael Juma
> > > wrote:
> > > >> >> > >> >> > > > > >> I agree that it would be good to have more time
> > to
> > > >> review
> > > >> >> > and
> > > >> >> > >> >> > > discuss
> > > >> >> > >> >> > > > > >> KIP-48.
> > > >> >> > >> >> > > > > >>
> > > >> >> > >> >> > > > > >> Ismael
> > > >> >> > >> >> > > > > >>
> > > >> >> > >> >> > > > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen Shapira <
> > > >> >> > >> >> g...@confluent.io>
> > > >> >> > >> >> > > > wrote:
> > > >> >> > >> >> > > > > >>
> > > >> >> > >> >> > > > > >> > Hi Team,
> > > >> >> > >> >> > > > > >> >
> > > >> >> > >> >> > > > > >> > Since KIP-48 depends on KIP-43, which is
> > > already a
> > > >> bit
> > > >> >> > of a
> > > >> >> > >> >> risk
> > > >> >> > >> >> > > for
> > > >> >> > >> >> > > > > >> > the next release - any chance we can delay
> > > >> delegation
> > > >> >> > tokens
> > > >> >> > >> >> to
> > > >> >> > >> >> > > > Kafka
> > > >> >> > >> >> > > > > >> > 0.10.1?
> > > >> >> > >> >> > > > > >> > With the community working on a release every
> > 3
> > > >> month,
> > > >> >> > this
> > > >> >> > >> >> is not
> > > >> >> > >> >> > > > a huge
> > > >> >> > >> >> > > > > >> > delay.
> > > >> >> > >> >> > > > > >> >
> > > >> >> > >> >> > > > > >> > Gwen
> > > >> >> > >> >> > > > > >> >
> > > >> >> > >> >> > > > > >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish Singh
> > <
> > > >> >> > >> >> > > asi...@cloudera.com>
> > > >> >> > >> >> > > > wrote:
> > > >> >> > >> >> > > > > >> > > Parth,
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > > Thanks again for the awesome write up.
> > > Following
> > > >> our
> > > >> >> > >> >> discussion
> > > >> >> > >> >> > > > from the
> > > >> >> > >> >> > > > > >> > > JIRA, I think it will be easier to compare
> > > >> various
> > > >> >> > >> >> alternatives
> > > >> >> > >> >> > > > if they
> > > >> >> > >> >> > > > > >> > are
> > > >> >> > >> >> > > > > >> > > listed together. I am stating below a few
> > > >> >> > alternatives along
> > > >> >> > >> >> > > with
> > > >> >> > >> >> > > > a the
> > > >> >> > >> >> > > > > >> > > current proposal.
> > > >> >> > >> >> > > > > >> > > (Current proposal) Store Delegation Token,
> > DT,
> > > >> on ZK.
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > > 1. Client authenticates with a broker.
> > > >> >> > >> >> > > > > >> > > 2. Once a client is authenticated, it
> > will
> > > >> make a
> > > >> >> > broker
> > > >> >> > >> >> side
> > > >> >> > >> >> > > > call to
> > > >> >> > >> >> > > > > >> > > issue a delegation token.
> > > >> >> > >> >> > > > > >> > > 3. The broker generates a shared secret
> > > based
> > > >> on
> > > >> >> > >> >> > > HMAC-SHA256(a
> > > >> >> > >> >> > > > > >> > > Password/Secret shared between all
> > brokers,
> > > >> >> > randomly
> > > >> >> > >> >> > > generated
> > > >> >> > >> >> > > > > >> > tokenId).
> > > >> >> > >> >> > > > > >> > > 4. Broker stores this token in its in
> > > memory
> > > >> cache.
> > > >> >> > >> >> Broker
> > > >> >> > >> >> > > > also stores
> > > >> >> > >> >> > > > > >> > > the DelegationToken without the hmac in
> > the
> > > >> >> > zookeeper.
> > > >> >> > >> >> > > > > >> > > 5. All brokers will have a cache backed
> > by
> > > >> >> > zookeeper so
> > > >> >> > >> >> they
> > > >> >> > >> >> > > > will all
> > > >> >> > >> >> > > > > >> > > get notified whenever a new token is
> > > >> generated and
> > > >> >> > they
> > > >> >> > >> >> will
> > > >> >> > >> >> > > > update
> > > >> >> > >> >> > > > > >> > their
> > > >> >> > >> >> > > > > >> > > local cache whenever token state changes.
> > > >> >> > >> >> > > > > >> > > 6. Broker returns the token to Client.
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > > 1. Probable race condition, client tries
> > to
> > > >> >> > authenticate
> > > >> >> > >> >> with
> > > >> >> > >> >> > > > a broker
> > > >> >> > >> >> > > > > >> > > that is yet to be updated with the newly
> > > >> generated
> > > >> >> > DT.
> > > >> >> > >> >> This
> > > >> >> > >> >> > > can
> > > >> >> > >> >> > > > > >> > probably be
> > > >> >> > >> >> > > > > >> > > dealt with making dtRequest block until
> > all
> > > >> >> > brokers have
> > > >> >> > >> >> > > > updated
> > > >> >> > >> >> > > > > >> > their DT
> > > >> >> > >> >> > > > > >> > > cache. Zk barrier or similar mechanism
> > can
> > > be
> > > >> used.
> > > >> >> > >> >> However,
> > > >> >> > >> >> > > > all such
> > > >> >> > >> >> > > > > >> > > mechanisms will increase complexity.
> > > >> >> > >> >> > > > > >> > > 2. Using a static secret key from config
> > > >> file. Will
> > > >> >> > >> >> require
> > > >> >> > >> >> > > yet
> > > >> >> > >> >> > > > > >> > another
> > > >> >> > >> >> > > > > >> > > config and uses a static secret key. It
> > is
> > > >> advised
> > > >> >> > to
> > > >> >> > >> >> rotate
> > > >> >> > >> >> > > > secret
> > > >> >> > >> >> > > > > >> > keys
> > > >> >> > >> >> > > > > >> > > periodically. This can be avoided with
> > > >> controller
> > > >> >> > >> >> generating
> > > >> >> > >> >> > > > > >> > secretKey and
> > > >> >> > >> >> > > > > >> > > passing to brokers periodically. However,
> > > >> this will
> > > >> >> > >> >> require
> > > >> >> > >> >> > > > brokers to
> > > >> >> > >> >> > > > > >> > > maintain certain counts of secretKeys.
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > > (Alternative 1) Have controller generate
> > > >> delegation
> > > >> >> > token.
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > > 1. Client authenticates with a broker.
> > > >> >> > >> >> > > > > >> > > 2. Once a client is authenticated, it
> > will
> > > >> make a
> > > >> >> > broker
> > > >> >> > >> >> side
> > > >> >> > >> >> > > > call to
> > > >> >> > >> >> > > > > >> > > issue a delegation token.
> > > >> >> > >> >> > > > > >> > > 3. Broker forwards the request to
> > > controller.
> > > >> >> > >> >> > > > > >> > > 4. Controller generates a DT and
> > broadcasts
> > > >> to all
> > > >> >> > >> >> brokers.
> > > >> >> > >> >> > > > > >> > > 5. Broker stores this token in its memory
> > > >> cache.
> > > >> >> > >> >> > > > > >> > > 6. Controller responds to broker’s DT
> > req.
> > > >> >> > >> >> > > > > >> > > 7. Broker returns the token to Client.
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > > 1. We will have to add new APIs to
> > support
> > > >> >> > controller
> > > >> >> > >> >> pushing
> > > >> >> > >> >> > > > tokens
> > > >> >> > >> >> > > > > >> > to
> > > >> >> > >> >> > > > > >> > > brokers on top of the minimal APIs that
> > are
> > > >> >> > currently
> > > >> >> > >> >> > > proposed.
> > > >> >> > >> >> > > > > >> > > 2. We will also have to add APIs to
> > support
> > > >> the
> > > >> >> > >> >> bootstrapping
> > > >> >> > >> >> > > > case,
> > > >> >> > >> >> > > > > >> > i.e,
> > > >> >> > >> >> > > > > >> > > when a new broker comes up it will have
> > to
> > > >> get all
> > > >> >> > >> >> delegation
> > > >> >> > >> >> > > > tokens
> > > >> >> > >> >> > > > > >> > from
> > > >> >> > >> >> > > > > >> > > the controller.
> > > >> >> > >> >> > > > > >> > > 3. In catastrophic failures where all
> > > brokers
> > > >> go
> > > >> >> > down,
> > > >> >> > >> >> the
> > > >> >> > >> >> > > > tokens will
> > > >> >> > >> >> > > > > >> > > be lost even if servers are restarted as
> > > >> tokens
> > > >> >> > are not
> > > >> >> > >> >> > > > persisted
> > > >> >> > >> >> > > > > >> > anywhere.
> > > >> >> > >> >> > > > > >> > > If this happens, then there are more
> > > important
> > > >> >> > things to
> > > >> >> > >> >> > > worry
> > > >> >> > >> >> > > > about
> > > >> >> > >> >> > > > > >> > and
> > > >> >> > >> >> > > > > >> > > maybe it is better to re-authenticate.
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > > (Alternative 2) Do not distribute DT to
> > other
> > > >> brokers
> > > >> >> > at
> > > >> >> > >> >> all.
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > > 1. Client authenticates with a broker.
> > > >> >> > >> >> > > > > >> > > 2. Once a client is authenticated, it
> > will
> > > >> make a
> > > >> >> > broker
> > > >> >> > >> >> side
> > > >> >> > >> >> > > > call to
> > > >> >> > >> >> > > > > >> > > issue a delegation token.
> > > >> >> > >> >> > > > > >> > > 3. The broker generates DT of form,
> > [hmac +
> > > >> (owner,
> > > >> >> > >> >> renewer,
> > > >> >> > >> >> > > > > >> > > maxLifeTime, id, hmac, expirationTime)]
> > and
> > > >> passes
> > > >> >> > back
> > > >> >> > >> >> this
> > > >> >> > >> >> > > > DT to
> > > >> >> > >> >> > > > > >> > client.
> > > >> >> > >> >> > > > > >> > > hmac is generated via {HMAC-SHA256(owner,
> > > >> renewer,
> > > >> >> > >> >> > > > maxLifeTime, id,
> > > >> >> > >> >> > > > > >> > hmac,
> > > >> >> > >> >> > > > > >> > > expirationTime) using SecretKey}. Note
> > that
> > > >> all
> > > >> >> > brokers
> > > >> >> > >> >> have
> > > >> >> > >> >> > > > this
> > > >> >> > >> >> > > > > >> > SecretKey.
> > > >> >> > >> >> > > > > >> > > 4. Client then goes to any broker and to
> > > >> >> > authenticate
> > > >> >> > >> >> sends
> > > >> >> > >> >> > > > the DT.
> > > >> >> > >> >> > > > > >> > > Broker recalculates hmac using (owner,
> > > >> renewer,
> > > >> >> > >> >> maxLifeTime,
> > > >> >> > >> >> > > > id, hmac,
> > > >> >> > >> >> > > > > >> > > expirationTime) info from DT and its
> > > >> SecretKey. If
> > > >> >> > it
> > > >> >> > >> >> matches
> > > >> >> > >> >> > > > with
> > > >> >> > >> >> > > > > >> > hmac of
> > > >> >> > >> >> > > > > >> > > DT, client is authenticated. Yes, it will
> > > do
> > > >> other
> > > >> >> > >> >> obvious
> > > >> >> > >> >> > > > checks of
> > > >> >> > >> >> > > > > >> > > timestamp expiry and such.
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > > Note that secret key will be generated by
> > > >> controller
> > > >> >> > and
> > > >> >> > >> >> passed
> > > >> >> > >> >> > > to
> > > >> >> > >> >> > > > > >> > brokers
> > > >> >> > >> >> > > > > >> > > periodically.
> > > >> >> > >> >> > > > > >> > > Probable issues and fixes
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > > 1. How to delete a DT? Yes, that is a
> > > downside
> > > >> >> > here.
> > > >> >> > >> >> However,
> > > >> >> > >> >> > > > this can
> > > >> >> > >> >> > > > > >> > > be handled with brokers maintaining a
> > > >> blacklist of
> > > >> >> > DTs,
> > > >> >> > >> >> DTs
> > > >> >> > >> >> > > > from this
> > > >> >> > >> >> > > > > >> > list
> > > >> >> > >> >> > > > > >> > > can be removed after expiry.
> > > >> >> > >> >> > > > > >> > > 2. In catastrophic failures where all
> > > brokers
> > > >> go
> > > >> >> > down,
> > > >> >> > >> >> the
> > > >> >> > >> >> > > > tokens will
> > > >> >> > >> >> > > > > >> > > be lost even if servers are restarted as
> > > >> tokens
> > > >> >> > are not
> > > >> >> > >> >> > > > persisted
> > > >> >> > >> >> > > > > >> > anywhere.
> > > >> >> > >> >> > > > > >> > > If this happens, then there are more
> > > important
> > > >> >> > things to
> > > >> >> > >> >> > > worry
> > > >> >> > >> >> > > > about
> > > >> >> > >> >> > > > > >> > and
> > > >> >> > >> >> > > > > >> > > maybe it is better to re-authenticate.
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > > On Fri, Feb 26, 2016 at 1:58 PM, Parth
> > > >> Brahmbhatt <
> > > >> >> > >> >> > > > > >> > > pbrahmbh...@hortonworks.com> wrote:
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > >> Hi,
> > > >> >> > >> >> > > > > >> > >>
> > > >> >> > >> >> > > > > >> > >> I have filed KIP-48 so we can offer hadoop
> > > like
> > > >> >> > delegation
> > > >> >> > >> >> > > > tokens in
> > > >> >> > >> >> > > > > >> > >> kafka. You can review the design
> > > >> >> > >> >> > > > > >> > >>
> > > >> >> > >> >> > > > > >> >
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > >
> > > >> >> > >> >>
> > > >> >> >
> > > >>
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> > > >> >> > >> >> > > > > >> > .
> > > >> >> > >> >> > > > > >> > >> This KIP depends on KIP-43 and we have also
> > > >> >> > discussed an
> > > >> >> > >> >> > > > alternative to
> > > >> >> > >> >> > > > > >> > >> proposed design here<
> > > >> >> > >> >> > > > > >> > >>
> > > >> >> > >> >> > > > > >> >
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > >
> > > >> >> > >> >>
> > > >> >> >
> > > >>
> > >
> > https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> > > >> >> > >> >> > > > > >> > >> >.
> > > >> >> > >> >> > > > > >> > >>
> > > >> >> > >> >> > > > > >> > >> Thanks
> > > >> >> > >> >> > > > > >> > >> Parth
> > > >> >> > >> >> > > > > >> > >>
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > > --
> > > >> >> > >> >> > > > > >> > >
> > > >> >> > >> >> > > > > >> > > Regards,
> > > >> >> > >> >> > > > > >> > > Ashish
> > > >> >> > >> >> > > > > >> >
> > > >> >> > >> >> > > >
> > > >> >> > >> >> > >
> > > >> >> > >> >> > >
> > > >> >> > >> >> > >
> > > >> >> > >> >> > > --
> > > >> >> > >> >> > >
> > > >> >> > >> >> > > Regards,
> > > >> >> > >> >> > > Ashish
> > > >> >> > >> >> > >
> > > >> >> > >> >>
> > > >> >> >
> > > >>
> > >
> >
