Thanks for review Jitendra.
I don't like the idea of infinite lifetime but I see the Streaming use
case. Even for Streaming use case I was hoping there will be some notion of
master/driver that can get new delegation tokens at fixed interval and
distribute to workers. If that is not the case for we can discuss
delegation tokens renewing them self and the security implications of the
same.
I did not want clients to fetch tokens from zookeeper, overall I think its
better if clients don't rely on our metadata store and I think we are
moving in that direction with all the KIP-4 improvements. I chose
zookeeper as in this case the client will still just talk to broker , its
the brokers that will use zookeeper which we already do for a lot of other
usecases + ease of development + and the ability so tokens will survive
even a rolling restart/cluster failure. if a majority agrees the added
complexity to have controller forwarding keys to all broker is justified as
it provides tighter security , I am fine with that option too.
Given zookeeper does not support SSL we can not store master keys in
zookeeper as master keys will be exposed on wire. To support rotation
without affecting current clients is something I need to put more thought
in. My current proposal assumes the rotation will invalidate all current
tokens.
I request committers to also review and post their comments so we can make
progress on this KIP.
Thanks
Parth
On Tue, Apr 19, 2016 at 8:39 AM, Ashish Singh <asi...@cloudera.com> wrote:
> On Mon, Apr 18, 2016 at 11:26 AM, Harsha <ka...@harsha.io> wrote:
>
> > Unifying the two discussion threads on this KIP.
> >
> > Here is the response from Jitendra
> >
> > "The need for a large number of clients that are running all over the
> > cluster that authenticate with Kafka brokers, is very similar to the
> > Hadoop use case of large number of tasks running across the cluster that
> > need authentication to Hdfs Namenode. Therefore, the delegation token
> > approach does seem like a good fit for this use case as we have seen it
> > working at large scale in HDFS and YARN.
> >
> > The proposed design is very much inline with Hadoop approach. A few
> > comments:
> >
> > 1) Why do you guys want to allow infinite renewable lifetime for a
> > token? HDFS restricts a token to a max life time (default 7 days). A
> > token's vulnerability is believed to increase with time.
> >
> I agree that having infinite lifetime might not be the best idea.
>
> >
> > 2) As I understand the tokens are stored in zookeeper as well, and can
> > be updated there. This is clever as it can allow replacing the tokens
> > once they run out of max life time, and clients can download new tokens
> > from zookeeper. It shouldn't be a big load on zookeeper as a client will
> > need to get a new token once in several days. In this approach you don't
> > need infinite lifetime on the token even for long running clients.
> >
> > 3) The token password are generated using a master key. The master key
> > should also be periodically changed. In Hadoop, the default renewal
> > period is 1 day.?
> >
> IIUC, this will require brokers maintaining a list of X most recent master
> keys. This list will have to be persisted somewhere, as if a broker goes
> down it will have to get that list again and storing master keys on ZK is
> not the best idea. However, if a broker goes down then we have much bigger
> issue to deal with and client can always re-authenticate is such events.
>
> Did you happen to take a look at other alternatives this list has
> suggested?
>
> >
> > Thanks for a thorough proposal, great work!"
> >
> >
> >
> > On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira wrote:
> > > Makes sense to me. Thanks!
> > >
> > > On Mon, Mar 7, 2016 at 9:25 PM, Harsha <ka...@harsha.io> wrote:
> > > > It doesn't need any release vehicle but still the work can move
> > forward.
> > > > If anyone is interested in the KIP please do the review and provide
> the
> > > > comments.
> > > >
> > > > -Harsha
> > > >
> > > > On Mon, Mar 7, 2016, at 04:59 PM, Ismael Juma wrote:
> > > >> I agree that it would be good to have more time to review and
> discuss
> > > >> KIP-48.
> > > >>
> > > >> Ismael
> > > >>
> > > >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen Shapira <g...@confluent.io>
> > wrote:
> > > >>
> > > >> > Hi Team,
> > > >> >
> > > >> > Since KIP-48 depends on KIP-43, which is already a bit of a risk
> for
> > > >> > the next release - any chance we can delay delegation tokens to
> > Kafka
> > > >> > 0.10.1?
> > > >> > With the community working on a release every 3 month, this is not
> > a huge
> > > >> > delay.
> > > >> >
> > > >> > Gwen
> > > >> >
> > > >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish Singh <
> asi...@cloudera.com>
> > wrote:
> > > >> > > Parth,
> > > >> > >
> > > >> > > Thanks again for the awesome write up. Following our discussion
> > from the
> > > >> > > JIRA, I think it will be easier to compare various alternatives
> > if they
> > > >> > are
> > > >> > > listed together. I am stating below a few alternatives along
> with
> > a the
> > > >> > > current proposal.
> > > >> > > (Current proposal) Store Delegation Token, DT, on ZK.
> > > >> > >
> > > >> > > 1. Client authenticates with a broker.
> > > >> > > 2. Once a client is authenticated, it will make a broker side
> > call to
> > > >> > > issue a delegation token.
> > > >> > > 3. The broker generates a shared secret based on
> HMAC-SHA256(a
> > > >> > > Password/Secret shared between all brokers, randomly
> generated
> > > >> > tokenId).
> > > >> > > 4. Broker stores this token in its in memory cache. Broker
> > also stores
> > > >> > > the DelegationToken without the hmac in the zookeeper.
> > > >> > > 5. All brokers will have a cache backed by zookeeper so they
> > will all
> > > >> > > get notified whenever a new token is generated and they will
> > update
> > > >> > their
> > > >> > > local cache whenever token state changes.
> > > >> > > 6. Broker returns the token to Client.
> > > >> > >
> > > >> > > Probable issues and fixes
> > > >> > >
> > > >> > > 1. Probable race condition, client tries to authenticate with
> > a broker
> > > >> > > that is yet to be updated with the newly generated DT. This
> can
> > > >> > probably be
> > > >> > > dealt with making dtRequest block until all brokers have
> > updated
> > > >> > their DT
> > > >> > > cache. Zk barrier or similar mechanism can be used. However,
> > all such
> > > >> > > mechanisms will increase complexity.
> > > >> > > 2. Using a static secret key from config file. Will require
> yet
> > > >> > another
> > > >> > > config and uses a static secret key. It is advised to rotate
> > secret
> > > >> > keys
> > > >> > > periodically. This can be avoided with controller generating
> > > >> > secretKey and
> > > >> > > passing to brokers periodically. However, this will require
> > brokers to
> > > >> > > maintain certain counts of secretKeys.
> > > >> > >
> > > >> > > (Alternative 1) Have controller generate delegation token.
> > > >> > >
> > > >> > > 1. Client authenticates with a broker.
> > > >> > > 2. Once a client is authenticated, it will make a broker side
> > call to
> > > >> > > issue a delegation token.
> > > >> > > 3. Broker forwards the request to controller.
> > > >> > > 4. Controller generates a DT and broadcasts to all brokers.
> > > >> > > 5. Broker stores this token in its memory cache.
> > > >> > > 6. Controller responds to broker’s DT req.
> > > >> > > 7. Broker returns the token to Client.
> > > >> > >
> > > >> > > Probable issues and fixes
> > > >> > >
> > > >> > > 1. We will have to add new APIs to support controller pushing
> > tokens
> > > >> > to
> > > >> > > brokers on top of the minimal APIs that are currently
> proposed.
> > > >> > > 2. We will also have to add APIs to support the bootstrapping
> > case,
> > > >> > i.e,
> > > >> > > when a new broker comes up it will have to get all delegation
> > tokens
> > > >> > from
> > > >> > > the controller.
> > > >> > > 3. In catastrophic failures where all brokers go down, the
> > tokens will
> > > >> > > be lost even if servers are restarted as tokens are not
> > persisted
> > > >> > anywhere.
> > > >> > > If this happens, then there are more important things to
> worry
> > about
> > > >> > and
> > > >> > > maybe it is better to re-authenticate.
> > > >> > >
> > > >> > > (Alternative 2) Do not distribute DT to other brokers at all.
> > > >> > >
> > > >> > > 1. Client authenticates with a broker.
> > > >> > > 2. Once a client is authenticated, it will make a broker side
> > call to
> > > >> > > issue a delegation token.
> > > >> > > 3. The broker generates DT of form, [hmac + (owner, renewer,
> > > >> > > maxLifeTime, id, hmac, expirationTime)] and passes back this
> > DT to
> > > >> > client.
> > > >> > > hmac is generated via {HMAC-SHA256(owner, renewer,
> > maxLifeTime, id,
> > > >> > hmac,
> > > >> > > expirationTime) using SecretKey}. Note that all brokers have
> > this
> > > >> > SecretKey.
> > > >> > > 4. Client then goes to any broker and to authenticate sends
> > the DT.
> > > >> > > Broker recalculates hmac using (owner, renewer, maxLifeTime,
> > id, hmac,
> > > >> > > expirationTime) info from DT and its SecretKey. If it matches
> > with
> > > >> > hmac of
> > > >> > > DT, client is authenticated. Yes, it will do other obvious
> > checks of
> > > >> > > timestamp expiry and such.
> > > >> > >
> > > >> > > Note that secret key will be generated by controller and passed
> to
> > > >> > brokers
> > > >> > > periodically.
> > > >> > > Probable issues and fixes
> > > >> > >
> > > >> > > 1. How to delete a DT? Yes, that is a downside here. However,
> > this can
> > > >> > > be handled with brokers maintaining a blacklist of DTs, DTs
> > from this
> > > >> > list
> > > >> > > can be removed after expiry.
> > > >> > > 2. In catastrophic failures where all brokers go down, the
> > tokens will
> > > >> > > be lost even if servers are restarted as tokens are not
> > persisted
> > > >> > anywhere.
> > > >> > > If this happens, then there are more important things to
> worry
> > about
> > > >> > and
> > > >> > > maybe it is better to re-authenticate.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Fri, Feb 26, 2016 at 1:58 PM, Parth Brahmbhatt <
> > > >> > > pbrahmbh...@hortonworks.com> wrote:
> > > >> > >
> > > >> > >> Hi,
> > > >> > >>
> > > >> > >> I have filed KIP-48 so we can offer hadoop like delegation
> > tokens in
> > > >> > >> kafka. You can review the design
> > > >> > >>
> > > >> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> > > >> > .
> > > >> > >> This KIP depends on KIP-43 and we have also discussed an
> > alternative to
> > > >> > >> proposed design here<
> > > >> > >>
> > > >> >
> >
> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> > > >> > >> >.
> > > >> > >>
> > > >> > >> Thanks
> > > >> > >> Parth
> > > >> > >>
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > --
> > > >> > >
> > > >> > > Regards,
> > > >> > > Ashish
> > > >> >
> >
>
>
>
> --
>
> Regards,
> Ashish
>
