Unifying the two discussion threads on this KIP.
Here is the response from Jitendra
"The need for a large number of clients that are running all over the
cluster that authenticate with Kafka brokers, is very similar to the
Hadoop use case of large number of tasks running across the cluster that
need authentication to Hdfs Namenode. Therefore, the delegation token
approach does seem like a good fit for this use case as we have seen it
working at large scale in HDFS and YARN.
The proposed design is very much inline with Hadoop approach. A few
comments:
1) Why do you guys want to allow infinite renewable lifetime for a
token? HDFS restricts a token to a max life time (default 7 days). A
token's vulnerability is believed to increase with time.
2) As I understand the tokens are stored in zookeeper as well, and can
be updated there. This is clever as it can allow replacing the tokens
once they run out of max life time, and clients can download new tokens
from zookeeper. It shouldn't be a big load on zookeeper as a client will
need to get a new token once in several days. In this approach you don't
need infinite lifetime on the token even for long running clients.
3) The token password are generated using a master key. The master key
should also be periodically changed. In Hadoop, the default renewal
period is 1 day.?
Thanks for a thorough proposal, great work!"
On Mon, Mar 7, 2016, at 10:28 PM, Gwen Shapira wrote:
> Makes sense to me. Thanks!
>
> On Mon, Mar 7, 2016 at 9:25 PM, Harsha <ka...@harsha.io> wrote:
> > It doesn't need any release vehicle but still the work can move forward.
> > If anyone is interested in the KIP please do the review and provide the
> > comments.
> >
> > -Harsha
> >
> > On Mon, Mar 7, 2016, at 04:59 PM, Ismael Juma wrote:
> >> I agree that it would be good to have more time to review and discuss
> >> KIP-48.
> >>
> >> Ismael
> >>
> >> On Tue, Mar 8, 2016 at 12:55 AM, Gwen Shapira <g...@confluent.io> wrote:
> >>
> >> > Hi Team,
> >> >
> >> > Since KIP-48 depends on KIP-43, which is already a bit of a risk for
> >> > the next release - any chance we can delay delegation tokens to Kafka
> >> > 0.10.1?
> >> > With the community working on a release every 3 month, this is not a huge
> >> > delay.
> >> >
> >> > Gwen
> >> >
> >> > On Fri, Feb 26, 2016 at 5:11 PM, Ashish Singh <asi...@cloudera.com>
> >> > wrote:
> >> > > Parth,
> >> > >
> >> > > Thanks again for the awesome write up. Following our discussion from
> >> > > the
> >> > > JIRA, I think it will be easier to compare various alternatives if they
> >> > are
> >> > > listed together. I am stating below a few alternatives along with a the
> >> > > current proposal.
> >> > > (Current proposal) Store Delegation Token, DT, on ZK.
> >> > >
> >> > > 1. Client authenticates with a broker.
> >> > > 2. Once a client is authenticated, it will make a broker side call
> >> > > to
> >> > > issue a delegation token.
> >> > > 3. The broker generates a shared secret based on HMAC-SHA256(a
> >> > > Password/Secret shared between all brokers, randomly generated
> >> > tokenId).
> >> > > 4. Broker stores this token in its in memory cache. Broker also
> >> > > stores
> >> > > the DelegationToken without the hmac in the zookeeper.
> >> > > 5. All brokers will have a cache backed by zookeeper so they will
> >> > > all
> >> > > get notified whenever a new token is generated and they will update
> >> > their
> >> > > local cache whenever token state changes.
> >> > > 6. Broker returns the token to Client.
> >> > >
> >> > > Probable issues and fixes
> >> > >
> >> > > 1. Probable race condition, client tries to authenticate with a
> >> > > broker
> >> > > that is yet to be updated with the newly generated DT. This can
> >> > probably be
> >> > > dealt with making dtRequest block until all brokers have updated
> >> > their DT
> >> > > cache. Zk barrier or similar mechanism can be used. However, all
> >> > > such
> >> > > mechanisms will increase complexity.
> >> > > 2. Using a static secret key from config file. Will require yet
> >> > another
> >> > > config and uses a static secret key. It is advised to rotate secret
> >> > keys
> >> > > periodically. This can be avoided with controller generating
> >> > secretKey and
> >> > > passing to brokers periodically. However, this will require brokers
> >> > > to
> >> > > maintain certain counts of secretKeys.
> >> > >
> >> > > (Alternative 1) Have controller generate delegation token.
> >> > >
> >> > > 1. Client authenticates with a broker.
> >> > > 2. Once a client is authenticated, it will make a broker side call
> >> > > to
> >> > > issue a delegation token.
> >> > > 3. Broker forwards the request to controller.
> >> > > 4. Controller generates a DT and broadcasts to all brokers.
> >> > > 5. Broker stores this token in its memory cache.
> >> > > 6. Controller responds to broker’s DT req.
> >> > > 7. Broker returns the token to Client.
> >> > >
> >> > > Probable issues and fixes
> >> > >
> >> > > 1. We will have to add new APIs to support controller pushing tokens
> >> > to
> >> > > brokers on top of the minimal APIs that are currently proposed.
> >> > > 2. We will also have to add APIs to support the bootstrapping case,
> >> > i.e,
> >> > > when a new broker comes up it will have to get all delegation tokens
> >> > from
> >> > > the controller.
> >> > > 3. In catastrophic failures where all brokers go down, the tokens
> >> > > will
> >> > > be lost even if servers are restarted as tokens are not persisted
> >> > anywhere.
> >> > > If this happens, then there are more important things to worry about
> >> > and
> >> > > maybe it is better to re-authenticate.
> >> > >
> >> > > (Alternative 2) Do not distribute DT to other brokers at all.
> >> > >
> >> > > 1. Client authenticates with a broker.
> >> > > 2. Once a client is authenticated, it will make a broker side call
> >> > > to
> >> > > issue a delegation token.
> >> > > 3. The broker generates DT of form, [hmac + (owner, renewer,
> >> > > maxLifeTime, id, hmac, expirationTime)] and passes back this DT to
> >> > client.
> >> > > hmac is generated via {HMAC-SHA256(owner, renewer, maxLifeTime, id,
> >> > hmac,
> >> > > expirationTime) using SecretKey}. Note that all brokers have this
> >> > SecretKey.
> >> > > 4. Client then goes to any broker and to authenticate sends the DT.
> >> > > Broker recalculates hmac using (owner, renewer, maxLifeTime, id,
> >> > > hmac,
> >> > > expirationTime) info from DT and its SecretKey. If it matches with
> >> > hmac of
> >> > > DT, client is authenticated. Yes, it will do other obvious checks of
> >> > > timestamp expiry and such.
> >> > >
> >> > > Note that secret key will be generated by controller and passed to
> >> > brokers
> >> > > periodically.
> >> > > Probable issues and fixes
> >> > >
> >> > > 1. How to delete a DT? Yes, that is a downside here. However, this
> >> > > can
> >> > > be handled with brokers maintaining a blacklist of DTs, DTs from
> >> > > this
> >> > list
> >> > > can be removed after expiry.
> >> > > 2. In catastrophic failures where all brokers go down, the tokens
> >> > > will
> >> > > be lost even if servers are restarted as tokens are not persisted
> >> > anywhere.
> >> > > If this happens, then there are more important things to worry about
> >> > and
> >> > > maybe it is better to re-authenticate.
> >> > >
> >> > >
> >> > >
> >> > > On Fri, Feb 26, 2016 at 1:58 PM, Parth Brahmbhatt <
> >> > > pbrahmbh...@hortonworks.com> wrote:
> >> > >
> >> > >> Hi,
> >> > >>
> >> > >> I have filed KIP-48 so we can offer hadoop like delegation tokens in
> >> > >> kafka. You can review the design
> >> > >>
> >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka
> >> > .
> >> > >> This KIP depends on KIP-43 and we have also discussed an alternative
> >> > >> to
> >> > >> proposed design here<
> >> > >>
> >> > https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
> >> > >> >.
> >> > >>
> >> > >> Thanks
> >> > >> Parth
> >> > >>
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > >
> >> > > Regards,
> >> > > Ashish
> >> >
