Hi Team,
Since KIP-48 depends on KIP-43, which is already a bit of a risk for
the next release - any chance we can delay delegation tokens to Kafka
0.10.1?
With the community working on a release every 3 month, this is not a huge delay.
Gwen
On Fri, Feb 26, 2016 at 5:11 PM, Ashish Singh <asi...@cloudera.com> wrote:
> Parth,
>
> Thanks again for the awesome write up. Following our discussion from the
> JIRA, I think it will be easier to compare various alternatives if they are
> listed together. I am stating below a few alternatives along with a the
> current proposal.
> (Current proposal) Store Delegation Token, DT, on ZK.
>
> 1. Client authenticates with a broker.
> 2. Once a client is authenticated, it will make a broker side call to
> issue a delegation token.
> 3. The broker generates a shared secret based on HMAC-SHA256(a
> Password/Secret shared between all brokers, randomly generated tokenId).
> 4. Broker stores this token in its in memory cache. Broker also stores
> the DelegationToken without the hmac in the zookeeper.
> 5. All brokers will have a cache backed by zookeeper so they will all
> get notified whenever a new token is generated and they will update their
> local cache whenever token state changes.
> 6. Broker returns the token to Client.
>
> Probable issues and fixes
>
> 1. Probable race condition, client tries to authenticate with a broker
> that is yet to be updated with the newly generated DT. This can probably be
> dealt with making dtRequest block until all brokers have updated their DT
> cache. Zk barrier or similar mechanism can be used. However, all such
> mechanisms will increase complexity.
> 2. Using a static secret key from config file. Will require yet another
> config and uses a static secret key. It is advised to rotate secret keys
> periodically. This can be avoided with controller generating secretKey and
> passing to brokers periodically. However, this will require brokers to
> maintain certain counts of secretKeys.
>
> (Alternative 1) Have controller generate delegation token.
>
> 1. Client authenticates with a broker.
> 2. Once a client is authenticated, it will make a broker side call to
> issue a delegation token.
> 3. Broker forwards the request to controller.
> 4. Controller generates a DT and broadcasts to all brokers.
> 5. Broker stores this token in its memory cache.
> 6. Controller responds to broker’s DT req.
> 7. Broker returns the token to Client.
>
> Probable issues and fixes
>
> 1. We will have to add new APIs to support controller pushing tokens to
> brokers on top of the minimal APIs that are currently proposed.
> 2. We will also have to add APIs to support the bootstrapping case, i.e,
> when a new broker comes up it will have to get all delegation tokens from
> the controller.
> 3. In catastrophic failures where all brokers go down, the tokens will
> be lost even if servers are restarted as tokens are not persisted anywhere.
> If this happens, then there are more important things to worry about and
> maybe it is better to re-authenticate.
>
> (Alternative 2) Do not distribute DT to other brokers at all.
>
> 1. Client authenticates with a broker.
> 2. Once a client is authenticated, it will make a broker side call to
> issue a delegation token.
> 3. The broker generates DT of form, [hmac + (owner, renewer,
> maxLifeTime, id, hmac, expirationTime)] and passes back this DT to client.
> hmac is generated via {HMAC-SHA256(owner, renewer, maxLifeTime, id, hmac,
> expirationTime) using SecretKey}. Note that all brokers have this
> SecretKey.
> 4. Client then goes to any broker and to authenticate sends the DT.
> Broker recalculates hmac using (owner, renewer, maxLifeTime, id, hmac,
> expirationTime) info from DT and its SecretKey. If it matches with hmac of
> DT, client is authenticated. Yes, it will do other obvious checks of
> timestamp expiry and such.
>
> Note that secret key will be generated by controller and passed to brokers
> periodically.
> Probable issues and fixes
>
> 1. How to delete a DT? Yes, that is a downside here. However, this can
> be handled with brokers maintaining a blacklist of DTs, DTs from this list
> can be removed after expiry.
> 2. In catastrophic failures where all brokers go down, the tokens will
> be lost even if servers are restarted as tokens are not persisted anywhere.
> If this happens, then there are more important things to worry about and
> maybe it is better to re-authenticate.
>
>
>
> On Fri, Feb 26, 2016 at 1:58 PM, Parth Brahmbhatt <
> pbrahmbh...@hortonworks.com> wrote:
>
>> Hi,
>>
>> I have filed KIP-48 so we can offer hadoop like delegation tokens in
>> kafka. You can review the design
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-48+Delegation+token+support+for+Kafka.
>> This KIP depends on KIP-43 and we have also discussed an alternative to
>> proposed design here<
>> https://issues.apache.org/jira/browse/KAFKA-1696?focusedCommentId=15167800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15167800
>> >.
>>
>> Thanks
>> Parth
>>
>
>
>
> --
>
> Regards,
> Ashish
