Hi, Parth, How does this work in Hive? I thought authorization in Hive always goes through it's SQL cli for any authorization plugin. When integrating with Ranger(Argus), does Hive do authorization through a separate CLI?
Thanks, Jun On Fri, Apr 17, 2015 at 11:01 AM, Parth Brahmbhatt < pbrahmbh...@hortonworks.com> wrote: > We could do this but I think its too simplistic plus now we are adding > authorization related options in CLI which I thought everyone wants to > avoid. > > When I say its too simplistic I mean there are missing options like > —hosts, what happens when we start supporting group now we will probably > end up adding "—grant —groups”. I think we will just endup polluting kafka > create CLI with all the different acl options or we will have 2 CLIs one > for the basic stuff and for anything advance you will have to use a > different tool. It might be better to just have a single separate ACL > management CLI. > > Thanks > Parth > > On 4/17/15, 10:42 AM, "Gwen Shapira" <gshap...@cloudera.com> wrote: > > >I've probably been a DBA for too long, but I imagined something like: > >kafka-topic --topic t1 --grant user --action action > >kafka-topic --topic t1 --revoke user --action action > >(i.e. the commandline equivalent of "grant select on table1 to > >gwenshap" and "revoke select on table2 from gwenshap") > > > >When you need gazillion of them, you generate a script with gazillion > >of those and execute. > > > >Maybe it just looks reasonable to me because I'm used to it, though :) > > > >Or maybe including the json parsing code in TopicCommand is not so bad? > > > > > > > >On Fri, Apr 17, 2015 at 10:30 AM, Parth Brahmbhatt > ><pbrahmbh...@hortonworks.com> wrote: > >> * Yes, Acl pretty much captures everything. Originally I had resource as > >> part of Acls, we can go back to that. > >> * The describe can call getAcl and I plan to do so. addAcl is tricky > >> because the user will have to specify the acls through command lines, > >> which will probably be a location to some file. Basically the CLI won¹t > >> know how to parse user input and convert it to a principal/acl that the > >> plugin understands. We could add an API in authorizer that can take a > >>file > >> as input if we want ‹acl as an option during create. > >> * Yes also getAcls(Principal principal). > >> > >> Thanks > >> Parth > >> > >> > >> On 4/17/15, 10:05 AM, "Gwen Shapira" <gshap...@cloudera.com> wrote: > >> > >>>On Fri, Apr 17, 2015 at 9:31 AM, Parth Brahmbhatt > >>><pbrahmbh...@hortonworks.com> wrote: > >>>> I was following the storm model but I think this is a reasonable > >>>>change. I recommend changing the API names to addAcls, removeAcls and > >>>>getAcls. > >>> > >>>And they probably just need to get List<Acl> instead of everything I > >>>specified? Looks like Acl encapsulates everything we need. > >>> > >>>> Couple of points to ensure we are on same page: > >>>> * With this approach the kafka command line will not provide a way to > >>>>add/edit acls during topic creation, neither it will provide a way to > >>>>modify the acls. It will be up to the authorizer to either define a > >>>>command line utility or to allow other means to add acls(CLI/UI/REST). > >>>>For the default implementation we can provide CLI. > >>> > >>>You looked into this deeper than I did - is there a reason > >>>TopicCommand can't invoke addACL and getACL? > >>> > >>>> * We probably want to add List<Acl> getAcls(Resource resource) so > >>>>users > >>>>can list all acls on a topic. > >>> > >>>Also getAcls(Principal princ)? > >>> > >>>> > >>>> I haven¹t looked at how consumer offsets are currently stored so I > >>>>will > >>>>have to take a look but I think that is implementation detail. > >>>> > >>>> Gwen,Jun and other interested parties, do you have time to jump on a > >>>>quick hangout so we can go over some of the lower level details? > >>>> > >>>> Thanks > >>>> Parth > >>>> From: Tong Li <liton...@us.ibm.com<mailto:liton...@us.ibm.com>> > >>>> Reply-To: "dev@kafka.apache.org<mailto:dev@kafka.apache.org>" > >>>><dev@kafka.apache.org<mailto:dev@kafka.apache.org>> > >>>> Date: Friday, April 17, 2015 at 7:34 AM > >>>> To: "dev@kafka.apache.org<mailto:dev@kafka.apache.org>" > >>>><dev@kafka.apache.org<mailto:dev@kafka.apache.org>> > >>>> Subject: Re: [DISCUSSION] KIP-11: ACL Management > >>>> > >>>> > >>>> Gwen, > >>>> There is one product called ElasticSearch which has been > >>>>quite > >>>>successful. They recently added security, what they actually did is > >>>>quite nice. They really separated Authentication and Authorization > >>>>which > >>>>many people get really confused about and often mix them up. I looked > >>>>through what they did and quite impressed by it, I think there are many > >>>>things we can borrow from. Here is a link to it. > >>>>http://www.elastic.co/guide/en/shield/current/architecture.html. The > >>>>product name is called "shield" which is implemented as an > >>>>ElasticSearch > >>>>plugin. The promise here is that you can have a running ElasticSearch, > >>>>then you install this plugin, configure it, then your ElasticSearch > >>>>service is secured. The goal should be really the same for Kafka, you > >>>>have a Kafka service running, you install a new plugin (in this case > >>>>security plugin), configure it, then your Kafka service is secured. I > >>>>think that the key here is that we should introduce a true pluggable > >>>>framework in Kafka which allows security, quota, encryption, > >>>>compression, serialization/deserialization all being developed as > >>>>plugins which can be all easily added and configured onto a running > >>>>Kafka service, then the functions/features provided by the plugins will > >>>>start working. Once we have this framework in, how a security plugin > >>>>works internally becomes the really the concern of that plugin, for > >>>>example, how a new user gets registered, permission granted, revoked, > >>>>all these will be the concern of that plugin, rest of the Kafka > >>>>components should not really be concerned about them. This way we are > >>>>really following the design principal (Separation of concerns). With > >>>>all that, what I am proposing is a true pluggable framework > >>>>introduction > >>>>into Kafka which I have also talked about in a previous email. For > >>>>security we can implement a simple file based security plugin, other > >>>>plugins such as LDAP, AD for authentication can come later, plugin for > >>>>authorization such as RBAC can also come later if people care so much > >>>>about using them. > >>>> > >>>> Thanks. > >>>> > >>>> Tong Li > >>>> OpenStack & Kafka Community Development > >>>> Building 501/B205 > >>>> liton...@us.ibm.com<mailto:liton...@us.ibm.com> > >>>> > >>>> [Inactive hide details for Gwen Shapira ---04/16/2015 12:44:54 PM---Hi > >>>>Kafka Authorization Fans, I'm starting a new thread on a]Gwen Shapira > >>>>---04/16/2015 12:44:54 PM---Hi Kafka Authorization Fans, I'm starting a > >>>>new thread on a specific sub-topic of KIP-11, since > >>>> > >>>> From: Gwen Shapira > >>>><gshap...@cloudera.com<mailto:gshap...@cloudera.com>> > >>>> To: "dev@kafka.apache.org<mailto:dev@kafka.apache.org>" > >>>><dev@kafka.apache.org<mailto:dev@kafka.apache.org>> > >>>> Date: 04/16/2015 12:44 PM > >>>> Subject: [DISCUSSION] KIP-11: ACL Management > >>>> > >>>> ________________________________ > >>>> > >>>> > >>>> > >>>> Hi Kafka Authorization Fans, > >>>> > >>>> I'm starting a new thread on a specific sub-topic of KIP-11, since > >>>> this is a bit long :) > >>>> > >>>> Currently KIP-11, as I understand it, proposes: > >>>> * Authorizers are pluggable, with Kafka providing DefaultAuthorizer. > >>>> * Kafka tools allow adding / managing ACLs. > >>>> * Those ACLs are stored in ZK and cached in a new TopicCache > >>>> * Authorizers can either use the ACLs defined and stored in Kafka, or > >>>> define and use their own. > >>>> > >>>> I am concerned of two possible issues with this design: > >>>> 1. Separation of concerns - only authorizers should worry about ACLs, > >>>> and therefore the less code for ACLs that exist in Kafka core, the > >>>> better. > >>>> 2. User confusion - It sounded like we can define ACLs in Kafka itself > >>>> but authorizers can also define their own, so "kafka-topics > >>>> --describe" may show an ACL different than the one in use. This can be > >>>> super confusing for admins. > >>>> > >>>> My alternative suggestion: > >>>> * Authorizer API will include: > >>>> grantPrivilege(List<Principals>, List<Privilege>) > >>>> revokePrivilege(List<Principals>, List<Privilege>), > >>>> getPrivilegesByPrincipal(Principal, Resource) > >>>> .... > >>>> (The exact API can be discussed in detail, but you get the idea) > >>>> * Kafka tools will simply invoke these APIs when topics are added / > >>>> modified / described. > >>>> * Each authorizer (including the default one) will be responsible for > >>>> storing, caching and using those ACLs. > >>>> > >>>> This way, we keep almost all ACL code with the Authorizer, where it > >>>> belongs and users get a nice unified interface that reflects what is > >>>> actually getting used in the system. > >>>> This is pretty much how Sqoop and Hive implement their authorization > >>>>APIs. > >>>> > >>>> What do you think? > >>>> > >>>> Gwen > >>>> > >> > >
