I agree. If we can really do pluggable authorization (and even pluggable authentication), it would not actually be hard to effectively implement a multi-tenant solution. I am hoping to attempt to implement something like this once there is code / patches for KIP-11 and KIP-12. So I wonder if we would actually need a KIP for this versus requirements for an authorization module (and possibly an authentication module) and then document a best practice for setting up a multi-tenant deployment.
I am a bit worried that KIPs 11/12 might not give us everything we need, but I think we can work through that as the implementation of both progresses On Tue, May 5, 2015 at 3:23 PM, Ashish Singh <asi...@cloudera.com> wrote: > Jay, > > I agree with you. This will kind of depend on how KIP-11 and KIP-12 shapes > up, but we can definitely start putting together the ideas. Would you > suggest starting a kip or just a generic shared document that can be later > translated into a KIP? > > On Tuesday, May 5, 2015, Jay Kreps <jay.kr...@gmail.com> wrote: > > > Hey guys, > > > > It would be nice to get a design around this. Though there are currently > so > > many big things in flight I do wonder if we should start another parallel > > thing...? But working out a design can't hurt. > > > > Personally I think since one of the goals of Kafka is data integration we > > really want to support a central cluster with many users. So I think we > > really want to have full hierarchies with all users being part of the > same > > "tree" of streams. This way granting permissions on a new stream just > means > > changing permissions on what "topic paths" a client has access to rather > > than needing a new client instance that connects to that separate > namespace > > (a la zookeeper). I think this will be a nice way to share config and acl > > defaults too. > > > > -Jay > > > > On Tue, May 5, 2015 at 10:36 AM, Ashish Singh <asi...@cloudera.com > > <javascript:;>> wrote: > > > > > Adrian, > > > > > > Trying to follow up the discussion here. Is my understanding correct > that > > > if we have topic hierarchies then we can do without namespaces. To me > > > namespace is an abstraction, it can be implemented with topic > hierarchies > > > as well, would you agree? If so I guess topic hierarchies is the way to > > go. > > > > > > One more thing I would like to put forward as an advantage of having > > topic > > > hierarchy is that we can support acls/ permissions inheritance in topic > > > hierarchies. This will avoid bootstrapping acls for each new topic. > > > > > > On Wed, Apr 29, 2015 at 7:42 AM, Adrian Preston <prest...@uk.ibm.com > > <javascript:;>> > > > wrote: > > > > > > > Thanks for your response. > > > > > > > > I agree. I think it would be useful to get consensus on how > > > > namespaces and topic-hierarchies relate to one another. To seed the > > > > discussion - here's my viewpoint, which I hope others will challenge: > > > > > > > > I see namespaces as something the broker provides to ensure that > > > > two tenants can never interact with one another - even if, > > > > for example, they both choose to use a topic called 'TOPIC1'. I > > > > imagine that this would be achieved by having the broker silently > > > > add a per tenant prefix to the topic name in each request, and > > > > strip it off in the response. So, for example, if 'TENANT1' sends > > > > a message to 'TOPIC1', this would be re-written so that the send is > > > > to 'TENANT1-TOPIC1'. > > > > > > > > If topic-hierarchies were available, then I think the prefix, > > > > added/removed to implement namespaces, would be the first level of > > > > qualification in the hierarchy. So, for example, if 'TENANT1' sends > > > > a message to 'TOPIC1', this would be re-written so that the send is > to > > > > 'TENANT1/TOPIC1'. > > > > > > > > Extrapolating from what's currently in KIP-21 (dynamic configuration) > > > > I guess that topic-hierarchies might result in the possibility for > > > > even finer grain topic configuration - e.g. a ZNode structure of: > > > > '/config/topics/<topic_level1>/<topic_level2>/...'. This would work > > > > best with an implementation of namespaces that was based on-top of > > > > topic-hierarchies, allowing configuration to be applied at the scope > > > > of: all tenants, one tenant, or one tenant's topics. > > > > > > > > So in summary: I think that namespaces can be usefully implemented > > > > independently of topic-hierarchies, and when topic-hierarchies are > > > > implemented would be easily integrated. > > > > > > > > Regards > > > > - Adrian > > > > > > > > -----Gwen Shapira <gshap...@cloudera.com <javascript:;>> wrote: > ----- > > > > To: "dev@kafka.apache.org <javascript:;>" <dev@kafka.apache.org > > <javascript:;>> > > > > From: Gwen Shapira <gshap...@cloudera.com <javascript:;>> > > > > Date: 04/28/2015 06:54PM > > > > Subject: Re: Adding multi-tenancy capabilities to Kafka > > > > > > > > I think recent discussion showed some need for topic namespaces - for > > > > example, Jun's use case for reserving topic names for specific users > > > > discussed under authorization. > > > > > > > > I think some discussion should happen on namespaces vs more > > full-fledged > > > > topic-hierarchy. > > > > I like the simplicity of namespaces, but there may be other > > requirements > > > > (such as inheriting configuration). > > > > > > > > Gwen > > > > > > > > On Tue, Apr 28, 2015 at 10:42 AM, Adrian Preston < > prest...@uk.ibm.com > > <javascript:;>> > > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > I've been looking at how a Kafka cluster could be used be deployed > so > > > > that > > > > > it can be used by multiple tenants. Specifically: a scheme where > the > > > > > clients belonging to each tenant receive the impression they are > > > > operating > > > > > against their own cluster. The ongoing security and quota work > looks > > > > like > > > > > it might provide a lot of the isolation requirements, but each > tenant > > > > would > > > > > currently share the same namespace for topics and consumer groups. > > So > > > > the > > > > > illusion of "it is my own cluster" is broken as soon as two tenants > > try > > > > > independent to use the same topic name. > > > > > > > > > > I wondered if other people on this list are interested in being > able > > to > > > > > support multiple tenants in this way? And / or if the ability to > > avoid > > > > > clashes in the topic namespace would be useful? I am considering > > > > > submitting a KIP in this area - but first wanted to get a feeling > for > > > > > whether these kinds of capabilities are of interest to others. > > > > > > > > > > Thanks in advance, > > > > > - Adrian > > > > > > > > > > Unless stated otherwise above: > > > > > IBM United Kingdom Limited - Registered in England and Wales with > > > number > > > > > 741598. > > > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire > > PO6 > > > > 3AU > > > > > > > > > > > > > > > > > > > > > > Unless stated otherwise above: > > > > IBM United Kingdom Limited - Registered in England and Wales with > > number > > > > 741598. > > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire > PO6 > > > 3AU > > > > > > > > > > > > > > > > > -- > > > > > > Regards, > > > Ashish > > > > > > > > -- > Ashish 🎤h >
