Hey guys, It would be nice to get a design around this. Though there are currently so many big things in flight I do wonder if we should start another parallel thing...? But working out a design can't hurt.
Personally I think since one of the goals of Kafka is data integration we really want to support a central cluster with many users. So I think we really want to have full hierarchies with all users being part of the same "tree" of streams. This way granting permissions on a new stream just means changing permissions on what "topic paths" a client has access to rather than needing a new client instance that connects to that separate namespace (a la zookeeper). I think this will be a nice way to share config and acl defaults too. -Jay On Tue, May 5, 2015 at 10:36 AM, Ashish Singh <asi...@cloudera.com> wrote: > Adrian, > > Trying to follow up the discussion here. Is my understanding correct that > if we have topic hierarchies then we can do without namespaces. To me > namespace is an abstraction, it can be implemented with topic hierarchies > as well, would you agree? If so I guess topic hierarchies is the way to go. > > One more thing I would like to put forward as an advantage of having topic > hierarchy is that we can support acls/ permissions inheritance in topic > hierarchies. This will avoid bootstrapping acls for each new topic. > > On Wed, Apr 29, 2015 at 7:42 AM, Adrian Preston <prest...@uk.ibm.com> > wrote: > > > Thanks for your response. > > > > I agree. I think it would be useful to get consensus on how > > namespaces and topic-hierarchies relate to one another. To seed the > > discussion - here's my viewpoint, which I hope others will challenge: > > > > I see namespaces as something the broker provides to ensure that > > two tenants can never interact with one another - even if, > > for example, they both choose to use a topic called 'TOPIC1'. I > > imagine that this would be achieved by having the broker silently > > add a per tenant prefix to the topic name in each request, and > > strip it off in the response. So, for example, if 'TENANT1' sends > > a message to 'TOPIC1', this would be re-written so that the send is > > to 'TENANT1-TOPIC1'. > > > > If topic-hierarchies were available, then I think the prefix, > > added/removed to implement namespaces, would be the first level of > > qualification in the hierarchy. So, for example, if 'TENANT1' sends > > a message to 'TOPIC1', this would be re-written so that the send is to > > 'TENANT1/TOPIC1'. > > > > Extrapolating from what's currently in KIP-21 (dynamic configuration) > > I guess that topic-hierarchies might result in the possibility for > > even finer grain topic configuration - e.g. a ZNode structure of: > > '/config/topics/<topic_level1>/<topic_level2>/...'. This would work > > best with an implementation of namespaces that was based on-top of > > topic-hierarchies, allowing configuration to be applied at the scope > > of: all tenants, one tenant, or one tenant's topics. > > > > So in summary: I think that namespaces can be usefully implemented > > independently of topic-hierarchies, and when topic-hierarchies are > > implemented would be easily integrated. > > > > Regards > > - Adrian > > > > -----Gwen Shapira <gshap...@cloudera.com> wrote: ----- > > To: "dev@kafka.apache.org" <dev@kafka.apache.org> > > From: Gwen Shapira <gshap...@cloudera.com> > > Date: 04/28/2015 06:54PM > > Subject: Re: Adding multi-tenancy capabilities to Kafka > > > > I think recent discussion showed some need for topic namespaces - for > > example, Jun's use case for reserving topic names for specific users > > discussed under authorization. > > > > I think some discussion should happen on namespaces vs more full-fledged > > topic-hierarchy. > > I like the simplicity of namespaces, but there may be other requirements > > (such as inheriting configuration). > > > > Gwen > > > > On Tue, Apr 28, 2015 at 10:42 AM, Adrian Preston <prest...@uk.ibm.com> > > wrote: > > > > > Hi all, > > > > > > I've been looking at how a Kafka cluster could be used be deployed so > > that > > > it can be used by multiple tenants. Specifically: a scheme where the > > > clients belonging to each tenant receive the impression they are > > operating > > > against their own cluster. The ongoing security and quota work looks > > like > > > it might provide a lot of the isolation requirements, but each tenant > > would > > > currently share the same namespace for topics and consumer groups. So > > the > > > illusion of "it is my own cluster" is broken as soon as two tenants try > > > independent to use the same topic name. > > > > > > I wondered if other people on this list are interested in being able to > > > support multiple tenants in this way? And / or if the ability to avoid > > > clashes in the topic namespace would be useful? I am considering > > > submitting a KIP in this area - but first wanted to get a feeling for > > > whether these kinds of capabilities are of interest to others. > > > > > > Thanks in advance, > > > - Adrian > > > > > > Unless stated otherwise above: > > > IBM United Kingdom Limited - Registered in England and Wales with > number > > > 741598. > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 > > 3AU > > > > > > > > > > > > Unless stated otherwise above: > > IBM United Kingdom Limited - Registered in England and Wales with number > > 741598. > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 > 3AU > > > > > > > -- > > Regards, > Ashish >
