Jay, I agree with you. This will kind of depend on how KIP-11 and KIP-12 shapes up, but we can definitely start putting together the ideas. Would you suggest starting a kip or just a generic shared document that can be later translated into a KIP?
On Tuesday, May 5, 2015, Jay Kreps <jay.kr...@gmail.com> wrote: > Hey guys, > > It would be nice to get a design around this. Though there are currently so > many big things in flight I do wonder if we should start another parallel > thing...? But working out a design can't hurt. > > Personally I think since one of the goals of Kafka is data integration we > really want to support a central cluster with many users. So I think we > really want to have full hierarchies with all users being part of the same > "tree" of streams. This way granting permissions on a new stream just means > changing permissions on what "topic paths" a client has access to rather > than needing a new client instance that connects to that separate namespace > (a la zookeeper). I think this will be a nice way to share config and acl > defaults too. > > -Jay > > On Tue, May 5, 2015 at 10:36 AM, Ashish Singh <asi...@cloudera.com > <javascript:;>> wrote: > > > Adrian, > > > > Trying to follow up the discussion here. Is my understanding correct that > > if we have topic hierarchies then we can do without namespaces. To me > > namespace is an abstraction, it can be implemented with topic hierarchies > > as well, would you agree? If so I guess topic hierarchies is the way to > go. > > > > One more thing I would like to put forward as an advantage of having > topic > > hierarchy is that we can support acls/ permissions inheritance in topic > > hierarchies. This will avoid bootstrapping acls for each new topic. > > > > On Wed, Apr 29, 2015 at 7:42 AM, Adrian Preston <prest...@uk.ibm.com > <javascript:;>> > > wrote: > > > > > Thanks for your response. > > > > > > I agree. I think it would be useful to get consensus on how > > > namespaces and topic-hierarchies relate to one another. To seed the > > > discussion - here's my viewpoint, which I hope others will challenge: > > > > > > I see namespaces as something the broker provides to ensure that > > > two tenants can never interact with one another - even if, > > > for example, they both choose to use a topic called 'TOPIC1'. I > > > imagine that this would be achieved by having the broker silently > > > add a per tenant prefix to the topic name in each request, and > > > strip it off in the response. So, for example, if 'TENANT1' sends > > > a message to 'TOPIC1', this would be re-written so that the send is > > > to 'TENANT1-TOPIC1'. > > > > > > If topic-hierarchies were available, then I think the prefix, > > > added/removed to implement namespaces, would be the first level of > > > qualification in the hierarchy. So, for example, if 'TENANT1' sends > > > a message to 'TOPIC1', this would be re-written so that the send is to > > > 'TENANT1/TOPIC1'. > > > > > > Extrapolating from what's currently in KIP-21 (dynamic configuration) > > > I guess that topic-hierarchies might result in the possibility for > > > even finer grain topic configuration - e.g. a ZNode structure of: > > > '/config/topics/<topic_level1>/<topic_level2>/...'. This would work > > > best with an implementation of namespaces that was based on-top of > > > topic-hierarchies, allowing configuration to be applied at the scope > > > of: all tenants, one tenant, or one tenant's topics. > > > > > > So in summary: I think that namespaces can be usefully implemented > > > independently of topic-hierarchies, and when topic-hierarchies are > > > implemented would be easily integrated. > > > > > > Regards > > > - Adrian > > > > > > -----Gwen Shapira <gshap...@cloudera.com <javascript:;>> wrote: ----- > > > To: "dev@kafka.apache.org <javascript:;>" <dev@kafka.apache.org > <javascript:;>> > > > From: Gwen Shapira <gshap...@cloudera.com <javascript:;>> > > > Date: 04/28/2015 06:54PM > > > Subject: Re: Adding multi-tenancy capabilities to Kafka > > > > > > I think recent discussion showed some need for topic namespaces - for > > > example, Jun's use case for reserving topic names for specific users > > > discussed under authorization. > > > > > > I think some discussion should happen on namespaces vs more > full-fledged > > > topic-hierarchy. > > > I like the simplicity of namespaces, but there may be other > requirements > > > (such as inheriting configuration). > > > > > > Gwen > > > > > > On Tue, Apr 28, 2015 at 10:42 AM, Adrian Preston <prest...@uk.ibm.com > <javascript:;>> > > > wrote: > > > > > > > Hi all, > > > > > > > > I've been looking at how a Kafka cluster could be used be deployed so > > > that > > > > it can be used by multiple tenants. Specifically: a scheme where the > > > > clients belonging to each tenant receive the impression they are > > > operating > > > > against their own cluster. The ongoing security and quota work looks > > > like > > > > it might provide a lot of the isolation requirements, but each tenant > > > would > > > > currently share the same namespace for topics and consumer groups. > So > > > the > > > > illusion of "it is my own cluster" is broken as soon as two tenants > try > > > > independent to use the same topic name. > > > > > > > > I wondered if other people on this list are interested in being able > to > > > > support multiple tenants in this way? And / or if the ability to > avoid > > > > clashes in the topic namespace would be useful? I am considering > > > > submitting a KIP in this area - but first wanted to get a feeling for > > > > whether these kinds of capabilities are of interest to others. > > > > > > > > Thanks in advance, > > > > - Adrian > > > > > > > > Unless stated otherwise above: > > > > IBM United Kingdom Limited - Registered in England and Wales with > > number > > > > 741598. > > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire > PO6 > > > 3AU > > > > > > > > > > > > > > > > > Unless stated otherwise above: > > > IBM United Kingdom Limited - Registered in England and Wales with > number > > > 741598. > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 > > 3AU > > > > > > > > > > > > -- > > > > Regards, > > Ashish > > > -- Ashish 🎤h
