Thanks for your response. I agree. I think it would be useful to get consensus on how namespaces and topic-hierarchies relate to one another. To seed the discussion - here's my viewpoint, which I hope others will challenge:
I see namespaces as something the broker provides to ensure that two tenants can never interact with one another - even if, for example, they both choose to use a topic called 'TOPIC1'. I imagine that this would be achieved by having the broker silently add a per tenant prefix to the topic name in each request, and strip it off in the response. So, for example, if 'TENANT1' sends a message to 'TOPIC1', this would be re-written so that the send is to 'TENANT1-TOPIC1'. If topic-hierarchies were available, then I think the prefix, added/removed to implement namespaces, would be the first level of qualification in the hierarchy. So, for example, if 'TENANT1' sends a message to 'TOPIC1', this would be re-written so that the send is to 'TENANT1/TOPIC1'. Extrapolating from what's currently in KIP-21 (dynamic configuration) I guess that topic-hierarchies might result in the possibility for even finer grain topic configuration - e.g. a ZNode structure of: '/config/topics/<topic_level1>/<topic_level2>/...'. This would work best with an implementation of namespaces that was based on-top of topic-hierarchies, allowing configuration to be applied at the scope of: all tenants, one tenant, or one tenant's topics. So in summary: I think that namespaces can be usefully implemented independently of topic-hierarchies, and when topic-hierarchies are implemented would be easily integrated. Regards - Adrian -----Gwen Shapira <gshap...@cloudera.com> wrote: ----- To: "dev@kafka.apache.org" <dev@kafka.apache.org> From: Gwen Shapira <gshap...@cloudera.com> Date: 04/28/2015 06:54PM Subject: Re: Adding multi-tenancy capabilities to Kafka I think recent discussion showed some need for topic namespaces - for example, Jun's use case for reserving topic names for specific users discussed under authorization. I think some discussion should happen on namespaces vs more full-fledged topic-hierarchy. I like the simplicity of namespaces, but there may be other requirements (such as inheriting configuration). Gwen On Tue, Apr 28, 2015 at 10:42 AM, Adrian Preston <prest...@uk.ibm.com> wrote: > Hi all, > > I've been looking at how a Kafka cluster could be used be deployed so that > it can be used by multiple tenants. Specifically: a scheme where the > clients belonging to each tenant receive the impression they are operating > against their own cluster. The ongoing security and quota work looks like > it might provide a lot of the isolation requirements, but each tenant would > currently share the same namespace for topics and consumer groups. So the > illusion of "it is my own cluster" is broken as soon as two tenants try > independent to use the same topic name. > > I wondered if other people on this list are interested in being able to > support multiple tenants in this way? And / or if the ability to avoid > clashes in the topic namespace would be useful? I am considering > submitting a KIP in this area - but first wanted to get a feeling for > whether these kinds of capabilities are of interest to others. > > Thanks in advance, > - Adrian > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > > Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
