Hi Christian, Answering your question below: > Let's assume we just have one topic with 10 partitions for simplicity. We can now use the environment id as a key for the messages to make sure the messages of each environment arrive in order while sharing the load on the partitions.
> Now we want each environment to only read the minimal number of messages while consuming. Ideally we would like to to only consume its own messages. Can we somehow filter to only receive messages with a certain key? Can we maybe only listen to a certain partition at least? Unfortunately, Kafka doesn't have the feature to filter the messages on broker before sending to consumer. But for your 2nd question: > Can we maybe only listen to a certain partition at least? Actually, yes. Kafka has a way to just fetch data from a certain partition of a topic. You can use Consumer#assign API to achieve that. So, to do that, I think you also need to have a custom producer partitioner for your purpose. Let's say, in your example, you have 10 partitions, and 10 environments. Your partitioner should send to the specific partition based on the environment ID, ex: env ID 1 -> partition 1, env ID 2 -> partition 2.... So, in your consumer, you can just assign to the partition containing its environment ID. And for the idea of encrypting the messages to achieve isolation, it's interesting! I've never thought about it! :) Hope it helps. Thank you. Luke On Wed, Dec 8, 2021 at 4:48 PM Christian Schneider <ch...@die-schneider.net> wrote: > We have a single tenant application that we deploy to a kubernetes cluster > in many instances. > Every customer has several environments of the application. Each > application lives in a separate namespace and should be isolated from other > applications. > > We plan to use kafka to communicate inside an environment (between the > different pods). > As setting up one kafka cluster per such environment is a lot of overhead > and cost we would like to just use a single multi tenant kafka cluster. > > Let's assume we just have one topic with 10 partitions for simplicity. > We can now use the environment id as a key for the messages to make sure > the messages of each environment arrive in order while sharing the load on > the partitions. > > Now we want each environment to only read the minimal number of messages > while consuming. Ideally we would like to to only consume its own messages. > Can we somehow filter to only > receive messages with a certain key? Can we maybe only listen to a certain > partition at least? > > Additionally we ideally would like to have enforced isolation. So each > environment can only see its own messages even if it might receive messages > of other environments from the same partition. > I think in worst case we can make this happen by encrypting the messages > but it would be great if we could filter on broker side. > > Christian > > -- > -- > Christian Schneider > http://www.liquid-reality.de > > Computer Scientist > http://www.adobe.com >
