Hi Christian, > For multiple partitions is it the correct behaviour to simply assign to partition number:offset or do I have to provide offsets for the other partitions too?
I'm not sure I get your question here. If you are asking if you should commit offsets of other partitions that this consumer doesn't consumed, then you don't have to. For example: consumer A assign to partition 0 consumer B assign to partition 1, So, when committing offsets, in consumer A, you only have to commit the offset of partition 0. And in consumer B, you just commit the offsets of partition 1. For the Consumer#assign, you can check this link for more information: https://stackoverflow.com/a/53938397 > In this case do we still need a custom producer partitioner or is it enough to simply assign to the topic like described above? No, if you want each consumer to receive all the messages of all environments, then you don't need custom partitioner. Thank you, Luke On Wed, Dec 8, 2021 at 11:05 PM Christian Schneider <ch...@die-schneider.net> wrote: > Hi Luke, > > thanks for the hints. This helps a lot already. > > We already use assign as we manage offsets on the consumer side. Currently > we only have one partition and simply assign a stored offset on partition > 0. > For multiple partitions is it the correct behaviour to simply assign to > partition number:offset or do I have to provide offsets for the other > partitions too? I only want to listen to one partition. > You mentioned custom producer partitioner. We currently use a random > consumer group name for each consumer as we want each consumer to receive > all messages of the environment. In this case do we still need a custom > producer partitioner or is it enough to simply assign to the topic like > described above? > > Christian > > Am Mi., 8. Dez. 2021 um 11:19 Uhr schrieb Luke Chen <show...@gmail.com>: > > > Hi Christian, > > Answering your question below: > > > > > Let's assume we just have one topic with 10 partitions for simplicity. > > We can now use the environment id as a key for the messages to make sure > > the messages of each environment arrive in order while sharing the load > on > > the partitions. > > > > > Now we want each environment to only read the minimal number of > messages > > while consuming. Ideally we would like to to only consume its own > messages. > > Can we somehow filter to only > > receive messages with a certain key? Can we maybe only listen to a > certain > > partition at least? > > > > > > Unfortunately, Kafka doesn't have the feature to filter the messages on > > broker before sending to consumer. > > But for your 2nd question: > > > Can we maybe only listen to a certain partition at least? > > > > Actually, yes. Kafka has a way to just fetch data from a certain > partition > > of a topic. You can use Consumer#assign API to achieve that. So, to do > > that, I think you also need to have a custom producer partitioner for > your > > purpose. Let's say, in your example, you have 10 partitions, and 10 > > environments. Your partitioner should send to the specific partition > based > > on the environment ID, ex: env ID 1 -> partition 1, env ID 2 -> partition > > 2.... So, in your consumer, you can just assign to the partition > containing > > its environment ID. > > > > And for the idea of encrypting the messages to achieve isolation, it's > > interesting! I've never thought about it! :) > > > > Hope it helps. > > > > Thank you. > > Luke > > > > > > On Wed, Dec 8, 2021 at 4:48 PM Christian Schneider < > > ch...@die-schneider.net> > > wrote: > > > > > We have a single tenant application that we deploy to a kubernetes > > cluster > > > in many instances. > > > Every customer has several environments of the application. Each > > > application lives in a separate namespace and should be isolated from > > other > > > applications. > > > > > > We plan to use kafka to communicate inside an environment (between the > > > different pods). > > > As setting up one kafka cluster per such environment is a lot of > overhead > > > and cost we would like to just use a single multi tenant kafka cluster. > > > > > > Let's assume we just have one topic with 10 partitions for simplicity. > > > We can now use the environment id as a key for the messages to make > sure > > > the messages of each environment arrive in order while sharing the load > > on > > > the partitions. > > > > > > Now we want each environment to only read the minimal number of > messages > > > while consuming. Ideally we would like to to only consume its own > > messages. > > > Can we somehow filter to only > > > receive messages with a certain key? Can we maybe only listen to a > > certain > > > partition at least? > > > > > > Additionally we ideally would like to have enforced isolation. So each > > > environment can only see its own messages even if it might receive > > messages > > > of other environments from the same partition. > > > I think in worst case we can make this happen by encrypting the > messages > > > but it would be great if we could filter on broker side. > > > > > > Christian > > > > > > -- > > > -- > > > Christian Schneider > > > http://www.liquid-reality.de > > > > > > Computer Scientist > > > http://www.adobe.com > > > > > > > > -- > -- > Christian Schneider > http://www.liquid-reality.de > > Computer Scientist > http://www.adobe.com >
