One approach you can take is to set an upper bound number of partitions a priori. Imagine your key was the username. If you had 2 partitions in your topic and 4 users writing messages, then Kafka would split these messages between two partitions of the topic (assuming the usernames are unique). For your design I would ask the questions:
- how many simultaneous users will I have - how long does it take to process a message - how many messages per second will I have Answering the above questions will allow you to define the expectations and limits of your applications. So you if know, for example, that users will send 2 messages per second, and say it takes 250ms to process a message, a user could wait potentially wait for 1 ( assuming the messages are equally distributed -- which they might not be) second to receive a message. That might be fine or you might want that to go faster. If it's the latter, then build it from the beginning with more partitions in mind. There is nothing stopping you from say declaring a topic with 64 partitions from day one (or more, just don't go crazy as to-many are not good either), but use only 2 consumers from the start. In this case, each consumer is reading from 32 partitions. Now let's say more users sign on, and you have some kind of KPI you want to meet, e.g. on average users shouldn't wait more than 500ms to receive a message. In this case, you can simply spin up more consumers. In the example above, say 8 users are now online, so you spin up 2 more consumers -- now each consumer will have 16 partitions each. You can scale up or down by the number of consumers you have. There can come a point when the demand on your system will be greater than the original limits you set when doing your initial design. This is the point I would consider adding more partitions. Hope this helps. Dimitry On Tue, Jan 8, 2019 at 6:29 PM Ryanne Dolan <ryannedo...@gmail.com> wrote: > Just key your messages by user ID and they'll be processed in sequence. > Every message with the same key will go to the same partition, regardless > of how many partitions there are. > > Ryanne > > On Tue, Jan 8, 2019, 8:57 AM marimuthu eee <marimuthuee...@gmail.com > wrote: > > > Hi Ryanne Dolan, > > > > My requirement is to process notification messages in parallel.So i have > > an idea to pick kafka as a messaging system.But i need dynamic > partitioning* > > to handle individual user notification messages in serial manner*.Is it > > possible to create dynamic partition creation for a particular topic. > > > > > > > > On Tue, Jan 8, 2019 at 8:20 PM Ryanne Dolan <ryannedo...@gmail.com> > wrote: > > > >> Can you explain why you think you need dynamic partitioning? Maybe we > can > >> help you design without it. > >> > >> Ryanne > >> > >> On Tue, Jan 8, 2019, 4:04 AM marimuthu eee <marimuthuee...@gmail.com > >> wrote: > >> > >> > Hi, > >> > > >> > My requirement is to process notification messages in parallel.So i > >> have an > >> > idea to pick kafka as a messaging system.But i need dynamic > >> partitioning to > >> > handle individual user notification messages.Is it possible to create > >> > dynamic partition creation for a particular topic. > >> > > >> > > >
