Jun's advice is exactly what the FAQ says but maybe it should be rephrased because it is true this question comes up once in a while.
Your requirement is: user must receive published messages in order. All messages published to a partition will be stored in order. A consumer reading a partition will also read them in the order they arrived. The partition key is the way for the producer to pick the partition in which the messages will be stored. If you use user id as the partition key, it means all messages for that user will end up in the same partition. Using user id as the key for partitioning does not mean you need a partition per user. A reason for partitioning is to be able to scale out your consumers. If you expect a lot of messages, you'll need to scale out by running multiple consumers in parallel. This requires setting the number of partitions greater or equal to the number of consumer, each partition being read by only one consumer (in a consumer group). Selecting the number of partitions is therefore a performance & scalability choice more than anything else. It may be necessary to split your notifications into different topic if, for example, some notification to are less important than others or are targeted at different consumers. But a generic notification topic can very well. cheers, marc On Fri, Dec 6, 2013 at 12:32 AM, Benjamin Black <[email protected]> wrote: > Deja vu! > > IMO, what you are describing is a database problem, even though you are > talking/thinking about it as a queue problem. I'm sure you could construct > something using Kafka (and Samza), but I think you'd have an easier time > with a database. The number of pending messages per user and the average > message size would be critical in selecting exactly which sort of database > to use. > > My $0.02. > > > > On Thu, Dec 5, 2013 at 7:47 PM, mission mission <[email protected] > >wrote: > > > Hello, > > > > According to the Kafka FAQ "How do I choose the number of partitions for > a > > topic", clusters with more than 10K partitions are not tested. I am > looking > > for advice on how to scale the number of partitions beyond that. My use > > case is to publish messages to 1 million users, each with an unique user > > id. Users are not always connected but a user must receive published > > messages in order. > > > > What is the best way to divide topics and partitions for this use case? > Do > > I need 1 million partitions? The FAQ seems to think so, i.e. "if we were > > storing notifications for users we would encourage a design with a single > > notifications topic partitioned by user id". But the FAQ implies strongly > > that 1 million partitions may wreak havoc on zookeeper because they will > > lead to X million znodes that have to be stored in memory. Any > suggestions? > > > > Thanks, > > > > mission > > >
