Having 1 million partitions is probably too many. You can still partition by user id with fewer partitions. Events for the same user are still kept in order. It's just that each partition will have multiple users' data.
Thanks, Jun On Thu, Dec 5, 2013 at 7:47 PM, mission mission <[email protected]>wrote: > Hello, > > According to the Kafka FAQ "How do I choose the number of partitions for a > topic", clusters with more than 10K partitions are not tested. I am looking > for advice on how to scale the number of partitions beyond that. My use > case is to publish messages to 1 million users, each with an unique user > id. Users are not always connected but a user must receive published > messages in order. > > What is the best way to divide topics and partitions for this use case? Do > I need 1 million partitions? The FAQ seems to think so, i.e. "if we were > storing notifications for users we would encourage a design with a single > notifications topic partitioned by user id". But the FAQ implies strongly > that 1 million partitions may wreak havoc on zookeeper because they will > lead to X million znodes that have to be stored in memory. Any suggestions? > > Thanks, > > mission >
