My mistake in explaining the formula N / x partitions per consumer - the ideal ratio should be 1.
On Sun, 11 Nov 2018 at 15:52, M. Manna <manme...@gmail.com> wrote: > If I understood your question correctly - you are looking for recommended > guideline for #partitions / consumer. > Kafka doesn’t hold any responsibility for that - and I’m happy to be > corrected. > > There are two partition assignment strategy - default and sticky. But you > should always aim for x/N partitions per consumer (in a given consumer > group). > > x = number of consumers > N = number of partitions in a given topic. > > This concept is irrelevant of Kafka. In any distributed ecosystem you > should maintain the above number such that > > 1) there’s no race between threads I.e. processing same task > > 2) tasks are evenly distributed amongst workers - unless a worker dies/ > exits their pool. > > So if you have not planned for an even distribution of partitions per > consumer thread, you may need to revisit how you have planned to spawn your > consumers in your cgroup. > > I hope that makes sense. > > Thanks, > > > > On Sun, 11 Nov 2018 at 15:29, Sachit Murarka <connectsac...@gmail.com> > wrote: > >> Hi, >> >> Thanks for reply. >> You have shared the command. I am looking for the formula or way to >> calculate the number of partitions for my topic. >> >> Regards, >> Sachit >> >> On Sun, 11 Nov 2018, 20:08 花儿向阳开 <2468946...@qq.com wrote: >> >> > Hi, >> > Maybe you can use the command like: >> > kafka-topic.sh --create --topic <topic_name> --partitions >> > <partition_number> --replication-factor <replica_number> --zookeeper >> > <zookeeperIP:Port> >> > I hope it helps! >> > >> > >> > Regards! >> > Cathy >> > ------------------ 原始邮件 ------------------ >> > 发件人: "Sachit Murarka"<connectsac...@gmail.com>; >> > 发送时间: 2018年11月11日(星期天) 晚上10:48 >> > 收件人: "users"<users@kafka.apache.org>; >> > >> > 主题: Deciding number of partitions >> > >> > >> > >> > Hello All, >> > >> > I am new to kafka. Could anyone please tell how to decided number of >> > partitions for our topic? >> > >> > >> > Kind Regards, >> > Sachit Murarka >> >
