Hi Divij, Thanks for the follow-up. A few comments/questions.
100. The stated motivation to increase the number of partitions above 50 is scale. Are we sure that 50 partitions is not enough to cover all valid use cases? An upper bound in the range 1 to 10 MB/s of ingress per partition gives 50 to 500 MB/s. Assuming 100 bytes per offset and metadata records, this gives between 500,000 and 5,000,000 offsets committed per second. Assuming 10,000 consumers active on the cluster, this would allow a rate of 50 to 500 offsets committed per second per consumer. Are there really use cases where there is a genuine need for more? Arguably, this does not include group metadata records which are generated at a low frequency. 101. The partitioning scheme applied for consumer offsets is also used in other parts such as the already mentioned transaction metadata or remote log metadata for the topic-based remote log metadata manager [1]. Have we considered a holistic approach for all these internal topics? Overall, I am not sure if changing the number of partitions for the consumer offsets topic should even be allowed unless there is evidence of it being required to accommodate throughput. Reassignment can be required after cluster expansion, but that is correctly supported IIRC. Thanks, Alexandre [1] https://github.com/Hangleton/kafka/blob/trunk/storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/RemoteLogMetadataTopicPartitioner.java#L37 Le jeu. 6 avr. 2023 à 16:01, hzh0425 <hzhka...@163.com> a écrit : > > I think it's a good idea as we may want to store segments in different buckets > > > > | | > hzhka...@163.com > | > | > 邮箱：hzhka...@163.com > | > > > > > ---- 回复的原邮件 ---- > | 发件人 | Divij Vaidya<divijvaidy...@gmail.com> | > | 日期 | 2023年04月04日 23:56 | > | 收件人 | dev@kafka.apache.org<dev@kafka.apache.org> | > | 抄送至 | | > | 主题 | Re: [DISCUSS] KIP-895: Dynamically refresh partition count of > __consumer_offsets | > FYI, a user faced this problem and reached out to us in the mailing list > [1]. Implementation of this KIP could have reduced the downtime for these > customers. > > Christo, would you like to create a JIRA and associate with the KIP so that > we can continue to collect cases in the JIRA where users have faced this > problem? > > [1] https://lists.apache.org/thread/zoowjshvdpkh5p0p7vqjd9fq8xvkr1nd > > -- > Divij Vaidya > > > > On Wed, Jan 18, 2023 at 9:52 AM Christo Lolov <christolo...@gmail.com> > wrote: > > > Greetings, > > > > I am bumping the below DISCUSSion thread for KIP-895. The KIP presents a > > situation where consumer groups are in an undefined state until a rolling > > restart of a cluster is performed. While I have demonstrated the behaviour > > using a cluster using Zookeeper I believe the same problem can be shown in > > a KRaft cluster. Please let me know your opinions on the problem and the > > presented solution. > > > > Best, > > Christo > > > > On Thursday, 29 December 2022 at 14:19:27 GMT, Christo > > > <christo_lo...@yahoo.com.invalid> wrote: > > > > > > > > > Hello! > > > I would like to start this discussion thread on KIP-895: Dynamically > > > refresh partition count of __consumer_offsets. > > > The KIP proposes to alter brokers so that they refresh the partition > > count > > > of __consumer_offsets used to determine group coordinators without > > > requiring a rolling restart of the cluster. > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-895%3A+Dynamically+refresh+partition+count+of+__consumer_offsets > > > > > > Let me know your thoughts on the matter! > > > Best, Christo > > > > >
