Hi Gokul, Thanks for the KIP.
>From what I understand, the objective of the new configuration is to protect a cluster from an overload driven by an excessive number of partitions independently from the load handled on the partitions themselves. As such, the approach uncouples the data-path load from the number of unit of distributions of throughput and intends to avoid the degradation of performance exhibited in the test results provided with the KIP by setting an upper-bound on that number. Couple of comments: 900. Multi-tenancy - one concern I would have with a cluster and broker-level configuration is that it is possible for a user to consume a large proportions of the allocatable partitions within the configured limit, leaving other users with not enough partitions to satisfy their requirements. 901. Quotas - an approach in Apache Kafka to set-up an upper-bound on resource consumptions is via client/user quotas. Could this framework be leveraged to add this limit? 902. Partition assignment - one potential problem with the new repartitioning scheme is that if a subset of brokers have reached their number of assignable partitions, yet their data path is under-loaded, new topics and/or partitions will be assigned exclusively to other brokers, which could increase the likelihood of data-path load imbalance. Fundamentally, the isolation of the constraint on the number of partitions from the data-path throughput can have conflicting requirements. 903. Rebalancing - as a corollary to 902, external tools used to balance ingress throughput may adopt an incremental approach in partition re-assignment to redistribute load, and could hit the limit on the number of partitions on a broker when a (too) conservative limit is used, thereby over-constraining the objective function and reducing the migration path. Thanks, Alexandre Le jeu. 9 avr. 2020 à 00:19, Gokul Ramanan Subramanian <gokul24...@gmail.com> a écrit : > > Hi. Requesting you to take a look at this KIP and provide feedback. > > Thanks. Regards. > > On Wed, Apr 1, 2020 at 4:28 PM Gokul Ramanan Subramanian < > gokul24...@gmail.com> wrote: > > > Hi. > > > > I have opened KIP-578, intended to provide a mechanism to limit the number > > of partitions in a Kafka cluster. Kindly provide feedback on the KIP which > > you can find at > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-578%3A+Add+configuration+to+limit+number+of+partitions > > > > I want to specially thank Stanislav Kozlovski who helped in formulating > > some aspects of the KIP. > > > > Many thanks, > > > > Gokul. > >
