+1 Penghui
On Thu, Apr 21, 2022 at 9:17 PM Andras Beni <andras.b...@streamnative.io.invalid> wrote: > Hi everyone, > > I've just created a proposal that will help scaling up the number of topics > per namespace. > It's available at https://github.com/apache/pulsar/issues/15254 and is > copied below. > Let me know what you think. > > Thanks, > Andras > > Motivation > > Pulsar is able to manage millions of topics but the number of topics within > a single namespace is limited by metadata storage. > > For each topic within a namespace there is a ZooKeeper node. Listing topics > thus requires listing children of a node, which at around 10K topics hits > the limits of ZK. > Goal > > This feature will allow a larger number of topics within a namespace by > inserting an intermediate layer (buckets) before the topic nodes like > /managed-ledgers/tenant/namespace/domain/bucket/topic. > > By default this feature will be switched off and would only be enabled on a > per namespace basis at the creation of namespaces by setting a policy. This > eliminates the need for migrating existing installations to this new > scheme. > > Buckets will not have correlation with bundles. > API Changes > > A new policy numberOfTopicBuckets will be added. The default value, 1 means > no bucketing, the current behaviour will be preserved for the namespace. > Greater values mean topics will be stored at a path including buckets. > Users will not be able to change the number of buckets after the namespace > is created. > Implementation > > The goal is to implement this feature transparently to the user. Clients > will continue to refer to topics by domain://tenant/namespace/topic but > pulsar will internally translate to the new persistence naming where > necessary. > > The way metadata stores work will not be affected either. > > Assigning topics to buckets will be based on the topic name's hash code's > absolute value modulo the number of buckets. > > The bulk of the changes necessary for this feature is to make namespace > policies available wherever persistence naming is calculated. Where listing > of topics within a namespace is necessary, the introduction of the new > layer will add some overhead in the form of multiple requests to the > metadata store. These include checking if the limit on topic number per > namespace has been reached. > Example > > Let's consider the following metadata hierarchy: > > managed-ledgers > \- tenant > \- namespace > \- persistent > +- nptopic1 > +- nptopic2 > +- ptopic-partition-0 > +- ptopic-partition-1 > +- ptopic-partition-2 > \- ptopic-partition-3 > > In case of 3 buckets the same topic metadata would be laid out the > following way: > > managed-ledgers > \- tenant > \- namespace > \- persistent > +- $0 > | +- ptopic-partition-0 > | \- ptopic-partition-3 > +- $1 > | +- nptopic2 > | \- ptopic-partition-1 > \- $2 > +- nptopic1 > \- ptopic-partition-2 > > Compatibility > > Existing namespaces and namespaces created without explicitly activating > this feature will not be affected. > > Namespaces created with this feature activated can be used just as others. > Rejected alternatives > > An alternative approach would be to introduce bucketing globally for all > namespaces. This would make metadata structure more homogeneous but would > require complex update logic to atomically move topics from their current > path to the new place once all brokers are upgraded. > For similar reasons changing the number of buckets is not a goal of this > proposal. > > Since the proposal intends to solve a problem related to ZK, it could be > handled within the ZK based metadata store implementation. This would have > to introduce knowledge of what paths mean thus breaking separation of > concerns. >
