How do we deal with Schema storage, Cursors and other usages of ManagedLedger?
Can you please clarify each case I'm which we use ManagedLedger? Did you try to put up a proff or concept with this proposal? I generally agree with this approach but it looks like we are not going deep enough in the design Enrico Il Lun 25 Apr 2022, 04:53 Hang Chen <chenh...@apache.org> ha scritto: > +1 > > Thanks, > Hang > > PengHui Li <peng...@apache.org> 于2022年4月25日周一 09:20写道： > > > > +1 > > > > Penghui > > > > On Thu, Apr 21, 2022 at 9:17 PM Andras Beni > > <andras.b...@streamnative.io.invalid> wrote: > > > > > Hi everyone, > > > > > > I've just created a proposal that will help scaling up the number of > topics > > > per namespace. > > > It's available at https://github.com/apache/pulsar/issues/15254 and is > > > copied below. > > > Let me know what you think. > > > > > > Thanks, > > > Andras > > > > > > Motivation > > > > > > Pulsar is able to manage millions of topics but the number of topics > within > > > a single namespace is limited by metadata storage. > > > > > > For each topic within a namespace there is a ZooKeeper node. Listing > topics > > > thus requires listing children of a node, which at around 10K topics > hits > > > the limits of ZK. > > > Goal > > > > > > This feature will allow a larger number of topics within a namespace by > > > inserting an intermediate layer (buckets) before the topic nodes like > > > /managed-ledgers/tenant/namespace/domain/bucket/topic. > > > > > > By default this feature will be switched off and would only be enabled > on a > > > per namespace basis at the creation of namespaces by setting a policy. > This > > > eliminates the need for migrating existing installations to this new > > > scheme. > > > > > > Buckets will not have correlation with bundles. > > > API Changes > > > > > > A new policy numberOfTopicBuckets will be added. The default value, 1 > means > > > no bucketing, the current behaviour will be preserved for the > namespace. > > > Greater values mean topics will be stored at a path including buckets. > > > Users will not be able to change the number of buckets after the > namespace > > > is created. > > > Implementation > > > > > > The goal is to implement this feature transparently to the user. > Clients > > > will continue to refer to topics by domain://tenant/namespace/topic but > > > pulsar will internally translate to the new persistence naming where > > > necessary. > > > > > > The way metadata stores work will not be affected either. > > > > > > Assigning topics to buckets will be based on the topic name's hash > code's > > > absolute value modulo the number of buckets. > > > > > > The bulk of the changes necessary for this feature is to make namespace > > > policies available wherever persistence naming is calculated. Where > listing > > > of topics within a namespace is necessary, the introduction of the new > > > layer will add some overhead in the form of multiple requests to the > > > metadata store. These include checking if the limit on topic number per > > > namespace has been reached. > > > Example > > > > > > Let's consider the following metadata hierarchy: > > > > > > managed-ledgers > > > \- tenant > > > \- namespace > > > \- persistent > > > +- nptopic1 > > > +- nptopic2 > > > +- ptopic-partition-0 > > > +- ptopic-partition-1 > > > +- ptopic-partition-2 > > > \- ptopic-partition-3 > > > > > > In case of 3 buckets the same topic metadata would be laid out the > > > following way: > > > > > > managed-ledgers > > > \- tenant > > > \- namespace > > > \- persistent > > > +- $0 > > > | +- ptopic-partition-0 > > > | \- ptopic-partition-3 > > > +- $1 > > > | +- nptopic2 > > > | \- ptopic-partition-1 > > > \- $2 > > > +- nptopic1 > > > \- ptopic-partition-2 > > > > > > Compatibility > > > > > > Existing namespaces and namespaces created without explicitly > activating > > > this feature will not be affected. > > > > > > Namespaces created with this feature activated can be used just as > others. > > > Rejected alternatives > > > > > > An alternative approach would be to introduce bucketing globally for > all > > > namespaces. This would make metadata structure more homogeneous but > would > > > require complex update logic to atomically move topics from their > current > > > path to the new place once all brokers are upgraded. > > > For similar reasons changing the number of buckets is not a goal of > this > > > proposal. > > > > > > Since the proposal intends to solve a problem related to ZK, it could > be > > > handled within the ZK based metadata store implementation. This would > have > > > to introduce knowledge of what paths mean thus breaking separation of > > > concerns. > > > >
