Hi everyone,
I've just created a proposal that will help scaling up the number of topics
per namespace.
It's available at https://github.com/apache/pulsar/issues/15254 and is
copied below.
Let me know what you think.
Thanks,
Andras
Motivation
Pulsar is able to manage millions of topics but the number of topics within
a single namespace is limited by metadata storage.
For each topic within a namespace there is a ZooKeeper node. Listing topics
thus requires listing children of a node, which at around 10K topics hits
the limits of ZK.
Goal
This feature will allow a larger number of topics within a namespace by
inserting an intermediate layer (buckets) before the topic nodes like
/managed-ledgers/tenant/namespace/domain/bucket/topic.
By default this feature will be switched off and would only be enabled on a
per namespace basis at the creation of namespaces by setting a policy. This
eliminates the need for migrating existing installations to this new scheme.
Buckets will not have correlation with bundles.
API Changes
A new policy numberOfTopicBuckets will be added. The default value, 1 means
no bucketing, the current behaviour will be preserved for the namespace.
Greater values mean topics will be stored at a path including buckets.
Users will not be able to change the number of buckets after the namespace
is created.
Implementation
The goal is to implement this feature transparently to the user. Clients
will continue to refer to topics by domain://tenant/namespace/topic but
pulsar will internally translate to the new persistence naming where
necessary.
The way metadata stores work will not be affected either.
Assigning topics to buckets will be based on the topic name's hash code's
absolute value modulo the number of buckets.
The bulk of the changes necessary for this feature is to make namespace
policies available wherever persistence naming is calculated. Where listing
of topics within a namespace is necessary, the introduction of the new
layer will add some overhead in the form of multiple requests to the
metadata store. These include checking if the limit on topic number per
namespace has been reached.
Example
Let's consider the following metadata hierarchy:
managed-ledgers
\- tenant
\- namespace
\- persistent
+- nptopic1
+- nptopic2
+- ptopic-partition-0
+- ptopic-partition-1
+- ptopic-partition-2
\- ptopic-partition-3
In case of 3 buckets the same topic metadata would be laid out the
following way:
managed-ledgers
\- tenant
\- namespace
\- persistent
+- $0
| +- ptopic-partition-0
| \- ptopic-partition-3
+- $1
| +- nptopic2
| \- ptopic-partition-1
\- $2
+- nptopic1
\- ptopic-partition-2
Compatibility
Existing namespaces and namespaces created without explicitly activating
this feature will not be affected.
Namespaces created with this feature activated can be used just as others.
Rejected alternatives
An alternative approach would be to introduce bucketing globally for all
namespaces. This would make metadata structure more homogeneous but would
require complex update logic to atomically move topics from their current
path to the new place once all brokers are upgraded.
For similar reasons changing the number of buckets is not a goal of this
proposal.
Since the proposal intends to solve a problem related to ZK, it could be
handled within the ZK based metadata store implementation. This would have
to introduce knowledge of what paths mean thus breaking separation of
concerns.
