On Mon, Apr 5, 2021 at 9:57 PM Michael Marshall <mikemars...@gmail.com> wrote:
> Hi All, > > Recently in the pulsar slack channel, someone asked about the scalability > (and limits) of the number of topics in a pulsar cluster. Addison Higham > responded with the following: > > > For numbers of topics, there is a few notes: > > - you will want to spread these topics over a sufficient number of > namespaces. We typically recommended about 10k topics at most per namespace > to handle some metadata scalability in Pulsar, but once you spread across > namespaces, it is fine. For the use case you mentioned, you might do that > by hashing/prefix mapping to a number of namespaces > > - with that number of topics, you typically will need more memory per > broker and more brokers, but you can scale out quite well to add more > topics > > - You will want to look into adding more "bundles" in each namespace. > > I had not previously seen any information on recommended topic limits per > namespace, and I found the recommended 10k limit lower than I would have > expected given that pulsar is stated to scale to 1 million topics but > doesn't mention a limit on number of topics per namespace in the > documentation. > > Does anyone have more information about this limit? What happens when a > namespace exceeds the limit? What is the main bottleneck leading to this > limit? I don't believe that pulsar documentation explicitly makes > recommendations on these limits or on the trade offs associated with > exceeding them. > The limitation is coming from zookeeper. ZooKeeper doesn't support iterating the children under one znode. So it has to fetch the entire children. If you put too many topics under one namespace, it will break the list-topics action. > > Additionally, is there a limit on the number of topics per broker? > The number of topics a broker can support is majorly dependent on the memory of a broker. > > If we don't already have these answers documented in the pulsar > documentation, which I don't think we do, I think it'd be valuable to add a > section to help users understand how to scale out the number of topics. > Yes. We should consider adding a section for it. > > Thanks, > Michael Marshall >
