Thank you for your response Sijie! It provides helpful context. On Tue, Apr 6, 2021 at 10:31 AM Sijie Guo <guosi...@gmail.com> wrote:
> On Mon, Apr 5, 2021 at 9:57 PM Michael Marshall <mikemars...@gmail.com> > wrote: > > > Hi All, > > > > Recently in the pulsar slack channel, someone asked about the scalability > > (and limits) of the number of topics in a pulsar cluster. Addison Higham > > responded with the following: > > > > > For numbers of topics, there is a few notes: > > > - you will want to spread these topics over a sufficient number of > > namespaces. We typically recommended about 10k topics at most per > namespace > > to handle some metadata scalability in Pulsar, but once you spread across > > namespaces, it is fine. For the use case you mentioned, you might do that > > by hashing/prefix mapping to a number of namespaces > > > - with that number of topics, you typically will need more memory per > > broker and more brokers, but you can scale out quite well to add more > > topics > > > - You will want to look into adding more "bundles" in each namespace. > > > > I had not previously seen any information on recommended topic limits per > > namespace, and I found the recommended 10k limit lower than I would have > > expected given that pulsar is stated to scale to 1 million topics but > > doesn't mention a limit on number of topics per namespace in the > > documentation. > > > > Does anyone have more information about this limit? What happens when a > > namespace exceeds the limit? What is the main bottleneck leading to this > > limit? I don't believe that pulsar documentation explicitly makes > > recommendations on these limits or on the trade offs associated with > > exceeding them. > > > > The limitation is coming from zookeeper. ZooKeeper doesn't support > iterating the children under one znode. > So it has to fetch the entire children. If you put too many topics under > one namespace, it will break the list-topics action. > > > > > > Additionally, is there a limit on the number of topics per broker? > > > > The number of topics a broker can support is majorly dependent on the > memory of a broker. > > > > > > If we don't already have these answers documented in the pulsar > > documentation, which I don't think we do, I think it'd be valuable to > add a > > section to help users understand how to scale out the number of topics. > > > > Yes. We should consider adding a section for it. > > > > > > Thanks, > > Michael Marshall > > >
