Hi All, Recently in the pulsar slack channel, someone asked about the scalability (and limits) of the number of topics in a pulsar cluster. Addison Higham responded with the following:
> For numbers of topics, there is a few notes: > - you will want to spread these topics over a sufficient number of namespaces. We typically recommended about 10k topics at most per namespace to handle some metadata scalability in Pulsar, but once you spread across namespaces, it is fine. For the use case you mentioned, you might do that by hashing/prefix mapping to a number of namespaces > - with that number of topics, you typically will need more memory per broker and more brokers, but you can scale out quite well to add more topics > - You will want to look into adding more "bundles" in each namespace. I had not previously seen any information on recommended topic limits per namespace, and I found the recommended 10k limit lower than I would have expected given that pulsar is stated to scale to 1 million topics but doesn't mention a limit on number of topics per namespace in the documentation. Does anyone have more information about this limit? What happens when a namespace exceeds the limit? What is the main bottleneck leading to this limit? I don't believe that pulsar documentation explicitly makes recommendations on these limits or on the trade offs associated with exceeding them. Additionally, is there a limit on the number of topics per broker? If we don't already have these answers documented in the pulsar documentation, which I don't think we do, I think it'd be valuable to add a section to help users understand how to scale out the number of topics. Thanks, Michael Marshall
