I'm not sure about Kafka, but Apache Pulsar is currently built to support up to 1mm topics. I know that there's active work on using Oxia instead of Zookeeper for the metadata store with the goal to support hundreds of millions of topics.
That said, millions of topics in Artemis may be a bit of an anti-pattern. Depending on your use case, it may be better to use filter expressions on fewer queues [1]. Nathan [1] https://activemq.apache.org/components/artemis/documentation/latest/filter-expressions.html On Wed, Mar 5, 2025, at 12:45, William Crowell wrote: > I am wondering if Apache Kafka might be more feasible for something like this. > > Regards, > > William Crowell > > From: Justin Bertram <jbert...@apache.org> > Date: Wednesday, March 5, 2025 at 3:26 PM > To: users@activemq.apache.org <users@activemq.apache.org> > Subject: Re: Maximum Amount of Topic/Queues Within Apache Artemis > That's it. > > > Justin > > On Wed, Mar 5, 2025 at 2:19 PM William Crowell > <wcrow...@perforce.com.invalid> wrote: > >> Justin, >> >> Never, I think I found it: >> >> “…the web console's behavior is configurable. Go to the "Preferences" >> (available from the menu in the top right) and click the "Jolokia" tab. >> Here you can turn off auto-refresh (i.e. "Update rate"). You can also >> decrease the amount of data fetched by lowering the "Max depth" and "Max >> collection size." >> >> Regards, >> >> William Crowell >> >> From: William Crowell <wcrow...@perforce.com.INVALID> >> Date: Wednesday, March 5, 2025 at 3:12 PM >> To: users@activemq.apache.org <users@activemq.apache.org> >> Subject: Re: Maximum Amount of Topic/Queues Within Apache Artemis >> Justin, >> >> Appreciate your reply. Your insight is greatly appreciated and invaluable. >> >> If you can provide more details on this, then I would appreciate it. I >> can search the archives for it as well. I am guessing there is some >> command line equivalent to represent the tree. >> >> What we have is several devices that have JMS clients that communicate >> with the broker on topics and durable queues. By “several devices” I mean >> maybe 100s or 1000s. >> >> Regards, >> >> William Crowell >> >> From: Justin Bertram <jbert...@apache.org> >> Date: Wednesday, March 5, 2025 at 2:57 PM >> To: users@activemq.apache.org <users@activemq.apache.org> >> Subject: Re: Maximum Amount of Topic/Queues Within Apache Artemis >> The broker itself doesn't impose any arbitrary limits on the number of >> addresses & queues. That said, there certainly are limits, the size of your >> heap probably being the most important. Every address and queue carry with >> it some memory overhead including the objects themselves but also related >> objects like JMX MBeans which enable management. >> >> The current management console will struggle by default with a huge number >> of addresses and/or queues due to the way the "tree" view works (i.e. >> refreshing the whole view on a regular basis). This is the main reason we >> removed this view in the new console [1]. That said, the console can be >> configured in such a way as to mitigate some of these problems. I can >> provide more details on that if necessary although it's been covered on >> this list a few times already. >> >> Regarding scalability...the broker was written with scalability in mind, >> but everything breaks down at some point. I'm interested to hear about your >> experiences if you go down this route especially regarding bottlenecks that >> you may find for your specific use-case. Keep us in the loop! >> >> >> Justin >> >> [1] >> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FARTEMIS-5319&data=05%7C02%7CWCrowell%40perforce.com%7C320052ba6ccd49ef708808dd5c240792%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638768032021655752%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=fiCybGio5R84gDL5NiT3uHHTrz%2FtKGU5k3AappaW%2FMc%3D&reserved=0<https://issues.apache.org/jira/browse/ARTEMIS-5319> >> < >> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FARTEMIS-5319&data=05%7C02%7CWCrowell%40perforce.com%7C320052ba6ccd49ef708808dd5c240792%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638768032021670531%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=x99X9Q4l3k5WDqM0eAAWMXIvYPR9ReXIlsBXyz3KBKo%3D&reserved=0<https://issues.apache.org/jira/browse/ARTEMIS-5319> >> ><https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FARTEMIS-5319&data=05%7C02%7CWCrowell%40perforce.com%7C320052ba6ccd49ef708808dd5c240792%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C638768032021679546%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=dtoeTZ5yS%2BtWShgjBs6svkrqwbvLSy6Y4OPAjTTsUwQ%3D&reserved=0<https://issues.apache.org/jira/browse/ARTEMIS-5319>> >> >> On Wed, Mar 5, 2025 at 12:33 PM William Crowell >> <wcrow...@perforce.com.invalid> wrote: >> >> > Good afternoon, >> > >> > Are there any limitations or concerns with creating millions of topics >> and >> > queues within Apache Artemis that have low volumes of messages in each >> > topic and queue? I do not think there are as I believe Artemis should be >> > able to handle this use case. Important question is: Is it scalable? >> > >> > Regards and have a great day, >> > >> > William Crowell >> > >> > >> > This e-mail may contain information that is privileged or confidential. >> If >> > you are not the intended recipient, please delete the e-mail and any >> > attachments and notify us immediately. >> > >> > >> >> >> CAUTION: This email originated from outside of the organization. Do not >> click on links or open attachments unless you recognize the sender and know >> the content is safe. >> >> >> This e-mail may contain information that is privileged or confidential. If >> you are not the intended recipient, please delete the e-mail and any >> attachments and notify us immediately. >> >> >> This e-mail may contain information that is privileged or confidential. If >> you are not the intended recipient, please delete the e-mail and any >> attachments and notify us immediately. >> >> > > > CAUTION: This email originated from outside of the organization. Do not > click on links or open attachments unless you recognize the sender and > know the content is safe. > > > This e-mail may contain information that is privileged or confidential. > If you are not the intended recipient, please delete the e-mail and any > attachments and notify us immediately. --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org For additional commands, e-mail: users-h...@activemq.apache.org For further information, visit: https://activemq.apache.org/contact
