Before proceeding with implementation details, it would be helpful to better understand the underlying use cases and requirements.
Could you provide more details in this Pulsar dev mailing list thread about: 1. The specific scenarios where topic processing speed differences create bottlenecks - what metrics or indicators suggest a topic is "slow" and blocking others? 2. An example of how you envision applications would use this dynamic management capability - perhaps some code examples showing the desired API usage? 3. The specific performance impacts you're seeing in production that motivated this proposal While MultiTopicsConsumerImpl does provide unsubscribe functionality, we should be cautious about depending on implementation classes rather than stable public APIs. Instead of directly extending the implementation approach, we might want to consider: - Defining a clear interface that abstracts the dynamic subscription management capabilities (or some other way to address the use case and the requirements) - Ensuring the solution works consistently across different consumer implementations (e.g. MultiTopicsConsumerImpl and PatternMultiTopicsConsumerImpl) - Exploring how this functionality could be provided in the Pulsar client API Would you be able to share a minimal reproducer that demonstrates the performance issues? This would help the community better understand the scope of the problem and evaluate potential solutions. Looking forward to learning more about your use case and collaborating on finding the right approach. -Lari On 2025/01/03 13:53:28 SiNan Liu wrote: > Hi, Pulsar Community. > > I added a new PR that aims to enhance our management of topic subscriptions > within PatternMultiTopicsConsumer in Pulsar client. This PR introduces the > ability to dynamically add or remove topics from a regex pattern > subscription, which can significantly improve our system’s responsiveness > and efficiency. > > When subscribing to topics using regex patterns, it’s possible to encounter > scenarios where some topics process messages slower than others. This can > lead to significant performance bottlenecks, as slower topics might block > others. There’s a need to manage such situations dynamically to ensure > efficient processing and fair resource allocation among topics. > > Related discussions can be found in [this Reddit thread]( > https://www.reddit.com/r/ApachePulsar/comments/1fssbbn/roundrobin_between_wildcard_topics/), > which highlights issues similar to what we aim to address. > > Currently, MultiTopicsConsumerImpl allows unsubscribing from specific > topics manually, as demonstrated here > <https://github.com/apache/pulsar/blob/5a3a1f169a7f90181bd5c213c8e9f479bc74f0f2/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java#L1246-L1258> > . > > > *This proposal seeks to integrate similar functionality into > PatternMultiTopicsConsumerImpl, allowing dynamically add or remove > specified topics.* > > > > Thanks, > sinan >
