lhotari commented on issue #24380: URL: https://github.com/apache/pulsar/issues/24380#issuecomment-2949114469
> > Not directly related to this issue, but related to the use of a large amount of delayed messages: [#23990 (reply in thread)](https://github.com/apache/pulsar/discussions/23990#discussioncomment-12233561) It's worth being aware that acknowledgement state could be lost when a topic is moved from one broker to another due to load balancing or broker restart if the configuration isn't tuned. > > We encountered this issue in production — and it hit us hard. As a result, we reworked the architecture for handling delayed messages and improved how the application deals with duplicates in such cases. Initially, a single topic could grow to hundreds of gigabytes. After refactoring the architecture, it’s now reduced to just a few gigabytes, thanks to proper and timely cleanup finally working. Thanks for sharing, @tarmacmonsterg. Just curious, how did you address the problem? One solution that I've heard about is having multiple topics for delayed messages where there would be a topic for some "time bucket", grouping more messages together that get delivered together. On the consuming side a topic regex pattern would be used to consume from the topics that might be dynamic (one topic for each day, etc.) and get deleted automatically after they are empty. Another one is just about separating to different topics based on the duration of the delay. That works if there's a set of fixed duration delays and there's a limited amount of fixed delays that are used. There could be a higher level client library that handles this internally. Acknowledgement state could also get lost or very large. I've recently added PRs https://github.com/apache/pulsar/pull/24392 and https://github.com/apache/pulsar/pull/24391 to address the problem that many of the related settings are missing from `broker.conf` and haven't been documented properly. There was also a setting which wasn't configurable at all. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
