On Wed, Mar 2, 2022 at 2:15 PM Michael Marshall <mmarsh...@apache.org> wrote:
> > A new class, org.apache.pulsar.TopicsListService will keep track > > of watchers and will listen to changes in the metadata. > > I think we should avoid creating a new service to distribute > notifications to consumers. Instead, we should consider using a > compacted topic to store and distribute topic name information. We > could have a system topic in each namespace that contains all of the > non-system topics in the namespace. This solution would not expand the > Pulsar protocol and would rely on core Pulsar features that are > already hardened. Note that the implementation for the producer to the > compacted topic of topic names would be nearly identical to the > `TopicsListService` class. The main difference would be how changes in > metadata are distributed. The problem is that this notification mechanism will not be reliable and thus it will not be possible to trust the compacted view of the topic. The operations of "create a topic" and "publish the notification in a system topic" are not atomic, therefore the notification might be missing, invalidating the state. The proposal is to augment the polling behavior with "best-effort" notifications, that will improve the discoverability in the best case but that we can correct in all cases where notifications are lost. > This solution would not expand the > Pulsar protocol and would rely on core Pulsar features that are > already hardened. We're talking of adding a core Pulsar feature here, so it doesn't look inappropriate to me for that to involve a (fully compatible) change in the protocol. Adding a system topic is actually a bigger change compared to adding a watch mechanism in the protocol. The reason is that one needs to address all the lifecycle issues of that topic, how it works with topic auto-create policies, or any other policy that is set on the namespace or at the broker level. > I concede that my solution does not support broker side message > filtering. Given that Pulsar (intentionally) does not support broker > side message filtering at this time, I think it is acceptable to skip > this optimization in favor of a more generic feature. There is no message filtering in this proposal. Only topic list filtering.
