Hi Michael, Thanks for having had a look at the proposal.
I share Matteo's concerns about the system topic. > One of my concerns is that this feature only solves the problem of > topic discovery for regex consumers. Topic discovery is a generic > problem in Pulsar, and any solution we implement for the regex > consumer should benefit user applications that also need to discover > topics. I agree and I believe the change I propose can be a basis for another improvement where PulsarAdmin makes use of these new messages. For this broader use case we could either - use ".*" as the regex and skip filtering on broker side when we see this specific pattern because we know everything will match or - make topics_pattern an optional field of CommandWatchTopicList. For compatibility reasons this should be done now, if this is the direction we want to go. Thanks, Andras On Thu, Mar 3, 2022 at 3:50 AM Matteo Merli <matteo.me...@gmail.com> wrote: > On Wed, Mar 2, 2022 at 2:15 PM Michael Marshall <mmarsh...@apache.org> > wrote: > > > > A new class, org.apache.pulsar.TopicsListService will keep track > > > of watchers and will listen to changes in the metadata. > > > > I think we should avoid creating a new service to distribute > > notifications to consumers. Instead, we should consider using a > > compacted topic to store and distribute topic name information. We > > could have a system topic in each namespace that contains all of the > > non-system topics in the namespace. This solution would not expand the > > Pulsar protocol and would rely on core Pulsar features that are > > already hardened. Note that the implementation for the producer to the > > compacted topic of topic names would be nearly identical to the > > `TopicsListService` class. The main difference would be how changes in > > metadata are distributed. > > The problem is that this notification mechanism will not be reliable > and thus it will not be possible to trust the compacted view of the > topic. > The operations of "create a topic" and "publish the notification in a > system topic" are not atomic, therefore the notification might be > missing, invalidating the state. > > The proposal is to augment the polling behavior with "best-effort" > notifications, that will improve the discoverability in the best case > but that we can correct in all cases where notifications are lost. > > > This solution would not expand the > > Pulsar protocol and would rely on core Pulsar features that are > > already hardened. > > We're talking of adding a core Pulsar feature here, so it doesn't look > inappropriate to me for that to involve a (fully compatible) change in > the protocol. > > Adding a system topic is actually a bigger change compared to adding a > watch mechanism in the protocol. The reason is that one needs to > address all the lifecycle issues of that topic, how it works with > topic auto-create policies, or any other policy that is set on the > namespace or at the broker level. > > > > I concede that my solution does not support broker side message > > filtering. Given that Pulsar (intentionally) does not support broker > > side message filtering at this time, I think it is acceptable to skip > > this optimization in favor of a more generic feature. > > There is no message filtering in this proposal. Only topic list filtering. >
