-- Matteo Merli <matteo.me...@gmail.com> On Tue, Mar 8, 2022 at 9:36 PM Michael Marshall <mmarsh...@apache.org> wrote: > > My primary concern is that our event driven messaging platform relies > on polling for distribution of system events, like newly created > topics. I would love to find a purely event driven solution.
As discussed above, I too would like a pure push notification that is 100% reliable, though that is not possible when the 2 operations (create topic and send the notification) are not atomic. I don't see any possible easy way to work around this fact. The purpose of maintaining polling (with caching of the results), is to ensure that we can eventually detect and correct any missed notifications. > I have some additional questions before I reply to the above feedback. > > How will this feature affect Pulsar's load balancing of topics? The > PIP leads me to think that all brokers will host the > TopicsListService, which means the same data will be stored across > many brokers. How will this affect clusters with many topics across > many namespaces? This proposal doesn't really change the existing behavior of how that works with respect to load balancer. Today, all brokers are responding to requests to get lists of topics. > > There is no message filtering in this proposal. Only topic list filtering. > > Broker side filtering of topic names based on arbitrary, user supplied > regex feels like message filtering, which is why I mentioned the > concept. I am pretty sure that arbitrary server side regex computation > opens up the attack surface of a broker by creating the possibility > for a ReDoS (regular expression denial of service). At the very least, it > could negatively impact tenant isolation within a broker if an expression > is computationally expensive. Am I correct in thinking that we'll need > to address this risk? Filtering of messaging is a completely different problem that involves breaking batches, decompress messages and parsing loads of metadata in order to extract the metadata on which to filter. Here we're just filtering a list of strings. I believe we can address the risk by providing a switch to enable/disable this feature. When the feature is disabled, it will be working exactly as it works today: no server side filtering or caching and no notifications. Matteo
