Hi Joe, I agree there is a risk in adding more interdependencies between brokers. I will point out that we have already accepted this risk with the implementation of PIP 39, which propagates namespace policy changes to other brokers using messages sent to a system topic. However, that doesn't necessarily mean we should build more interdependencies between brokers.
Here is the link to PIP 39: https://github.com/apache/pulsar/wiki/PIP-39%3A-Namespace-Change-Events. I will look into the implementation of PIP 39 to better understand its design, as I think it will likely influence this feature's design. Thanks, Michael On Wed, Apr 21, 2021 at 5:50 PM Joe F <joefranc...@gmail.com> wrote: > I would be very careful about implementing such a feature, because of > introducing undesirable interdependencies. Broker processes only talk to > the metadata store or data store. This keeps brokers isolated from each > other - one broker is not dependent on the functioning of another broker. > > A broker publishing to a topic hosted on another broker (which for eg: is > serving "system topic"), sets up an undesirable dependency, which reduces > total system resiliency and availability for the cluster. These are better > implemented as notifications off the metadata changes. > > Good feature, but needs careful thought to do it right > Joe > > On Wed, Apr 21, 2021 at 4:03 PM Michael Marshall <mikemars...@gmail.com> > wrote: > > > Thanks for your response, PengHui. > > > > I think this feature would be useful to end users for cluster management, > > which is why I want to contribute a first class feature instead of > writing > > my own plugin that would add little value to the community. > > > > > With the broker interceptor you can intercept all the REST API request > > and response, Pulsar commands between the broker and clients. > > > > Based on looking through the interceptor trait, I don't see a way to > > trigger events based on auto created/deleted topics. For example, when a > > producer connects to a broker for a nonexistent topic (assuming auto > topic > > creation is allowed), a managed ledger, and thus a topic, is created > > without ever interacting with that interceptor trait. The same appears to > > be true for garbage collected topics. I think we'll need more than this > > interceptor to properly capture all cases where topics are created or > > deleted. > > > > Regarding my reference to potential further work, it does appear that low > > level auditing of connections and pulsar commands could be covered by the > > interceptor. However, it would still be on the end user to implement such > > functionality. > > > > Thanks, > > Michael > > > > > > On Wed, Apr 21, 2021 at 3:51 AM PengHui Li <codelipeng...@gmail.com> > > wrote: > > > > > Hi Michael, > > > > > > Currently, Pulsar supports a pluginable Broker Interceptor, you can > find > > > it here > > > > > > https://github.com/apache/pulsar/blob/6704f12104219611164aa2bb5bbdfc929613f1bf/pulsar-broker/src/main/java/org/apache/pulsar/broker/intercept/BrokerInterceptor.java > > > > > > With the broker interceptor you can intercept all the REST API request > > and > > > response, Pulsar commands between the broker and clients. > > > This can be used to audit the system events. > > > > > > Thanks, > > > Penghui > > > On Apr 21, 2021, 5:13 AM +0800, Michael Marshall < > mikemars...@gmail.com > > >, > > > wrote: > > > > Hello all, > > > > > > > > I would like to propose adding a new feature to Pulsar that will > > require > > > a > > > > PIP. In addition to feedback on the proposed feature, I am looking > for > > > > guidance on how to go about creating the PIP. Thanks for any help you > > can > > > > provide. > > > > > > > > I would like to add an optional system topic where topic creation and > > > topic > > > > deletion events are published. This feature will make it easier to > > > leverage > > > > the auto topic creation and inactive topic deletion features by > > > providing a > > > > way for users to reactively discover changes to topics. The largest > > > benefit > > > > is that users won't need to poll for these updates with an admin > > client. > > > > Instead, they will get them as messages. > > > > > > > > I looked to see if an equivalent feature already exists, but I don't > > see > > > > one. For reference, the `PatternMultiTopicsConsumerImpl` currently > > polls > > > > for all topics in a namespace and then does set operations to compute > > the > > > > "new" topics to which it should subscribe. This client implementation > > > could > > > > possibly leverage the new feature. > > > > > > > > There are still details I need to work out, like how it will work for > > > > partitioned vs unpartitioned topics and what kind of guarantees we > have > > > > regarding messaging semantics (I think we'll want at least once > message > > > > delivery here). I plan to include these details in the PIP with > > > discussions > > > > about trade offs for different implementations. > > > > > > > > Does this feature sound helpful and reasonable to others? If so, is > the > > > > next step to formally write a proposal in a Google Doc or to put > > > together a > > > > doc on the Pulsar GitHub Wiki? > > > > > > > > Related and/or future work to consider in this design: I can see > adding > > > > different system topics for these types of auditable system events. > We > > > > currently rely on log lines as our primary way for end users to audit > > > > system events, e.g. a producer connecting to a broker or a > subscription > > > > getting created, but we could instead have topics that represent > > streams > > > of > > > > these different kinds of events. A persistent topic could make these > > > audit > > > > events more durable and more structured which should lend themselves > to > > > > being more easily analyzed. Further, users could choose to turn > on/off > > > > these audit events, perhaps at the broker or namespace level, to fit > > > their > > > > own needs. > > > > > > > > Let me know what you think and how I should proceed. > > > > > > > > Regards, > > > > Michael Marshall > > > > > >
