Hello all, I would like to propose adding a new feature to Pulsar that will require a PIP. In addition to feedback on the proposed feature, I am looking for guidance on how to go about creating the PIP. Thanks for any help you can provide.
I would like to add an optional system topic where topic creation and topic deletion events are published. This feature will make it easier to leverage the auto topic creation and inactive topic deletion features by providing a way for users to reactively discover changes to topics. The largest benefit is that users won't need to poll for these updates with an admin client. Instead, they will get them as messages. I looked to see if an equivalent feature already exists, but I don't see one. For reference, the `PatternMultiTopicsConsumerImpl` currently polls for all topics in a namespace and then does set operations to compute the "new" topics to which it should subscribe. This client implementation could possibly leverage the new feature. There are still details I need to work out, like how it will work for partitioned vs unpartitioned topics and what kind of guarantees we have regarding messaging semantics (I think we'll want at least once message delivery here). I plan to include these details in the PIP with discussions about trade offs for different implementations. Does this feature sound helpful and reasonable to others? If so, is the next step to formally write a proposal in a Google Doc or to put together a doc on the Pulsar GitHub Wiki? Related and/or future work to consider in this design: I can see adding different system topics for these types of auditable system events. We currently rely on log lines as our primary way for end users to audit system events, e.g. a producer connecting to a broker or a subscription getting created, but we could instead have topics that represent streams of these different kinds of events. A persistent topic could make these audit events more durable and more structured which should lend themselves to being more easily analyzed. Further, users could choose to turn on/off these audit events, perhaps at the broker or namespace level, to fit their own needs. Let me know what you think and how I should proceed. Regards, Michael Marshall
