>> > > Based on looking through the interceptor trait, I don't see a way to >> > > trigger events based on auto created/deleted topics. For example, >> when a >> > > producer connects to a broker for a nonexistent topic (assuming auto >> > topic >> > > creation is allowed), a managed ledger, and thus a topic, is created >> > > without ever interacting with that interceptor trait. The same >> appears to >> > > be true for garbage collected topics. I think we'll need more than >> this >> > > interceptor to properly capture all cases where topics are created or >> > > deleted.
What are you hoping to accomplish by knowing when a topic is automatically created? You're trying to determine when a topic is used for the first time or when it hasn't been used after some period of time? I'm just looking for some additional context to better understand the motivation. If you can please provide some additional explanation of what you're trying to accomplish (and why you're wanting this functionality), that will help us zoom out and brainstorm the best path forward with the big picture in mind. I'm not opposed to tapping into Pulsar’s metadata; personally, I think there's a lot of power in a capability that will allow us to subscribe to changes in the cluster. I think it could open additional doors for intelligent automation, monitoring/alerting, and integration with other services. I agree with Joe that it needs to be done in a way that doesn't weaken Pulsar’s guarantees. So, that will require some brainstorming and evaluation of available options. The way my team has gotten around some of this is we only have one way to create topics/functions/etc. Basically, any time someone wants to create something in our cluster, they publish to a topic that one of our services (which we call "fast-deploy") listens to, and fast-deploy uses Pulsar’s REST API to make the changes. To simplify the implementation, we created a UI that automates producing messages to the topic ingested by fast-deploy. When we provision a topic, we're typically issuing tokens to control access, constructing a tenant and namespace for ingestion, a namespace in a shared tenant for consumption, and creating a passthrough function that sends the data between them. These tasks are all done through additional services that send messages to fast-deploy. However, we're not attempting to listen to topic auto-creation events. So, there's still a piece missing. >> > A broker publishing to a topic hosted on another broker (which for eg: >> is >> > serving "system topic"), sets up an undesirable dependency, which >> reduces >> > total system resiliency and availability for the cluster Is the concern here that a topic is owned by a single broker? It seems like each broker would need it's own local topic that would get replicated (maybe that's the wrong word here) to the other brokers. That data could be shared through bookkeeper entries, and brokers could be notified of new data by watching for changes on the metadata layer (e.g. on a ZK znode) that indicate new entries have been written to the bookies. It would add latency if everything needed to pass through the data layer, but I'm not sure how data could safely be shared between brokers directly without using a distributed consensus algorithm like Raft. That would be a huge architectural change for Pulsar, and it wouldn't be without latency either. -- Devin G. Bost On Thu, Apr 22, 2021, 10:22 PM Joe Francis <j...@verizonmedia.com.invalid> wrote: > To be clear, I would love to have this feature. But I would not use this > feature if that means whenever a broker that hosts a "system topic" has a > hiccup, it would result in an outage for N other brokers. I run 100+ > brokers/million+ topics in a cluster (hence an "audit topic" would be > wonderful for all kinds of purposes), and would not want an "system topic" > as the single point of failure. > > So you have to make this log local to the broker, or sacrifice the > reliability of the log (best case log). Local log has its advantages - you > can log a lot more about the system itself into it, (eg: security events > like failed auth etc), but you will need to provide an aggregate view for > the cluster as a whole from all the brokers > > Joe > > > > > On Thu, Apr 22, 2021 at 6:10 AM Joe Francis <j...@verizonmedia.com> wrote: > > > Completely disagree that we have accepted this risk with PIP-39. That is > > different because it is an admin flow. A failure in a namespace policy > > change does not affect data flow. > > > > What you are proposing is in the data path. Topics and subs are > > created in the data flow path. Failure means outages. PIP-39 is not going > > to help you there. > > > > Joe > > > > On Wed, Apr 21, 2021 at 11:10 PM Michael Marshall <mikemars...@gmail.com > > > > wrote: > > > >> Hi Joe, > >> > >> I agree there is a risk in adding more interdependencies between > brokers. > >> I > >> will point out that we have already accepted this risk with the > >> implementation of PIP 39, which propagates namespace policy changes to > >> other brokers using messages sent to a system topic. However, that > doesn't > >> necessarily mean we should build more interdependencies between brokers. > >> > >> Here is the link to PIP 39: > >> > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_pulsar_wiki_PIP-2D39-253A-2DNamespace-2DChange-2DEvents&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ke-uQGYh--_pwtgn13szgq-axZcRVTJXoSurefbZEk4&m=ACSaHFP9BC5MVQZZWMp0KvSJnvwUr4Jvd08xKKbQWBI&s=G_K7a-seNfGGb-Z4Wy0Q5iMrbdL2j9WCoMUWfwUH5RY&e= > >> . > >> > >> I will look into the implementation of PIP 39 to better understand its > >> design, as I think it will likely influence this feature's design. > >> > >> Thanks, > >> Michael > >> > >> On Wed, Apr 21, 2021 at 5:50 PM Joe F <joefranc...@gmail.com> wrote: > >> > >> > I would be very careful about implementing such a feature, because of > >> > introducing undesirable interdependencies. Broker processes only talk > >> to > >> > the metadata store or data store. This keeps brokers isolated from > each > >> > other - one broker is not dependent on the functioning of another > >> broker. > >> > > >> > A broker publishing to a topic hosted on another broker (which for eg: > >> is > >> > serving "system topic"), sets up an undesirable dependency, which > >> reduces > >> > total system resiliency and availability for the cluster. These are > >> better > >> > implemented as notifications off the metadata changes. > >> > > >> > Good feature, but needs careful thought to do it right > >> > Joe > >> > > >> > On Wed, Apr 21, 2021 at 4:03 PM Michael Marshall < > mikemars...@gmail.com > >> > > >> > wrote: > >> > > >> > > Thanks for your response, PengHui. > >> > > > >> > > I think this feature would be useful to end users for cluster > >> management, > >> > > which is why I want to contribute a first class feature instead of > >> > writing > >> > > my own plugin that would add little value to the community. > >> > > > >> > > > With the broker interceptor you can intercept all the REST API > >> request > >> > > and response, Pulsar commands between the broker and clients. > >> > > > >> > > Based on looking through the interceptor trait, I don't see a way to > >> > > trigger events based on auto created/deleted topics. For example, > >> when a > >> > > producer connects to a broker for a nonexistent topic (assuming auto > >> > topic > >> > > creation is allowed), a managed ledger, and thus a topic, is created > >> > > without ever interacting with that interceptor trait. The same > >> appears to > >> > > be true for garbage collected topics. I think we'll need more than > >> this > >> > > interceptor to properly capture all cases where topics are created > or > >> > > deleted. > >> > > > >> > > Regarding my reference to potential further work, it does appear > that > >> low > >> > > level auditing of connections and pulsar commands could be covered > by > >> the > >> > > interceptor. However, it would still be on the end user to implement > >> such > >> > > functionality. > >> > > > >> > > Thanks, > >> > > Michael > >> > > > >> > > > >> > > On Wed, Apr 21, 2021 at 3:51 AM PengHui Li <codelipeng...@gmail.com > > > >> > > wrote: > >> > > > >> > > > Hi Michael, > >> > > > > >> > > > Currently, Pulsar supports a pluginable Broker Interceptor, you > can > >> > find > >> > > > it here > >> > > > > >> > > > >> > > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_pulsar_blob_6704f12104219611164aa2bb5bbdfc929613f1bf_pulsar-2Dbroker_src_main_java_org_apache_pulsar_broker_intercept_BrokerInterceptor.java&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ke-uQGYh--_pwtgn13szgq-axZcRVTJXoSurefbZEk4&m=ACSaHFP9BC5MVQZZWMp0KvSJnvwUr4Jvd08xKKbQWBI&s=6Li1guS8lImjrxPo9A0nnQAmDMnYEKHlAGqlVYvB8Ug&e= > >> > > > > >> > > > With the broker interceptor you can intercept all the REST API > >> request > >> > > and > >> > > > response, Pulsar commands between the broker and clients. > >> > > > This can be used to audit the system events. > >> > > > > >> > > > Thanks, > >> > > > Penghui > >> > > > On Apr 21, 2021, 5:13 AM +0800, Michael Marshall < > >> > mikemars...@gmail.com > >> > > >, > >> > > > wrote: > >> > > > > Hello all, > >> > > > > > >> > > > > I would like to propose adding a new feature to Pulsar that will > >> > > require > >> > > > a > >> > > > > PIP. In addition to feedback on the proposed feature, I am > looking > >> > for > >> > > > > guidance on how to go about creating the PIP. Thanks for any > help > >> you > >> > > can > >> > > > > provide. > >> > > > > > >> > > > > I would like to add an optional system topic where topic > creation > >> and > >> > > > topic > >> > > > > deletion events are published. This feature will make it easier > to > >> > > > leverage > >> > > > > the auto topic creation and inactive topic deletion features by > >> > > > providing a > >> > > > > way for users to reactively discover changes to topics. The > >> largest > >> > > > benefit > >> > > > > is that users won't need to poll for these updates with an admin > >> > > client. > >> > > > > Instead, they will get them as messages. > >> > > > > > >> > > > > I looked to see if an equivalent feature already exists, but I > >> don't > >> > > see > >> > > > > one. For reference, the `PatternMultiTopicsConsumerImpl` > currently > >> > > polls > >> > > > > for all topics in a namespace and then does set operations to > >> compute > >> > > the > >> > > > > "new" topics to which it should subscribe. This client > >> implementation > >> > > > could > >> > > > > possibly leverage the new feature. > >> > > > > > >> > > > > There are still details I need to work out, like how it will > work > >> for > >> > > > > partitioned vs unpartitioned topics and what kind of guarantees > we > >> > have > >> > > > > regarding messaging semantics (I think we'll want at least once > >> > message > >> > > > > delivery here). I plan to include these details in the PIP with > >> > > > discussions > >> > > > > about trade offs for different implementations. > >> > > > > > >> > > > > Does this feature sound helpful and reasonable to others? If so, > >> is > >> > the > >> > > > > next step to formally write a proposal in a Google Doc or to put > >> > > > together a > >> > > > > doc on the Pulsar GitHub Wiki? > >> > > > > > >> > > > > Related and/or future work to consider in this design: I can see > >> > adding > >> > > > > different system topics for these types of auditable system > >> events. > >> > We > >> > > > > currently rely on log lines as our primary way for end users to > >> audit > >> > > > > system events, e.g. a producer connecting to a broker or a > >> > subscription > >> > > > > getting created, but we could instead have topics that represent > >> > > streams > >> > > > of > >> > > > > these different kinds of events. A persistent topic could make > >> these > >> > > > audit > >> > > > > events more durable and more structured which should lend > >> themselves > >> > to > >> > > > > being more easily analyzed. Further, users could choose to turn > >> > on/off > >> > > > > these audit events, perhaps at the broker or namespace level, to > >> fit > >> > > > their > >> > > > > own needs. > >> > > > > > >> > > > > Let me know what you think and how I should proceed. > >> > > > > > >> > > > > Regards, > >> > > > > Michael Marshall > >> > > > > >> > > > >> > > >> > > >
