This is a great topic, thanks for starting the discussion Mattison. The PIP only mentions topic events right now--I think we should include (or plan to include) metadata related events, too.
I proposed a similar feature in April 2021. There was good discussion then that is relevant now [0]. I think a system or audit topic (more likely a collection of topics) would be extremely valuable. However, last year's discussion raised several important concerns that PIP 132 does not yet address. First, we need to ensure this feature is not a single point of failure for the rest of the cluster. That requires defining how we will handle failures when publishing audit messages. Joe Francis had several good notes on this topic [0]. Also, the feature must be scalable: it should work with 100+ brokers and 1,000,000+ topics. > Also if this pip is for user use cases then there needs to be isolation at > the tenant level. Great point. Further, if these events are also meant for system admins (super users), we'll need to prevent tenant admins from hiding events or creating fake events (by deleting system topics/subscriptions or publishing to system event topics). That will require careful attention to authorization. The question of tenant isolation presupposes event classification. Namespace and topic level events obviously belong to a tenant, so the event topic for those events could live in a tenant namespace. Tenant level events like tenant creation/deletion or changes in broker state would likely require a "system event tenant" to host this tenant agnostic information. Alternatively, we could use a system event tenant for all system events. The namespaces could map to tenant names with an extra namespace for tenant agnostic events. We might need to use topic level policies to achieve proper tenant isolation. Personally, I think a dedicated tenant might be the best solution, but I need to think about it more. If I understand it correctly, PIP 45 opens the path for us to move towards a zookeeper-less metadata solution. If that solution relies on Pulsar topics for metadata, would we get a kind of system event topic for free? We should make sure we're not duplicating efforts. This is a great feature, and I'd like to make sure it is robust and well defined before we start its implementation. Thanks, Michael [0] https://lists.apache.org/thread/h4cbvwjdomktsq2jo66x5qpvhdrqk871 On Thu, Jan 6, 2022 at 3:42 AM Haiting Jiang <jianghait...@apache.org> wrote: > > Hi mattison, > > > For this part, I think we need to discuss. Do we create a specific event > > class or use properties in json format? > > I think String value would be enough for most cases. > IMHO, these system event is mostly for administrators, so the data should be > readable. > > > > Maybe the better way is to record some system internal event like node > > changed, ledger deleted etc. > > This PIP can start with a small scope of events that requires the attention > of administrators, like ledger delete failure, zk session timeout, etc. > > Thanks, > Haiting Jiang > > > On 2022/01/06 05:23:52 mattison chao wrote: > > Hi, Haiting Jiang > > > > > > >> CompletableFuture<MessageId> trigger(SystemEventMessage message); > > > > > > Maybe `record` make more sense? > > > > +1 > > > > >> SystemEventMessageBuilder properties(Map<String,Object> properties); > > >> > > >> SystemEventMessageBuilder addProperty(String key, Object value); > > > > > > How to handle the serialization and deserialization if the value type is > > > Object? > > > > For this part, I think we need to discuss. Do we create a specific event > > class or use properties in json format? > > > > > > > I think we don't need to record normal message producing and consuming > > > event? > > > That would make the system topic store too much data. > > > > > > Agree with you. > > > > Maybe the better way is to record some system internal event like node > > changed, ledger deleted etc. > > > > > > > > > > > > > > > On Jan 6, 2022, at 12:10 PM, Haiting Jiang <jianghait...@apache.org > > > <mailto:jianghait...@apache.org>> wrote: > > > > > > > > >> CompletableFuture<MessageId> trigger(SystemEventMessage message); > > > > > > Maybe `record` make more sense? > > > > > >> SystemEventMessageBuilder properties(Map<String,Object> properties); > > >> > > >> SystemEventMessageBuilder addProperty(String key, Object value); > > > > > > How to handle the serialization and deserialization if the value type is > > > Object? > > > > > >> - [ ] Producer publish > > >> - [ ] Message delivered > > >> - [ ] Message acked > > > > > > I think we don't need to record normal message producing and consuming > > > event? > > > That would make the system topic store too much data. > > > > > > > > > On 2022/01/05 15:27:33 mattison chao wrote: > > >> Original PIP : https://github.com/apache/pulsar/issues/13628 > > >> <https://github.com/apache/pulsar/issues/13628> > > >> > > >> Pasted bellow for quoting convenience. > > >> > > >> —— > > >> > > >> ## Motivation > > >> > > >> In some case or for better observability. We can support Pulsar event > > >> notification mechanism, that can let user subscribe this system topic to > > >> receive some **"we need notice to user"** event. > > >> > > >> ## Goal > > >> > > >> Enable users to receive Pulsar system event. > > >> > > >> ## API Changes > > >> > > >> 1. Add config ``pulsarSystemEventEnabled`` > > >> > > >> ```java > > >> pulsarSystemEventEnabled=false > > >> ``` > > >> 2. Add ``SystemEventMessageBuilder`` > > >> ```java > > >> > > >> > > >> public interface SystemEventMessageBuilder { > > >> > > >> SystemEventMessageBuilder eventType(SystemEvent event); > > >> > > >> SystemEventMessageBuilder properties(Map<String,Object> properties); > > >> > > >> SystemEventMessageBuilder addProperty(String key, Object value); > > >> > > >> SystemEventMessage build(); > > >> > > >> } > > >> > > >> ``` > > >> 3. Add ``SystemEventService`` > > >> > > >> ```java > > >> > > >> public interface SystemEventService { > > >> > > >> void init(); > > >> > > >> CompletableFuture<MessageId> trigger(SystemEventMessage message); > > >> > > >> void close(); > > >> } > > >> > > >> ``` > > >> > > >> ## Implementation > > >> > > >> This PIP just is a simple version, it's purpose is to discuss what event > > >> we need to trigger? how to trigger? and what is the event message > > >> structure. > > >> > > >> I think we need to trigger some event as bellow: > > >> > > >> - [ ] Client connected > > >> - [ ] Client disconnected > > >> - [ ] Consumer subscribe > > >> - [ ] Consumer unsubscribe > > >> - [ ] Producer publish > > >> - [ ] Message delivered > > >> - [ ] Message acked > > >> > > >> These are far from enough, and we need to continue to make supplements > > >> here. > > >> Please fell free to give me more advice for this PIP. > > >> > > >> ## Reject Alternatives > > >> > > >> none. > > >> > > >> Thanks, > > >> Mattisonchao > > > > > > Thanks, > > > Haiting Jiang > > > >
