Hi, I am on the ActiveMQ PMC and I think this is a very good idea to have a way to do advisories/notifications/events (whatever you want to call it). In ActiveMQ classic you have advisories and in Artemis you have notifications. Having management messages that can be subscribed to in real time is actually a major feature that is missing from Kafka that many other brokers have.
The idea here would be to publish notifications of different configurable events when something important happens so a consumer can listen in on things it cares about and be able to do something instead of having to poll the admin API. There are many events that happen in a broker that would be useful to be notified about. Events such as new connections to the cluster, new topics created or destroyed, consumer group creation, authorization errors, new leader election, etc. The list is pretty much endless. The metadata topic that will exist is probably not going to have all of this information so some other mechanism would be needed to handle publishing these messages to a specific management topic that would be useful for a consumer. Chris On Wed, Jan 20, 2021 at 4:12 PM Boyang Chen <reluctanthero...@gmail.com> wrote: > Hey Knowles, > > in Kafka people normally use admin clients to get those metadata. I'm not > sure why you mentioned specifically that having a topic to manage these > information is useful, but a good news is that in KIP-500 > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum > > > we > are trying to deprecate Zookeeper and migrate to a self-managed metadata > topic quorum. At the time this feature is fully done, you should be able to > use consumers to pull the metadata log. > > Best, > Boyang > > On Wed, Jan 20, 2021 at 11:22 AM Knowles Atchison Jr < > katchiso...@gmail.com> > wrote: > > > Good afternoon all, > > > > In our Kafka clusters we have a need to know when certain activities are > > performed, mainly topics being created, but brokers coming up/down is > also > > useful. This would be akin to what ActiveMQ does via advisory messages ( > > https://activemq.apache.org/advisory-message). > > > > Since there did not appear to be anything in the ecosystem currently, I > > wrote a standalone Java program that watches the various ZooKeeper > > locations that the Kafka broker writes to and deltas can tell us > > topic/broker actions etc... and writes to a kafka topic for downstream > > consumption. > > > > Ideally, we would rather have the broker handle this internally rather > > than yet another service stood up in our systems. I began digging through > > the broker source (my Scala is basically hello world level) and there > does > > not appear to be any mechanism in which this could be easily patched into > > the broker. > > > > Specifically, a producer or consumer acting upon an nonexistent topic or > a > > manual CreateTopic would trigger a Produce to this advisory topic and the > > KafkaApis framework would handle it like any other request. However, by > the > > time we are inside the getTopicMetadata call there doesn't seem to be a > > clean way to fire off another message that would make its way through > > KafkaApis. Perhaps another XManager type object is required? > > > > Looking for alternative ideas or guidance (or I missed something in the > > broker). > > > > Thank you. > > > > Knowles > > >
