Good afternoon all, In our Kafka clusters we have a need to know when certain activities are performed, mainly topics being created, but brokers coming up/down is also useful. This would be akin to what ActiveMQ does via advisory messages ( https://activemq.apache.org/advisory-message).
Since there did not appear to be anything in the ecosystem currently, I wrote a standalone Java program that watches the various ZooKeeper locations that the Kafka broker writes to and deltas can tell us topic/broker actions etc... and writes to a kafka topic for downstream consumption. Ideally, we would rather have the broker handle this internally rather than yet another service stood up in our systems. I began digging through the broker source (my Scala is basically hello world level) and there does not appear to be any mechanism in which this could be easily patched into the broker. Specifically, a producer or consumer acting upon an nonexistent topic or a manual CreateTopic would trigger a Produce to this advisory topic and the KafkaApis framework would handle it like any other request. However, by the time we are inside the getTopicMetadata call there doesn't seem to be a clean way to fire off another message that would make its way through KafkaApis. Perhaps another XManager type object is required? Looking for alternative ideas or guidance (or I missed something in the broker). Thank you. Knowles
