I'd like to add a notification API to Kafka so that I can receive the following notifications for each broker in the cluster:
1. An event notification when a broker is promoted from follower to leader, or when a follower catches up with the leader or falls behind the leader. 2. An event notification each time a new event is written to the leader partition. 3. An event notification each time an event is received by a partition follower. My motivation for having this API is two-fold: 1. I want to be able to do 'stream processing' without having to run a separate cluster for my app logic. 2. I want to keep the processing load next to the data, avoiding all unnecessary network communication. This makes sense for me because I want to use Kafka as an event-sourcing platform, rather than as a general streaming backbone between legacy applications. Could somebody with knowledge of the code base please provide me some pointers as to how I should approach adding an API like this, or provide further information if this is considered to be a bad idea. Thanks in advance, Dominic Chambers.
