Hi All, I want to get your advice about one subject. I want to create a KIP for message header base filtering on Fetch API.
Our current use case We have 1k+ topics and per topic, have 10+ consumers for different use cases. However all consumers are interested in different sets of messages on the same topic. Currently We read all messages from a given topic and drop logs on the consumer side. To reduce our stream processing cost I want to drop logs on the broker side. So far my understanding *Broker send messages as is (No serilization cost) -> Network Transfer -> > Consumer Deserialize Messages(User side deserilization cost) -> User Space > drop or use messages (User Sidefiltering cost)* If I can drop messages based on their headers without serialization and deserialization messages. It will help us save network bandwidth and as well as consumer side cpu cost. My approach is building a header index. Consumer clients will define their filter in the fetch call. If the filter is matching, the broker will send the messages. I would like to hear your suggestions about my solution. Thanks
