Hi, Talat, We also need the filter feature, just like apache rocketmq filter feature ( https://rocketmq.apache.org/rocketmq/filter-messages-by-sql92-in-rocketmq/).
Talat Uyarer <tuya...@paloaltonetworks.com> 于2021年11月30日周二 02:49写道： > Hi All, > > I want to get your advice about one subject. I want to create a KIP for > message header base filtering on Fetch API. > > Our current use case We have 1k+ topics and per topic, have 10+ consumers > for different use cases. However all consumers are interested in different > sets of messages on the same topic. Currently We read all messages from a > given topic and drop logs on the consumer side. To reduce our stream > processing cost I want to drop logs on the broker side. So far my > understanding > > *Broker send messages as is (No serilization cost) -> Network Transfer -> > > Consumer Deserialize Messages(User side deserilization cost) -> User > Space > > drop or use messages (User Sidefiltering cost)* > > > If I can drop messages based on their headers without serialization and > deserialization messages. It will help us save network bandwidth and as > well as consumer side cpu cost. > > My approach is building a header index. Consumer clients will define > their filter in the fetch call. If the filter is matching, the broker will > send the messages. I would like to hear your suggestions about my solution. > > Thanks >
