server-side message filter had been talked for a very very long time, I
notice serveral years ago kafka community already discuss this feature.
kafka community still no plan implement it for some reason:
- they dont want to break the fetch zero-copy advantage
- kafka think stream project is enough
Hi, Talat,
We also need the filter feature, just like apache rocketmq filter feature (
https://rocketmq.apache.org/rocketmq/filter-messages-by-sql92-in-rocketmq/).
Talat Uyarer 于2021年11月30日周二 02:49写道：
> Hi All,
>
> I want to get your advice about one subject. I want to create a KIP for
> message
Hi Talat,
I've been following this with interest. Following on from the multiple
topics answer have you considered creating new logical topics server side?
These would overlay existing topics but have filters applied.
For instance, given a topic: topic1 you could have an admin API that
create
Hi Talat,
Have you considered using 10x more topics - perhaps using multiple clusters -
and avoid having to do any filtering in the clients?
--
Igor
On Tue, Nov 30, 2021, at 8:16 PM, Talat Uyarer wrote:
> Hi Eric,
>
> Thanks for your comments. My goal is apply filter without any
> serialization
Hi Eric,
Thanks for your comments. My goal is apply filter without any
serialization.
I will generate headers distinct values on Record Batch in producer. Broker
will build an index for header values like as timeindex. When consumer
apply filter broker will filter only record batch level. Filter
Something to keep in mind with your proposal is that you're moving the
Decompression and Filtering costs into the Brokers. It probably also adds a
new Compression cost if you want the Broker to send compressed data over
the network. Centralizing that cost on the cluster may not be desirable and
wou
Hi All,
I want to get your advice about one subject. I want to create a KIP for
message header base filtering on Fetch API.
Our current use case We have 1k+ topics and per topic, have 10+ consumers
for different use cases. However all consumers are interested in different
sets of messages on the
