[
https://issues.apache.org/jira/browse/KAFKA-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15112349#comment-15112349
]
Stevo Slavic commented on KAFKA-1448:
-------------------------------------
Would be nice to be able as consumer to pass a lambda or something for
filtering to occur on server side.
One potential use case would be to model multiple different logical topics in
messages stored in Kafka, on top of smaller number of physical Kafka topics,
and have filtering to occur on server side. Kafka+ZooKeeper is limited in
number of physical topics or partitions per cluster, there are reports of ~1M
being a limit, so this may be one workaround.
> Filter-plugins for messages
> ---------------------------
>
> Key: KAFKA-1448
> URL: https://issues.apache.org/jira/browse/KAFKA-1448
> Project: Kafka
> Issue Type: Wish
> Components: consumer
> Reporter: Moritz Möller
> Assignee: Neha Narkhede
>
> Hi,
> we use Kafka to transmit different events that occur on different products,
> and would like to be able to subscribe only to a certain set of events for
> certain products.
> Using one topic for each event * product combination would yield around 2000
> topics, which seems to be not what kafka is designed for.
> What we would need is a way to add a consumer filter plugin to kafka (a
> simple class that can accept or reject a message) and to pass a parameter
> from the consumer to that filter class.
> Is there a better way to do this already, or if not, would you accept a patch
> upstream that adds such a mechanism?
> Thanks,
> Mo
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
