Hello Kafka brokers are designed to be dumb i.e. they simply act as a pipe to transmit messages from producer to consumer. They don’t have capabilities to selectively filter or perform any analytics/transformation of the messages in flights. In fact, a brokers doesn’t ever read the key/value pair in a message (unless compaction is desired in which case it reads the keys).
It’s a long winded way to say, no, you cannot push filtering to the broker/server. You would either need to do it client-side manually or use some stream processing to consume “raw” messages, perform the transformation/filtering/enrichment and produce them back again into an “enriched” topic. Your consumes can then consume from the “enriched” topic. I am not super familiar but I believe one of the choices of such a stream processing library could be Kafka streams which is provided with Apache Kafka. Note that it would still perform filtering at the client-side, just in a more managed way than manually writing a consumer to do so. — Divij Vaidya On Thu 29. Aug 2024 at 20:51, vaidya nathan <vaidyaat...@gmail.com> wrote: > Hi Kafka users, > I have a producer consumer setup where I want the consumer to get only a > subset of messages based on some conditions. > > Trying to figure out whether there are any configuration on the Kafka side > that we can do to achieve this . Most of the solutions I see is to filter > them at the consumer end but I would like to push that up to the Kafka > configuration level > > How do I achieve this ? > > > Sent from my iPhone
