The filtering logic there is topic filtering and not message filtering. The idea is to subscribe to multiple topics via a regex whitelist or black list. This does exist today as it does not depend on understanding the content of the message, but I don't think it is what you are looking for.
As far as message filtering goes; As Gwen said, "The way Kafka is currently implemented is that Kafka is not aware of the content of messages, so there is no Selector logic available." However, If you know upfront how you would like to filter the messages you could write your producer to use multiple topics, or even some custom partitioning. And implement a consumer that can understand and filter based on that logic. However, that would be an involved and creative implementation based on your use case. I would recommend starting simple and just dropping the messages you don't care about on the consumer side. If throughput becomes a problem, then consider alternatives. On Thu, Aug 6, 2015 at 9:47 AM, Alvaro Gareppe <agare...@gmail.com> wrote: > Is this implemented ? > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+API+changes ? > > This is message filtering on the client or server side ? > > On Tue, Aug 4, 2015 at 9:54 PM, Gwen Shapira <g...@confluent.io> wrote: > > > The way Kafka is currently implemented is that Kafka is not aware of the > > content of messages, so there is no Selector logic available. > > > > The way to go is to implement the Selector in your client - i.e. your > > consume() loop will get all messages but will throw away those that don't > > fit your pattern. > > > > > > It may be worthwhile to add a ticket for pluggable selector logic in the > > new consumer. I can't guarantee it will happen, there are infinite things > > that can be plugged into consumers and we need to draw the line > somewhere, > > but worth a discussion. > > > > On Tue, Aug 4, 2015 at 2:05 PM, Alvaro Gareppe <agare...@gmail.com> > wrote: > > > > > The is way to implement a "selector" logic in kafka (similar to JMS > > > selectors) > > > > > > So, allow to consume a message if only the message contains certain > > header > > > or content ? > > > > > > I'm evaluating to migrate from ActiveMQ to kafka and I'm using the > > selector > > > logic widely in the application > > > > > > -- > > > Ing. Alvaro Gareppe > > > agare...@gmail.com > > > > > > > > > -- > Ing. Alvaro Gareppe > agare...@gmail.com > -- Grant Henke Software Engineer | Cloudera gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
