If you want something really simple and hacky, you could use kafkatee[1] and kafkacat[2] together:
kafkatee.conf: input [encoding=string] pipe tail -f a.log output pipe 1 grep -v ’not this’ | kafkacat -P -b b1:9092 -t mytopic [1] https://github.com/wikimedia/analytics-kafkatee [2] https://github.com/edenhill/kafkacat On Fri, Apr 29, 2016 at 12:24 PM, Rad Gruchalski <ra...@gruchalski.com> wrote: > Apache Samza is the way to go. Never used Kafka Streams so no opinion on > that one. > > > > > > > > > > > Best regards, > Radek Gruchalski > ra...@gruchalski.com (mailto:ra...@gruchalski.com) (mailto: > ra...@gruchalski.com) > de.linkedin.com/in/radgruchalski/ ( > http://de.linkedin.com/in/radgruchalski/) > > Confidentiality: > This communication is intended for the above-named person and may be > confidential and/or legally privileged. > If it has come to you in error you must take no action based on it, nor > must you copy or show it to anyone; please delete/destroy and inform the > sender immediately. > > > > On Friday, 29 April 2016 at 13:51, Subramanian Karunanithi wrote: > > > Hi, > > > > When a stream of data passes through Kafka, wanted to apply the filter > and > > then let that message pass through to partitions. > > > > Regards, > > Subramanian. K > > On Apr 26, 2016 12:33, "Marko Bonaći" <marko.bon...@sematext.com > (mailto:marko.bon...@sematext.com)> wrote: > > > > > Instantly reminded me of Streams API, where you can use Java8 streams > > > semantics (filter being one of them) to do the first thing in > Gouzhang's > > > response (filter messages from one topic into another - I assume that's > > > what you were looking for). > > > > > > Marko Bonaći > > > Monitoring | Alerting | Anomaly Detection | Centralized Log Management > > > Solr & Elasticsearch Support > > > Sematext <http://sematext.com/> | Contact > > > <http://sematext.com/about/contact.html> > > > > > > On Tue, Apr 26, 2016 at 6:22 PM, Guozhang Wang <wangg...@gmail.com > (mailto:wangg...@gmail.com)> wrote: > > > > > > > Hi Subramanian, > > > > > > > > Could you elaborate a bit more on "filtering"? Do you want to read > raw > > > data > > > > from Kafka, and send the filtered data back to Kafka as a separate > topic, > > > > or do you want to read raw data from an external service and send the > > > > filtered data into Kafka? > > > > > > > > > > > > Guozhang > > > > > > > > > > > > On Tue, Apr 26, 2016 at 8:07 AM, Subramanian Karunanithi < > > > > subub...@gmail.com (mailto:subub...@gmail.com) > > > > > wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > Do we have any plugin which is available, which can be used as a > > > > filtering > > > > > mechanism on the data it's working on? > > > > > > > > > > Regards, > > > > > Subramanian. K > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > -- Guozhang > > > > > > > > > > > > > > > > > > >
