Thanks everyone, shall try these options. Regards, Subramanian. K
On Mon, May 2, 2016 at 9:43 AM, Andrew Otto <o...@wikimedia.org> wrote: > If you want something really simple and hacky, you could use kafkatee[1] > and kafkacat[2] together: > > kafkatee.conf: > > input [encoding=string] pipe tail -f a.log > output pipe 1 grep -v ’not this’ | kafkacat -P -b b1:9092 -t mytopic > > [1] https://github.com/wikimedia/analytics-kafkatee > [2] https://github.com/edenhill/kafkacat > > On Fri, Apr 29, 2016 at 12:24 PM, Rad Gruchalski <ra...@gruchalski.com> > wrote: > > > Apache Samza is the way to go. Never used Kafka Streams so no opinion on > > that one. > > > > > > > > > > > > > > > > > > > > > > Best regards, > > Radek Gruchalski > > ra...@gruchalski.com (mailto:ra...@gruchalski.com) (mailto: > > ra...@gruchalski.com) > > de.linkedin.com/in/radgruchalski/ ( > > http://de.linkedin.com/in/radgruchalski/) > > > > Confidentiality: > > This communication is intended for the above-named person and may be > > confidential and/or legally privileged. > > If it has come to you in error you must take no action based on it, nor > > must you copy or show it to anyone; please delete/destroy and inform the > > sender immediately. > > > > > > > > On Friday, 29 April 2016 at 13:51, Subramanian Karunanithi wrote: > > > > > Hi, > > > > > > When a stream of data passes through Kafka, wanted to apply the filter > > and > > > then let that message pass through to partitions. > > > > > > Regards, > > > Subramanian. K > > > On Apr 26, 2016 12:33, "Marko Bonaći" <marko.bon...@sematext.com > > (mailto:marko.bon...@sematext.com)> wrote: > > > > > > > Instantly reminded me of Streams API, where you can use Java8 streams > > > > semantics (filter being one of them) to do the first thing in > > Gouzhang's > > > > response (filter messages from one topic into another - I assume > that's > > > > what you were looking for). > > > > > > > > Marko Bonaći > > > > Monitoring | Alerting | Anomaly Detection | Centralized Log > Management > > > > Solr & Elasticsearch Support > > > > Sematext <http://sematext.com/> | Contact > > > > <http://sematext.com/about/contact.html> > > > > > > > > On Tue, Apr 26, 2016 at 6:22 PM, Guozhang Wang <wangg...@gmail.com > > (mailto:wangg...@gmail.com)> wrote: > > > > > > > > > Hi Subramanian, > > > > > > > > > > Could you elaborate a bit more on "filtering"? Do you want to read > > raw > > > > data > > > > > from Kafka, and send the filtered data back to Kafka as a separate > > topic, > > > > > or do you want to read raw data from an external service and send > the > > > > > filtered data into Kafka? > > > > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > On Tue, Apr 26, 2016 at 8:07 AM, Subramanian Karunanithi < > > > > > subub...@gmail.com (mailto:subub...@gmail.com) > > > > > > wrote: > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > Do we have any plugin which is available, which can be used as a > > > > > filtering > > > > > > mechanism on the data it's working on? > > > > > > > > > > > > Regards, > > > > > > Subramanian. K > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
