Re: Kafka to Hadoop HDFS

2015-03-07 Thread Lin Ma
Thanks for the good reference, Aditya, Daniel and Max. After study the materials, I have a general question regarding architecture for advice, it seems Flume could already implement messaging and tiered messaging mechanism, it seems overlap with the function of Kafka (from messaging system perspec

Re: Kafka to Hadoop HDFS

2015-03-07 Thread Jiangjie Qin
Yet another option is camus, that we are using in LinkedIn: https://github.com/linkedin/camus Jiangjie (Becket) Qin On 3/6/15, 10:01 PM, "max square" wrote: >This presentation from a recent Kafka meetup in NYC describes different >approaches. >http://www.slideshare.net/gwenshap/kafka-hadoop-fo

Re: Kafka to Hadoop HDFS

2015-03-06 Thread max square
This presentation from a recent Kafka meetup in NYC describes different approaches. http://www.slideshare.net/gwenshap/kafka-hadoop-for-nyc-kafka-meetup?ref=http://ingest.tips/2014/10/16/notes-from-kafka-meetup/ It´s companion blog post is this: http://ingest.tips/2014/10/16/notes-from-kafka-meetu

Re: Kafka to Hadoop HDFS

2015-03-06 Thread Daniel Moreno
You can also try the approach described here http://blog.cloudera.com/blog/2014/11/flafka-apache-flume-meets-apache-kafka-for-event-processing/ Kind Regards, Daniel On Mar 6, 2015, at 23:20, Lin Ma mailto:lin...@gmail.com>> wrote: Hi Kafka masters, Wondering if any open source solutions, to

RE: Kafka to Hadoop HDFS

2015-03-06 Thread Aditya Auradkar
Try this. https://github.com/linkedin/camus Aditya From: Lin Ma [lin...@gmail.com] Sent: Friday, March 06, 2015 8:19 PM To: users@kafka.apache.org Subject: Kafka to Hadoop HDFS Hi Kafka masters, Wondering if any open source solutions, to transfer message