Thanks for the good reference, Aditya, Daniel and Max. After study the materials, I have a general question regarding architecture for advice, it seems Flume could already implement messaging and tiered messaging mechanism, it seems overlap with the function of Kafka (from messaging system perspective), and shall we just use Flume + HDFS directly? Want to reduce Kafka is not necessary.
I could be wrong and please feel free to correct me. My use case is easy, just collect logs from Apache servers and transfer them to Hadoop HDFS. In the past, I think about Kafka since I think Flume is more kinds of light-weighted, in-capable of doing large scale of messaging processing, and learned Flume already doing messaging using its built-in Channel. Have a good weekend. regards, Lin On Fri, Mar 6, 2015 at 10:01 PM, max square <max2subscr...@gmail.com> wrote: > This presentation from a recent Kafka meetup in NYC describes different > approaches. > > http://www.slideshare.net/gwenshap/kafka-hadoop-for-nyc-kafka-meetup?ref=http://ingest.tips/2014/10/16/notes-from-kafka-meetup/ > > It´s companion blog post is this: > http://ingest.tips/2014/10/16/notes-from-kafka-meetup/ > > Cheers > > Max > > On Sat, Mar 7, 2015 at 12:18 AM, Daniel Moreno <d...@max2.com> wrote: > > > You can also try the approach described here > > > http://blog.cloudera.com/blog/2014/11/flafka-apache-flume-meets-apache-kafka-for-event-processing/ > > > > Kind Regards, > > > > Daniel > > > > > > On Mar 6, 2015, at 23:20, Lin Ma <lin...@gmail.com<mailto: > lin...@gmail.com>> > > wrote: > > > > Hi Kafka masters, > > > > Wondering if any open source solutions, to transfer message received from > > Kakfa to Hadoop HDFS? Thanks. > > > > regards, > > Lin > > >
