There is also hadoop contrib producer and consumer https://github.com/apache/kafka/tree/0.8/contrib for hdfs
/******************************************* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> ********************************************/ On Tue, Jan 14, 2014 at 4:16 PM, Blender Bl <bl3nd...@aol.com> wrote: > > Hi, > > > My team trying to implement lambda architecture. > We need to stream all our new data though Kafka to storm, and HDFS. > > > As i see it were are two options: > > Using Camus - not very efficent > Streaming via Storm - not very efficent > > Is it possible to persist the queue's files over the HDFS (with > short-circuit setting switched on) instead of the local filesystem? > > > It suppose to have similar performance to the local file system, doesn't > it? >
