Hello,
I've been studying different options to consume messages from kafka to
hadoop(hdfs) and found three odds.
Linkedin Camus - https://github.com/linkedin/camus
kafka-hadoop-loader - https://github.com/michal-harish/kafka-hadoop-loader
hadoop-consumer -
https://github.com/apache/kafka/tree/0.8
We’ve been using Miniway’s hadoop-consumer in production for over a year
without any problems. It stores offsets in zookeeper rather than HDFS and it
uses the more recent mapreduce api.
https://github.com/miniway/kafka-hadoop-consumer
On Feb 13, 2014, at 11:18 AM, Marcelo Valle wrote:
> Hell
Hi,
Camus does support raw text messages. If I remember correctly, you just
need to provide your own record decoder and record writer.
We are using Camus to consume messages from Kafka and store them to S3 and
it works quite well.
Maxime
On Thu, Feb 13, 2014 at 8:18 AM, Marcelo Valle wrote:
Hello,
I've been studying different options to consume messages from kafka to
hadoop(hdfs) and found three odds.
Linkedin Camus - https://github.com/linkedin/camus
kafka-hadoop-loader - https://github.com/michal-harish/kafka-hadoop-loader
hadoop-consumer -
https://github.com/apache/kafka/tree/0.8
