Why not just have a parallel consumer read all messages from whichever topics
you're interested in and store them wherever you want to? You don't need to
"backup" Kafka messages.
_____________________________
From: Aseem Bansal <asmbans...@gmail.com>
Sent: Tuesday, December 6, 2016 4:55 PM
Subject: Storing Kafka Message JSON to deep storage like S3
To: <users@kafka.apache.org>
Hi
Has anyone done a storage of Kafka JSON messages to deep storage like S3.
We are looking to back up all of our raw Kafka JSON messages for
Exploration. S3, HDFS, MongoDB come to mind initially.
I know that it can be stored in kafka itself but storing them in Kafka
itself does not seem like a good option as we won't be able to query it and
the configurations of machines containing kafka will have to be increased
as we go. Something like S3 we won't have to manage.
