I know several people that use the qubole Kafka Sink Connector for S3 ( see https://github.com/qubole/streamx <https://github.com/qubole/streamx>) to store Kafka messages in S3 for long term archiving. You can also do this with the Confluent HDFS Kafka Connector if you have access to a Hadoop cluster
-hans > On Dec 6, 2016, at 3:25 AM, Aseem Bansal <asmbans...@gmail.com> wrote: > > Hi > > Has anyone done a storage of Kafka JSON messages to deep storage like S3. > We are looking to back up all of our raw Kafka JSON messages for > Exploration. S3, HDFS, MongoDB come to mind initially. > > I know that it can be stored in kafka itself but storing them in Kafka > itself does not seem like a good option as we won't be able to query it and > the configurations of machines containing kafka will have to be increased > as we go. Something like S3 we won't have to manage.
