I would give this a look and see if it works, since you mention HDFS https://github.com/linkedin/gobblin
On Tue, Jan 26, 2016 at 3:49 PM, Nikhil Joshi <toughsh...@gmail.com> wrote: > Hi, > > I'm new to the Kafka community. Has there been any discussion around > plugging-in external filesystems (like HDFS) for Kafka persistence? Though > local filesystem gives the best throughput for the append-only Kafka log > data-structure, other filesystems might be able to provide better storage > efficiency, scale or geo-replication at the cost of throughput. > > Any pointers here? > > Nikhil >
