This topic comes up often on this list. Kafka can be used as a datastore if that’s what your application wants with the caveat that Kafka isn’t designed to keep data around forever. There is a default retention time after which older data gets deleted. The high level consumer essentially reads data as a stream and while you can do sort of random access with the low level consumer, its not ideal.
> On 15-Feb-2016, at 10:26 PM, Ted Swerve <ted.swe...@gmail.com> wrote: > > Hello, > > Is it viable to use infinite-retention Kafka topics as a master data > store? I'm not talking massive volumes of data here, but still potentially > extending into tens of terabytes. > > Are there any drawbacks or pitfalls to such an approach? It seems like a > compelling design, but there seem to be mixed messages about its > suitability for this kind of role. > > Regards, > Ted
