HI Ben, Sharninder, Thanks for your responses, I appreciate it.
Ben - thanks for the tips on settings. A backup could certainly be a possibility, although if only with similar durability guarantees, I'm not sure what the purpose would be? Sharninder - yes, we would only be using the logs as forward-only streams - i.e. picking an offset to read from and moving forwards - and would be setting retention time to essentially infinite. Regards, Ted. On Tue, Feb 16, 2016 at 5:05 AM, Sharninder Khera <sharnin...@gmail.com> wrote: > This topic comes up often on this list. Kafka can be used as a datastore > if that’s what your application wants with the caveat that Kafka isn’t > designed to keep data around forever. There is a default retention time > after which older data gets deleted. The high level consumer essentially > reads data as a stream and while you can do sort of random access with the > low level consumer, its not ideal. > > > > > On 15-Feb-2016, at 10:26 PM, Ted Swerve <ted.swe...@gmail.com> wrote: > > > > Hello, > > > > Is it viable to use infinite-retention Kafka topics as a master data > > store? I'm not talking massive volumes of data here, but still > potentially > > extending into tens of terabytes. > > > > Are there any drawbacks or pitfalls to such an approach? It seems like a > > compelling design, but there seem to be mixed messages about its > > suitability for this kind of role. > > > > Regards, > > Ted > >
