Would it be possible to document how to configure Kafka to never delete messages in a topic? It took a good while to figure this out, and I see it as an important use case for Kafka.
On Sun, Jul 12, 2015 at 3:02 PM Daniel Schierbeck < daniel.schierb...@gmail.com> wrote: > > > On 10. jul. 2015, at 23.03, Jay Kreps <j...@confluent.io> wrote: > > > > If I recall correctly, setting log.retention.ms and log.retention.bytes > to > > -1 disables both. > > Thanks! > > > > > On Fri, Jul 10, 2015 at 1:55 PM, Daniel Schierbeck < > > daniel.schierb...@gmail.com> wrote: > > > >> > >>> On 10. jul. 2015, at 15.16, Shayne S <shaynest...@gmail.com> wrote: > >>> > >>> There are two ways you can configure your topics, log compaction and > with > >>> no cleaning. The choice depends on your use case. Are the records > >> uniquely > >>> identifiable and will they receive updates? Then log compaction is the > >> way > >>> to go. If they are truly read only, you can go without log compaction. > >> > >> I'd rather be free to use the key for partitioning, and the records are > >> immutable — they're event records — so disabling compaction altogether > >> would be preferable. How is that accomplished? > >>> > >>> We have a small processes which consume a topic and perform upserts to > >> our > >>> various database engines. It's easy to change how it all works and > simply > >>> consume the single source of truth again. > >>> > >>> I've written a bit about log compaction here: > >>> > http://www.shayne.me/blog/2015/2015-06-25-everything-about-kafka-part-2/ > >>> > >>> On Fri, Jul 10, 2015 at 3:46 AM, Daniel Schierbeck < > >>> daniel.schierb...@gmail.com> wrote: > >>> > >>>> I'd like to use Kafka as a persistent store – sort of as an > alternative > >> to > >>>> HDFS. The idea is that I'd load the data into various other systems in > >>>> order to solve specific needs such as full-text search, analytics, > >> indexing > >>>> by various attributes, etc. I'd like to keep a single source of truth, > >>>> however. > >>>> > >>>> I'm struggling a bit to understand how I can configure a topic to > retain > >>>> messages indefinitely. I want to make sure that my data isn't deleted. > >> Is > >>>> there a guide to configuring Kafka like this? > >> >
