Using -1 for log.retention.ms should work only for 0.8.3 ( https://issues.apache.org/jira/browse/KAFKA-1990).
2015-07-13 17:08 GMT-03:00 Shayne S <shaynest...@gmail.com>: > Did this work for you? I set the topic settings to retention.ms=-1 and > retention.bytes=-1 and it looks like it is deleting segments immediately. > > On Sun, Jul 12, 2015 at 8:02 AM, Daniel Schierbeck < > daniel.schierb...@gmail.com> wrote: > > > > > > On 10. jul. 2015, at 23.03, Jay Kreps <j...@confluent.io> wrote: > > > > > > If I recall correctly, setting log.retention.ms and > log.retention.bytes > > to > > > -1 disables both. > > > > Thanks! > > > > > > > > On Fri, Jul 10, 2015 at 1:55 PM, Daniel Schierbeck < > > > daniel.schierb...@gmail.com> wrote: > > > > > >> > > >>> On 10. jul. 2015, at 15.16, Shayne S <shaynest...@gmail.com> wrote: > > >>> > > >>> There are two ways you can configure your topics, log compaction and > > with > > >>> no cleaning. The choice depends on your use case. Are the records > > >> uniquely > > >>> identifiable and will they receive updates? Then log compaction is > the > > >> way > > >>> to go. If they are truly read only, you can go without log > compaction. > > >> > > >> I'd rather be free to use the key for partitioning, and the records > are > > >> immutable — they're event records — so disabling compaction altogether > > >> would be preferable. How is that accomplished? > > >>> > > >>> We have a small processes which consume a topic and perform upserts > to > > >> our > > >>> various database engines. It's easy to change how it all works and > > simply > > >>> consume the single source of truth again. > > >>> > > >>> I've written a bit about log compaction here: > > >>> > > http://www.shayne.me/blog/2015/2015-06-25-everything-about-kafka-part-2/ > > >>> > > >>> On Fri, Jul 10, 2015 at 3:46 AM, Daniel Schierbeck < > > >>> daniel.schierb...@gmail.com> wrote: > > >>> > > >>>> I'd like to use Kafka as a persistent store – sort of as an > > alternative > > >> to > > >>>> HDFS. The idea is that I'd load the data into various other systems > in > > >>>> order to solve specific needs such as full-text search, analytics, > > >> indexing > > >>>> by various attributes, etc. I'd like to keep a single source of > truth, > > >>>> however. > > >>>> > > >>>> I'm struggling a bit to understand how I can configure a topic to > > retain > > >>>> messages indefinitely. I want to make sure that my data isn't > deleted. > > >> Is > > >>>> there a guide to configuring Kafka like this? > > >> > > >
