Thanks, I'm on 0.8.2 so that explains it. Should retention.ms affect segment rolling? In my experiment it did ( retention.ms = -1), which was unexpected since I thought only segment.bytes and segment.ms would control that.
On Mon, Jul 13, 2015 at 7:57 PM, Daniel Tamai <daniel.ta...@gmail.com> wrote: > Using -1 for log.retention.ms should work only for 0.8.3 ( > https://issues.apache.org/jira/browse/KAFKA-1990). > > 2015-07-13 17:08 GMT-03:00 Shayne S <shaynest...@gmail.com>: > > > Did this work for you? I set the topic settings to retention.ms=-1 and > > retention.bytes=-1 and it looks like it is deleting segments immediately. > > > > On Sun, Jul 12, 2015 at 8:02 AM, Daniel Schierbeck < > > daniel.schierb...@gmail.com> wrote: > > > > > > > > > On 10. jul. 2015, at 23.03, Jay Kreps <j...@confluent.io> wrote: > > > > > > > > If I recall correctly, setting log.retention.ms and > > log.retention.bytes > > > to > > > > -1 disables both. > > > > > > Thanks! > > > > > > > > > > > On Fri, Jul 10, 2015 at 1:55 PM, Daniel Schierbeck < > > > > daniel.schierb...@gmail.com> wrote: > > > > > > > >> > > > >>> On 10. jul. 2015, at 15.16, Shayne S <shaynest...@gmail.com> > wrote: > > > >>> > > > >>> There are two ways you can configure your topics, log compaction > and > > > with > > > >>> no cleaning. The choice depends on your use case. Are the records > > > >> uniquely > > > >>> identifiable and will they receive updates? Then log compaction is > > the > > > >> way > > > >>> to go. If they are truly read only, you can go without log > > compaction. > > > >> > > > >> I'd rather be free to use the key for partitioning, and the records > > are > > > >> immutable — they're event records — so disabling compaction > altogether > > > >> would be preferable. How is that accomplished? > > > >>> > > > >>> We have a small processes which consume a topic and perform upserts > > to > > > >> our > > > >>> various database engines. It's easy to change how it all works and > > > simply > > > >>> consume the single source of truth again. > > > >>> > > > >>> I've written a bit about log compaction here: > > > >>> > > > > http://www.shayne.me/blog/2015/2015-06-25-everything-about-kafka-part-2/ > > > >>> > > > >>> On Fri, Jul 10, 2015 at 3:46 AM, Daniel Schierbeck < > > > >>> daniel.schierb...@gmail.com> wrote: > > > >>> > > > >>>> I'd like to use Kafka as a persistent store – sort of as an > > > alternative > > > >> to > > > >>>> HDFS. The idea is that I'd load the data into various other > systems > > in > > > >>>> order to solve specific needs such as full-text search, analytics, > > > >> indexing > > > >>>> by various attributes, etc. I'd like to keep a single source of > > truth, > > > >>>> however. > > > >>>> > > > >>>> I'm struggling a bit to understand how I can configure a topic to > > > retain > > > >>>> messages indefinitely. I want to make sure that my data isn't > > deleted. > > > >> Is > > > >>>> there a guide to configuring Kafka like this? > > > >> > > > > > >
