Hi, David, The "compaction" referred to together w/ TTL is referring to RocksDb's compaction, not the Kafka-based changelog topic. Currently, TTL is not applied to Kafka-based changelog topic. SAMZA-677 is opened for this.
-Yi On Thu, Jan 28, 2016 at 11:36 AM, David Garcia <dgar...@homeaway.com.invalid > wrote: > Ok, that makes sense. I had assumed that the changelog was supported > because the docs mention that TTL is enforced upon ³compaction² (I had > assumed compaction of the DB changelog). Which topic does the TTL policy > listen for the compaction of (since compaction policies of topics can > differ)? > > -David > > On 1/27/16, 8:46 PM, "Jacob Maes" <jacob.m...@gmail.com> wrote: > > >Here's my understanding. The others can correct me if I'm mistaken. > > > >Samza provides the changelog functionality by intercepting RocksDB "put" > >and "delete" operations. However, TTL is managed by RocksDB internally and > >there aren't any hooks exposed in the RocksDB JNI. So there are 2 problems > >that arise with TTL and change logging: > >1. Samza doesn't know when an entry expires, so it can't delete the > >expired > >entry from the changelog. > >2. The changelog currently has no concept of entry age/timestamp, so when > >the changelog is restored, it's unknown whether some subset (or all) of > >the > >entries should be immediately expired. > > > >These issues aren't insurmountable, but they weren't pursued for the > >initial implementation. Perhaps because there was a shortage of use cases > >that needed both TTL and changelogging, but I'm not sure. > > > >-Jake > > > >On Wed, Jan 27, 2016 at 6:19 PM, David Garcia > ><dgar...@homeaway.com.invalid> > >wrote: > > > >> So, I saw this very scary message: > >> > >> > >> ERROR - e.kv.RocksDbKeyValueStore$ - sessionJoinStore is a TTL based > >> store, changelog is not supported for TTL based stores, use at your own > >> discretion > >> > >> > >> > >> > >> A few of questions: > >> > >> 1.) Does this mean that this store is NOT backed by the changelog? > >> > >> 2.) Provided that the store IS backed by a change log, do the TTL > >> expirations commit removals from the changelog (I.e. Nulls)...presumably > >> upon compaction > >> > >> 3.) Can I please get a bit more detail on how TTL affects a changelog > >> store? > >> > >> > >> -David > >> > >
