Ok, that makes sense. I had assumed that the changelog was supported because the docs mention that TTL is enforced upon ³compaction² (I had assumed compaction of the DB changelog). Which topic does the TTL policy listen for the compaction of (since compaction policies of topics can differ)?
-David On 1/27/16, 8:46 PM, "Jacob Maes" <jacob.m...@gmail.com> wrote: >Here's my understanding. The others can correct me if I'm mistaken. > >Samza provides the changelog functionality by intercepting RocksDB "put" >and "delete" operations. However, TTL is managed by RocksDB internally and >there aren't any hooks exposed in the RocksDB JNI. So there are 2 problems >that arise with TTL and change logging: >1. Samza doesn't know when an entry expires, so it can't delete the >expired >entry from the changelog. >2. The changelog currently has no concept of entry age/timestamp, so when >the changelog is restored, it's unknown whether some subset (or all) of >the >entries should be immediately expired. > >These issues aren't insurmountable, but they weren't pursued for the >initial implementation. Perhaps because there was a shortage of use cases >that needed both TTL and changelogging, but I'm not sure. > >-Jake > >On Wed, Jan 27, 2016 at 6:19 PM, David Garcia ><dgar...@homeaway.com.invalid> >wrote: > >> So, I saw this very scary message: >> >> >> ERROR - e.kv.RocksDbKeyValueStore$ - sessionJoinStore is a TTL based >> store, changelog is not supported for TTL based stores, use at your own >> discretion >> >> >> >> >> A few of questions: >> >> 1.) Does this mean that this store is NOT backed by the changelog? >> >> 2.) Provided that the store IS backed by a change log, do the TTL >> expirations commit removals from the changelog (I.e. Nulls)...presumably >> upon compaction >> >> 3.) Can I please get a bit more detail on how TTL affects a changelog >> store? >> >> >> -David >>
