Re: aggregation tables mirrored in kafka & rocksdb

2018-02-28 Thread Guozhang Wang
Hello Nicu, For your aggregation application, is it windowed or non windowed? If it is windowed aggregation then you can specify your window specs so that the underlying RocksDB state store would only keep the most recent windows, while your Cassandra keeps the full history of all past windows. Y

aggregation tables mirrored in kafka & rocksdb

2018-02-28 Thread Marasoiu, Nicu
Hi, Currently we have an aggregation system (without kafka) where events are aggregated into Cassandra tables holding aggregate results. We are considering moving to a KafkaStreams solution with exactly-once processing but in this case it seems that all the aggregation tables (reaching TB) need