Re: Kafka Stream: RocksDBKeyValueStoreSupplier performance

2017-03-16 Thread Tianji Li
Hi Eno, Thanks for your help. Very appreciated. Thanks Tianji On Wed, Mar 15, 2017 at 4:29 PM, Eno Thereska wrote: > Tianji, > > A couple of things: > > - for now could you use RocksDb without the cache? I've opened a JIRA to > verify why it's slower with the cache: https://issues.apache.org/

Re: Kafka Stream: RocksDBKeyValueStoreSupplier performance

2017-03-15 Thread Eno Thereska
Tianji, A couple of things: - for now could you use RocksDb without the cache? I've opened a JIRA to verify why it's slower with the cache: https://issues.apache.org/jira/browse/KAFKA-4904 - you can tune the RocksDb performance further by in

Re: Kafka Stream: RocksDBKeyValueStoreSupplier performance

2017-03-15 Thread Tianji Li
Hi Eno, Rocksdb without caching took around 7 minutes. Tianji On Wed, Mar 15, 2017 at 9:40 AM, Eno Thereska wrote: > Tianji, > > Could you provide a third data point, running with RocksDb, but without > caching, i.e: > > > StateStoreSupplier stateStoreSupplier = Stores.create(storeName) > >

Re: Kafka Stream: RocksDBKeyValueStoreSupplier performance

2017-03-15 Thread Eno Thereska
Tianji, Could you provide a third data point, running with RocksDb, but without caching, i.e: > StateStoreSupplier stateStoreSupplier = Stores.create(storeName) >.withKeys(stringSerde) >.withValues(avroSerde) >.persistent() >.disableLogging() >.build();

Kafka Stream: RocksDBKeyValueStoreSupplier performance

2017-03-15 Thread Tianji Li
Hi there, It seems that the RocksDB state store is quite slow in my case and I wonder if I did anything wrong. I have a topic, that I groupBy() and then aggregate() 50 times. That is, I will create 50 results topics and a lot more changelog and repartition topics. There are a few things that are