Hi Eno, Rocksdb without caching took around 7 minutes.
Tianji On Wed, Mar 15, 2017 at 9:40 AM, Eno Thereska <eno.there...@gmail.com> wrote: > Tianji, > > Could you provide a third data point, running with RocksDb, but without > caching, i.e: > > > StateStoreSupplier stateStoreSupplier = Stores.create(storeName) > > .withKeys(stringSerde) > > .withValues(avroSerde) > > .persistent() > > .disableLogging() > > .build(); > > > Thanks > Eno > > > > On 15 Mar 2017, at 13:02, Tianji Li <skyah...@gmail.com> wrote: > > > > Hi there, > > > > It seems that the RocksDB state store is quite slow in my case and I > wonder > > if I did anything wrong. > > > > I have a topic, that I groupBy() and then aggregate() 50 times. That is, > I > > will create 50 results topics and a lot more changelog and repartition > > topics. > > > > There are a few things that are weird and here I report one, which is the > > State store speed. > > > > If I use: > > > > StateStoreSupplier stateStoreSupplier = Stores.create(storeName) > > .withKeys(stringSerde) > > .withValues(avroSerde) > > .inMemory() > > .build(); > > > > Then processing 1 millions records takes around 5 minutes on my coding > > computer. > > > > If I use: > > > > StateStoreSupplier stateStoreSupplier = Stores.create(storeName) > > .withKeys(stringSerde) > > .withValues(avroSerde) > > .persistent() > > .disableLogging() > > .enableCaching() > > .build(); > > > > Processing the same 1 million records takes around 10 minutes. > > > > I believe in the first case, changelog is backed up to Kafka and in the > > second case, only RocketsDB is used. > > > > But why the RocketsDB is so slow? > > > > Eventually, I am hoping to do windowed aggregation and it seems I have to > > use RocketsDB, but given the performance, I am hesitating. > > > > Thanks > > Tianji > >
