Re: Improving Kafka State Store performance

Hello, Since you are running on EBS not SSD, the first suspicion I'd have is its write and storage amplification. This can possibly be verified from RocksDB's own stats, and Bill once shared the code to expose such metrics for investigaion: https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-

Re: Improving Kafka State Store performance

Have you checked the EBS burst balance on your disks that the streams application is running on? On 21 September 2017 at 04:28, dev loper wrote: > Hi Bill, > > I will repeat my tests with Rocks DB enabled and I will revert to you with > details. I might take 1-2 days to get back to you with deta

Re: Improving Kafka State Store performance

Hi Bill, I will repeat my tests with Rocks DB enabled and I will revert to you with details. I might take 1-2 days to get back to you with details since I am traveling. But I will try my level best to get it tonight. On Mon, Sep 18, 2017 at 5:30 PM, Bill Bejeck wrote: > I'm following up from y

Re: Improving Kafka State Store performance

Understood, but since we haven't updated to use 5.7.3 yet, I think it's best to test against what is currently deployed. Thanks. On Mon, Sep 18, 2017 at 9:56 AM, Ted Yu wrote: > We're using rocksdb 5.3.6 > > It would make more sense to perform next round of experiment using rocksdb > 5.7.3 whic

Re: Improving Kafka State Store performance

We're using rocksdb 5.3.6 It would make more sense to perform next round of experiment using rocksdb 5.7.3 which is latest. Cheers On Mon, Sep 18, 2017 at 5:00 AM, Bill Bejeck wrote: > I'm following up from your other thread as well here. Thanks for the info > above, that is helpful. > > I th

Re: Improving Kafka State Store performance

I'm following up from your other thread as well here. Thanks for the info above, that is helpful. I think the AWS instance type might be a factor here, but let's do some more homework first. For a next step, we could enable logging for RocksDB so we can observe the performance. Here is some sam

Re: Improving Kafka State Store performance

@James, I will enable the monitoring metrics and try out. But I seriously think the performance degradation was due to rocks DB. As soon as I switched to in Memory State Store all my problems disappeared. @Sabarish, Only my kafka instances are on T2 Instance. My application instances are on M4

Re: Improving Kafka State Store performance

T2 instances are credit based instances that do not provide constant throughout in cpu and io. You would want to reconsider using t2. That said we too face issues with scaling rocksdb. Although I don't remember the performance benchmark numbers but I do remember it was at least 3x slower when comp

Re: Improving Kafka State Store performance

In addition to the measurements that you are doing yourself, Kafka Streams also has its own metrics. They are exposed via JMX, if you have that enabled: http://kafka.apache.org/documentation/#monitoring If you set metrics.recording.level="debu

Re: Improving Kafka State Store performance

Hi Bill. Thank you pointing out, But in actual code I am calling iter.close() in the finally block if the iterator is not null. I don't see any issues when I am running it on light traffic. As soon as I switch to production traffic I start seeing these issues. Below I have provided additional det

Re: Improving Kafka State Store performance

Hi, It's hard to say exactly without a little more information. On a side note, I don't see where you are closing the KeyValueIterator in the code above. Not closing a KeyValueIterator on a Permanent State Store can cause a resource leak over time, so I'd add `iter.close()` right before your `log

Re: Improving Kafka State Store performance

Hi Ted, I am using 0.11.0.0 . I am not using external state store. I am using the persistent store that comes with kafka stream 0.11.0.0. My assumption is put just inserts my object into the state store. I guess the state store is manged by rocksdb internally by kafkastreams. I am not sure why

Re: Improving Kafka State Store performance

Looking at the calculation of totalTimeTakenToStoreRecords, it covers store.put() call. Can you tell us more about what the put() does ? Does it involve external key value store ? Are you using 0.11.0.0 ? Thanks On Sat, Sep 16, 2017 at 6:14 AM, dev loper wrote: > Hi Kafka Streams Users, > > I