: "user@flink.apache.org"
Date: Thursday, February 16, 2017 at 8:24 AM
To: "user@flink.apache.org"
Subject: Re: Resource under-utilization when using RocksDb state backend
[SOLVED]
Hi Cliff,
It will be really helpful if you could share your RocksDB configuration.
I am also runni
;>>>>
>>>>> Here is a sample top:
>>>>>
>>>>> Cpu0 : 20.5%us, 0.0%sy, 0.0%ni, 79.5%id, 0.0%wa, 0.0%hi, 0.0%si,
>>>>> 0.0%st
>>>>> Cpu1 : 18.5%us, 0.0%sy, 0.0%ni, 81.5%id, 0.0%wa, 0.0%hi, 0.0%si,
>
Great to hear!
On Fri, 9 Dec 2016 at 01:02 Cliff Resnick wrote:
> It turns out that most of the time in RocksDBFoldingState was spent on
> serialization/deserializaton. RocksDb read/write was performing well. By
> moving from Kryo to custom serialization we were able to increase
> throughput dra
It turns out that most of the time in RocksDBFoldingState was spent on
serialization/deserializaton. RocksDb read/write was performing well. By
moving from Kryo to custom serialization we were able to increase
throughput dramatically. Load is now where it should be.
On Mon, Dec 5, 2016 at 1:15 PM,
Another Flink user using RocksDB with large state on SSDs recently posted
this video for oprimizing the performance of Rocks on SSDs:
https://www.youtube.com/watch?v=pvUqbIeoPzM
That could be relevant for you.
For how long did you look at iotop. It could be that the IO access happens
in bursts, de
Hi Robert,
We're following 1.2-SNAPSHOT, using event time. I have tried "iotop" and I
see usually less than 1 % IO. The most I've seen was a quick flash here or
there of something substantial (e.g. 19%, 52%) then back to nothing. I also
assumed we were disk-bound, but to use your metaphor I'm hav
Hi Cliff,
which Flink version are you using?
Are you using Eventtime or processing time windows?
I suspect that your disks are "burning" (= your job is IO bound). Can you
check with a tool like "iotop" how much disk IO Flink is producing?
Then, I would set this number in relation with the theoret
In tests comparing RocksDb to fs state backend we observe much lower
throughput, around 10x slower. While the lowered throughput is expected,
what's perplexing is that machine load is also very low with RocksDb,
typically falling to < 25% CPU and negligible IO wait (around 0.1%). Our
test instance
