Hi Aljoscha,
thanks for your answers. I am currently not in the office, so I can not
run any further analysis until Monday. Just some quick answers to your
questions.
We are using the partitioned state abstraction, most of the state should
correspond to buffered events in windows. Parallelism is
ding, so it's
> great to know that it's already on the radar. If we beat you to it, I'll
> definitely try to make it a contribution.
>
> Shannon
>
> From: Aljoscha Krettek
> Date: Wednesday, April 13, 2016 at 1:46 PM
> To:
> Subject: Re: RocksDB Statebacke
3, 2016 at 1:46 PM
To: mailto:user@flink.apache.org>>
Subject: Re: RocksDB Statebackend
Hi Maxim,
yes the plan is to have a cache of hot values that uses the managed memory
abstraction of Flink so that we can make sure that we stay within memory bounds
and don't run into OOM exception
Hi Maxim,
yes the plan is to have a cache of hot values that uses the managed memory
abstraction of Flink so that we can make sure that we stay within memory
bounds and don't run into OOM exceptions.
On Tue, 12 Apr 2016 at 23:37 Maxim wrote:
> Is it possible to add an option to store the state i
Is it possible to add an option to store the state in the Java HashMap and
write its content to RocksDB when checkpointing? For "hot" keys that are
updated very frequently such optimization would help with performance.
I know that you are also working on incremental checkpoints which would
also be
Concerning the size of RocksDB snapshots - I am wondering if RocksDB simply
does not compact for a long time, thus having a lot of stale data in the
snapshot.
That would be especially the case, if you have a lot of changing values for
the same set of keys.
On Tue, Apr 12, 2016 at 6:41 PM, Aljosch
Hi,
I'm going to try and respond to each point:
1. This seems strange, could you give some background on parallelism,
number of operators with state and so on? Also, I'm assuming you are using
the partitioned state abstraction, i.e. getState(), correct?
2. your observations are pretty much correc
