Thanks a lot Stefan. This clarifies everything.
Regards,
Chirag
On Thursday, 26 April, 2018, 7:16:52 PM IST, Stefan Richter
<[email protected]> wrote:
Adding one thing, the format of the non-incremental is similar but
unfortunately not (yet) identical with the FS backend. This is because of some
internal implementation details that allow the FS checkpoints to be slightly
more consise in the file format but we might „de-optimize“ this minor
difference for the sake of compatibility in the near future.
Am 26.04.2018 um 15:22 schrieb Chirag Dewan <[email protected]>:
Wow never considered it that way.
Thanks a lot for clarifying Stefan.
This gives rise to another question. Whats the format of this data? Is it the
same format which is used to store checkpoints when FS state backend is used?
Regards,
Chirag
Sent from Yahoo Mail on Android
On Thu, 26 Apr 2018 at 5:30 PM, Stefan Richter<[email protected]>
wrote: Hi,
I think there is a misunderstanding. RocksDB state backend always operates on
local disk of the node that runs your task to give you optimal performance. You
can think of this as a transient working area that does not require any
durability. Durability always happens through checkpoints (or savepoints)
which, in turn, go to a distributed storage. Checkpoints and checkpoints are
like a consistent moment-in-time image of the backends content and can be used
to recover under failure (checkpoints) or manually resume your job (savepoints).
Best,Stefan
Am 26.04.2018 um 13:16 schrieb Chirag Dewan <[email protected]>:
Hi,
I am working on a use case where I need to store a large amount of data in
state. I am using RocksDB as my state backend. Now to ensure data replication,
I want to store the RocksDB files in some distributed file system.
>From the documentation I can see that Flink recommends a list of FileSystem to
>be used for state backend. Given here :
Apache Flink 1.4 Documentation: File Systems
|
|
| |
Apache Flink 1.4 Documentation: File Systems
|
|
|
But I cannot figure out the file system for RocksDB. What are the
recommendations for File Systems to be used with RocksDB?
Thanks in advance.
Regards,
Chirag
