Hi Jinhua,
I will try to answer your questions:
Flink checkpoints the state of each operator. For a Kafka consumer
operator this is only the offset. For other operators (such as Windows
or a ProcessFunction) the values/list/maps stored in the state are
checkpointed. If you are interested in the internals, I would recommend
this page [1]. Only the MemoryStateBackend sends entire states to the
JobManager (see [2]). But you are right, this is a bottleneck and not
very fault-tolerant. Usually, Flink assumes to have some distributed
file system (such as HDFS) to which each Flink operator can be
checkpointed in a fault-tolerant way. For the RocksDbStateBackend the
local files are copied to HDFS as well. At the time of writing, only the
RocksDBBackend supports incremental checkpoints. The JobManager can then
read from HDFS and restore the operator on a different machine.
Feel free to ask further questions.
Regards,
Timo
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.4/internals/stream_checkpointing.html
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/state/state_backends.html
Am 1/1/18 um 3:50 PM schrieb Jinhua Luo:
Hi All,
I have two questions:
a) does the records/elements themselves would be checkpointed? or just
record offset checkpointed? That is, what data included in the
checkpoint except for states?
b) where flink stores the state globally? so that the job manager
could restore them on each task manger at failure restart.
For the heap backend, all task managers would send states to job
manager, and job manager would save it in its heap, correct?
For the fs/rocksdb backend, all task managers would save states
(incrementally or not) in local path temporarily, and send them (in
rocksdb snapshot format for the rocksdb case?) to the job manager at
checkpoint?
The path we used to configure backend is the path on the job manager
machine but not on the task managers' machines? So that's the
bottleneck and single failure point? So it's better to use hdfs path
so that we could scale the storage and make it high availability as
well?
Thank you all.
