Following up on my question regarding backed Filesystem (HCFS) requirements.
Appreciate any inputs.
---Regarding the Filesystem abstraction support, we are planning to use a
distributed file system which complies with Hadoop Compatible File System
(HCFS) standard in place of standard HDFS.
According to the documentation
(https://ci.apache.org/projects/flink/flink-docs-release-1.3/internals/filesystems.html),
persistence gurantees is listed as one of the main requirement and to be
precises it qualifies both visibility and durability gurantees.
My question is,
1) Are we expecting the file system to support "Atomic Rename" characteristics?
I believe checkpoint mechanism involves in renaming the files and will that
have an impact if "atomic rename" is not guranteed by the underlying file
system?
2) How does one certify Flink with HCFS (in place of standard HDFS) in terms of
the scenarios/usecase that needs to be tested? Is there any general guidance on
this?---
RegardsVijay
On Wednesday, February 15, 2017 11:28 AM, Vijay Srinivasaraghavan
<vijikar...@yahoo.com> wrote:
Hello,
Regarding the Filesystem abstraction support, we are planning to use a
distributed file system which complies with Hadoop Compatible File System
(HCFS) standard in place of standard HDFS.
According to the documentation
(https://ci.apache.org/projects/flink/flink-docs-release-1.3/internals/filesystems.html),
persistence gurantees is listed as one of the main requirement and to be
precises it qualifies both visibility and durability gurantees.
My question is,
1) Are we expecting the file system to support "Atomic Rename" characteristics?
I believe checkpoint mechanism involves in renaming the files and will that
have an impact if "atomic rename" is not guranteed by the underlying file
system?
2) How does one certify Flink with HCFS (in place of standard HDFS) in terms of
the scenarios/usecase that needs to be tested? Is there any general guidance on
this?
ThanksVijay
