Hi Everyone, as outlined in FLIP-194 discussion [1], for the future directions of Flink HA services, I'd like to verify my thoughts around guarantees of the distributed filesystems used with Flink.
Currently some of the services (*JobGraphStore*, *CompletedCheckpointStore*) are implemented using a combination of strongly consistent Metadata storage (ZooKeeper, K8s CM) and the actual FileSystem. Reasoning behind this dates back to days, when S3 was an eventually consistent FileSystem and we needed a strongly consistent view of the data. I did some research, and my feeling is that all the major FileSystems that Flink supports already provide strong read-after-write consistency, which would be sufficient to decrease a complexity of the current HA implementations. FileSystems that I've checked and that seem to support strong read-after-write consistency: - S3 - GCS - Azure Blob Storage - Aliyun OSS - HDFS - Minio Are you aware of other FileSystems that are used with Flink? Do they support the consistency that is required for starting a new initiatives towards simpler / less error-prone HA services? Are you aware of any problems with the above mentioned FileSystems that I might have missed? I'm also bringing this up to user@f.a.o, to make sure we don't miss any FileSystems. [1] https://lists.apache.org/thread/wlzv02jqtq221kb8dnm82v4xj8tomd94 Best, D.
