Hi Hao, You do not necessarily need HDFS but some kind of distribute filesystem that can be accessed from all nodes is required. Flink doesn't need the FS just for job meta data, but also to store checkpoints of the application state for fault tolerance.
Best, Fabian 2018-05-08 4:31 GMT+02:00 Hao Chen <cn.haoc...@gmail.com>: > Hi, > > As for lots of companies, we have different architectures for > near-time/off-line analytics platform (fink vs hadoop), so we may not > want flink cluster to depend on a shared HDFS as an additional dependency > to operate especially we may deploy the streaming stack all on k8s > including kafka/flink without hdfs. > > Is it possible to implement a way to sync metadata across different HA job > managers instead of depending on a shared file system > for high-availability.zookeeper.storageDir. > > > -- > > Hao >
