Hello Chesnay, Cluster (in kubernetes)-wide checkpointing directory using glusterfs volume mount (thus file access protocol file:///) was working fine till 1.4.2 for us. So we like to understand where the breakage happened in 1.5.0. Can you please mention me the relevant source code files related to rocksdb “custom file path” parsing logic? We would be interested to investigate this.
I also observed below in the log – Config uses deprecated configuration key 'state.backend.rocksdb.checkpointdir' instead of proper key 'state.backend.rocksdb.localdir' Regards, Shaswata From: Chesnay Schepler [mailto:ches...@apache.org] Sent: Tuesday, July 03, 2018 5:52 PM To: Data Engineer <dataenginee...@gmail.com> Cc: user@flink.apache.org Subject: Re: Checkpointing in Flink 1.5.0 The code appears to be working fine. This may happen because you're using a GlusterFS volume. The RocksDBStateBackend uses java Files internally (NOT nio Paths), which AFAIK only work properly against the plain local file-system. The GlusterFS nio FIleSystem implementation also explicitly does not support conversions to File<https://github.com/gluster/glusterfs-java-filesystem/blob/master/glusterfs-java-filesystem/src/main/java/com/peircean/glusterfs/GlusterPath.java#L271>. On 03.07.2018 13:53, Chesnay Schepler wrote: Thanks. Looks like RocksDBStateBackend.setDbStoragePaths has some custom file path parsing logic, will probe it a bit to see what the issue is. On 03.07.2018 13:45, Data Engineer wrote: 2018-07-03 11:30:35,703 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -------------------------------------------------------------------------------- 2018-07-03 11:30:35,705 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting StandaloneSessionClusterEntrypoint (Version: <unknown>, Rev:c61b108, Date:24.05.2018 @ 16:54:44 CEST) 2018-07-03 11:30:35,705 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS current user: flink 2018-07-03 11:30:35,705 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Current Hadoop/Kerberos user: <no hadoop dependency found> 2018-07-03 11:30:35,706 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.171-b10 2018-07-03 11:30:35,706 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum heap size: 981 MiBytes 2018-07-03 11:30:35,706 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JAVA_HOME: /etc/alternatives/jre_openjdk/ 2018-07-03 11:30:35,707 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - No Hadoop Dependency available 2018-07-03 11:30:35,707 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM Options: 2018-07-03 11:30:35,707 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xms1024m 2018-07-03 11:30:35,707 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xmx1024m 2018-07-03 11:30:35,708 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog.file=/opt/flink-1.5.0/log/flink--standalonesession-0-myfl-flink-jobmanager-7b4d8c4dd4-bv6zf.log 2018-07-03 11:30:35,708 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog4j.configuration=file:/opt/flink-1.5.0/conf/log4j.properties<file://opt/flink-1.5.0/conf/log4j.properties> 2018-07-03 11:30:35,708 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlogback.configurationFile=file:/opt/flink-1.5.0/conf/logback.xml<file://opt/flink-1.5.0/conf/logback.xml> 2018-07-03 11:30:35,708 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Program Arguments: 2018-07-03 11:30:35,709 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --configDir 2018-07-03 11:30:35,709 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - /opt/flink-1.5.0/conf 2018-07-03 11:30:35,709 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --executionMode 2018-07-03 11:30:35,709 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - cluster 2018-07-03 11:30:35,710 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --host 2018-07-03 11:30:35,710 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - cluster 2018-07-03 11:30:35,710 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Classpath: /opt/flink-1.5.0/lib/flink-cep_2.11-1.5.0.jar:/opt/flink-1.5.0/lib/flink-connectors-1.5.0.jar:/opt/flink-1.5.0/lib/flink-gelly_2.11-1.5.0.jar:/opt/flink-1.5.0/lib/flink-ml_2.11-1.5.0.jar:/opt/flink-1.5.0/lib/flink-python_2.11-1.5.0.jar:/opt/flink-1.5.0/lib/flink-table_2.11-1.5.0.jar:/opt/flink-1.5.0/lib/log4j-1.2.17.jar:/opt/flink-1.5.0/lib/slf4j-log4j12-1.7.7.jar:/opt/flink-1.5.0/lib/flink-dist_2.11-1.5.0.jar::: 2018-07-03 11:30:35,710 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -------------------------------------------------------------------------------- 2018-07-03 11:30:35,712 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT] 2018-07-03 11:30:35,720 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: blob.server.port, 4124 2018-07-03 11:30:35,720 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, myfl-flink-jobmanager 2018-07-03 11:30:35,720 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 4123 2018-07-03 11:30:35,721 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.mb, 1024 2018-07-03 11:30:35,721 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.mb, 1024 2018-07-03 11:30:35,721 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.rpc.port, 4122 2018-07-03 11:30:35,721 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.data.port, 4121 2018-07-03 11:30:35,721 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.query.port, 4125 2018-07-03 11:30:35,722 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.port, 8081 2018-07-03 11:30:35,762 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.address, myfl-flink-jobmanager-ui 2018-07-03 11:30:35,762 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.backend, rocksdb 2018-07-03 11:30:35,762 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.checkpoints.dir, file:///opt/flink/share/myfl-flink/checkpoints/ext_checkpoints 2018-07-03 11:30:35,763 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.backend.fs.checkpointdir, file:///opt/flink/share/myfl-flink/checkpoints/fs_state 2018-07-03 11:30:35,763 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.backend.rocksdb.checkpointdir, file:///opt/flink/share/myfl-flink/checkpoints/rocksdb_state 2018-07-03 11:30:35,763 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 4 2018-07-03 11:30:35,763 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: restart-strategy, fixed-delay 2018-07-03 11:30:35,764 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: restart-strategy.fixed-delay.attempts, 100 2018-07-03 11:30:35,764 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: restart-strategy.fixed-delay.delay, 1 s 2018-07-03 11:30:35,885 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting StandaloneSessionClusterEntrypoint. 2018-07-03 11:30:35,885 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem. 2018-07-03 11:30:35,892 INFO org.apache.flink.core.fs.FileSystem - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available. 2018-07-03 11:30:35,963 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install security context. 2018-07-03 11:30:35,970 INFO org.apache.flink.runtime.security.modules.HadoopModuleFactory - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath. 2018-07-03 11:30:35,988 INFO org.apache.flink.runtime.security.SecurityUtils - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath. 2018-07-03 11:30:35,989 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services. 2018-07-03 11:30:36,003 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start actor system at myfl-flink-jobmanager:4123 2018-07-03 11:30:37,288 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 2018-07-03 11:30:37,396 INFO akka.remote.Remoting - Starting remoting 2018-07-03 11:30:37,583 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@myfl-flink-jobmanager:4123] 2018-07-03 11:30:37,591 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system started at akka.tcp://flink@myfl-flink-jobmanager:4123 2018-07-03 11:30:37,611 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /tmp/blobStore-e445bc66-cee3-4a3d-b810-74df02627eca 2018-07-03 11:30:37,613 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:4124<http://0.0.0.0:4124> - max concurrent requests: 50 - max backlog: 1000 2018-07-03 11:30:37,629 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported. 2018-07-03 11:30:37,664 INFO org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore - Initializing FileArchivedExecutionGraphStore: Storage directory /tmp/executionGraphStore-4ff546b1-4bfb-4911-9314-89c61d7e7149, expiration time 3600000, maximum cache size 52428800 bytes. 2018-07-03 11:30:37,694 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /tmp/blobStore-7e0efdb8-f70b-42ed-9387-c0e1b8090b36 2018-07-03 11:30:37,702 WARN org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload directory /tmp/flink-web-e68a12b9-b9cc-4508-be00-4bf9f113afcd/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available. 2018-07-03 11:30:37,703 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Created directory /tmp/flink-web-e68a12b9-b9cc-4508-be00-4bf9f113afcd/flink-web-upload for file uploads. 2018-07-03 11:30:37,706 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Starting rest endpoint. 2018-07-03 11:30:38,369 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component log file: /opt/flink-1.5.0/log/flink--standalonesession-0-myfl-flink-jobmanager-7b4d8c4dd4-bv6zf.log 2018-07-03 11:30:38,369 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component stdout file: /opt/flink-1.5.0/log/flink--standalonesession-0-myfl-flink-jobmanager-7b4d8c4dd4-bv6zf.out 2018-07-03 11:30:38,567 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest endpoint listening at myfl-flink-jobmanager-ui:8081 2018-07-03 11:30:38,568 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - http://myfl-flink-jobmanager-ui:8081 was granted leadership with leaderSessionID=00000000-0000-0000-0000-000000000000 2018-07-03 11:30:38,568 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Web frontend listening at http://myfl-flink-jobmanager-ui:8081. 2018-07-03 11:30:38,578 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at akka://flink/user/resourcemanager . 2018-07-03 11:30:38,966 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/dispatcher . 2018-07-03 11:30:39,068 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - ResourceManager akka.tcp://flink@myfl-flink-jobmanager:4123/user/resourcemanager was granted leadership with fencing token 00000000000000000000000000000000 2018-07-03 11:30:39,069 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Starting the SlotManager. 2018-07-03 11:30:39,164 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Dispatcher akka.tcp://flink@myfl-flink-jobmanager:4123/user/dispatcher was granted leadership with fencing token 00000000000000000000000000000000 2018-07-03 11:30:39,165 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Recovering all persisted jobs. 2018-07-03 11:30:39,682 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Replacing old registration of TaskExecutor 068c693b9585900f68c53b00507ee889. 2018-07-03 11:30:39,683 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Unregister TaskManager 8a5cee3aa38081030dc8558ac477d3b3 from the SlotManager. 2018-07-03 11:30:39,683 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - The target with resource ID 068c693b9585900f68c53b00507ee889 is already been monitored. 2018-07-03 11:30:39,770 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Registering TaskManager 068c693b9585900f68c53b00507ee889 under 03d409e5166fad4f4082b6165eb0de2e at the SlotManager. 2018-07-03 11:34:20,257 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Submitting job b684656d9afd75cc384a7bcd071bf55e (CSV Files Read -> CSV to Avro encode -> Kafka publish). 2018-07-03 11:34:20,269 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/jobmanager_0 . 2018-07-03 11:34:20,278 INFO org.apache.flink.runtime.jobmaster.JobMaster - Initializing job CSV Files Read -> CSV to Avro encode -> Kafka publish (b684656d9afd75cc384a7bcd071bf55e). 2018-07-03 11:34:20,285 INFO org.apache.flink.runtime.jobmaster.JobMaster - Using restart strategy FixedDelayRestartStrategy(maxNumberRestartAttempts=2147483647, delayBetweenRestartAttempts=0) for CSV Files Read -> CSV to Avro encode -> Kafka publish (b684656d9afd75cc384a7bcd071bf55e). 2018-07-03 11:34:20,289 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.slotpool.SlotPool at akka://flink/user/67ceb2ae-1cb1-44be-a09e-601032e23fb5 . 2018-07-03 11:34:20,481 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job recovers via failover strategy: full graph restart 2018-07-03 11:34:20,562 INFO org.apache.flink.runtime.jobmaster.JobMaster - Running initialization on master for job CSV Files Read -> CSV to Avro encode -> Kafka publish (b684656d9afd75cc384a7bcd071bf55e). 2018-07-03 11:34:20,563 INFO org.apache.flink.runtime.jobmaster.JobMaster - Successfully ran initialization on master in 0 ms. 2018-07-03 11:34:20,580 INFO org.apache.flink.runtime.jobmaster.JobMaster - Loading state backend via factory org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory 2018-07-03 11:34:20,590 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'state.backend.rocksdb.checkpointdir' instead of proper key 'state.backend.rocksdb.localdir' 2018-07-03 11:34:20,592 ERROR org.apache.flink.runtime.rest.handler.job.JobSubmitHandler - Implementation error: Unhandled exception. org.apache.flink.util.FlinkException: Failed to submit job b684656d9afd75cc384a7bcd071bf55e. at org.apache.flink.runtime.dispatcher.Dispatcher.submitJob(Dispatcher.java:254) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:247) at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:162) at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:70) at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:142) at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.onReceive(FencedAkkaRpcActor.java:40) at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165) at akka.actor.Actor$class.aroundReceive(Actor.scala:502) at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) at akka.actor.ActorCell.invoke(ActorCell.scala:495) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) at akka.dispatch.Mailbox.run(Mailbox.scala:224) at akka.dispatch.Mailbox.exec(Mailbox.scala:234) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: org.apache.flink.runtime.client.JobExecutionException: Could not set up JobManager at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:169) at org.apache.flink.runtime.dispatcher.Dispatcher$DefaultJobManagerRunnerFactory.createJobManagerRunner(Dispatcher.java:885) at org.apache.flink.runtime.dispatcher.Dispatcher.createJobManagerRunner(Dispatcher.java:287) at org.apache.flink.runtime.dispatcher.Dispatcher.runJob(Dispatcher.java:277) at org.apache.flink.runtime.dispatcher.Dispatcher.persistAndRunJob(Dispatcher.java:262) at org.apache.flink.runtime.dispatcher.Dispatcher.submitJob(Dispatcher.java:249) ... 21 more Caused by: org.apache.flink.runtime.client.JobExecutionException: Could not instantiate configured state backend at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:308) at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:100) at org.apache.flink.runtime.jobmaster.JobMaster.createExecutionGraph(JobMaster.java:1150) at org.apache.flink.runtime.jobmaster.JobMaster.createAndRestoreExecutionGraph(JobMaster.java:1130) at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:298) at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:151) ... 26 more Caused by: org.apache.flink.configuration.IllegalConfigurationException: Invalid configuration for RocksDB state backend's local storage directories: Relative paths are not supported at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.<init>(RocksDBStateBackend.java:273) at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.configure(RocksDBStateBackend.java:296) at org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory.createFromConfig(RocksDBStateBackendFactory.java:47) at org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory.createFromConfig(RocksDBStateBackendFactory.java:32) at org.apache.flink.runtime.state.StateBackendLoader.loadStateBackendFromConfig(StateBackendLoader.java:157) at org.apache.flink.runtime.state.StateBackendLoader.fromApplicationOrConfigOrDefault(StateBackendLoader.java:222) at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:304) ... 31 more Caused by: java.lang.IllegalArgumentException: Relative paths are not supported at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.setDbStoragePaths(RocksDBStateBackend.java:518) at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.<init>(RocksDBStateBackend.java:269) ... 37 more On Tue, Jul 3, 2018 at 5:11 PM, Chesnay Schepler <ches...@apache.org<mailto:ches...@apache.org>> wrote: Doesn't sound like intended behavior, can you give us the stacktrace? On 03.07.2018 13:17, Data Engineer wrote: The Flink documentation says that we need to specify the filesystem type (file://<file:///>, hdfs://) when configuring the rocksdb backend dir. https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/state/state_backends.html#the-rocksdbstatebackend But when I do this, I get an error on job submission saying that relative paths are not permitted in the rocksdb stand backend. I am submitting the job via flink cli (bin/flink run). Also, even though I give a local file system path "file:///home/abc/share"<file:///home/abc/share>, it is a shared GlusterFS volume mount, so it will be accessible by the JobManager and all TaskManagers. I removed the filesystem type from the rocksdb backend dir configuration, and though the job got submitted, the rocksdb checkpoint directory was not created. I have enabled checkpointing in my Flink application. I am using Flink 1.5.0. Any help or pointers would be appreciated.
