I got some logs and stack traces from our backend storage. This is not the entire log though. Can this be useful? With these set of logs messages the job manager kept restarting.
Regards Ram On Mon, Sep 26, 2022 at 3:11 PM ramkrishna vasudevan < ramvasu.fl...@gmail.com> wrote: > Thank you very much for the reply. I have lost the k8s cluster in this > case before I could capture the logs. I will try to repro this and get back > to you. > > Regards > Ram > > On Mon, Sep 26, 2022 at 12:42 PM Matthias Pohl <matthias.p...@aiven.io> > wrote: > >> Hi Ramkrishna, >> thanks for reaching out to the Flink community. Could you share the >> JobManager logs to get a better understanding of what's going on? I'm >> wondering why the JobManager is failing when the actual problem is that the >> job is struggling to access a folder. It sounds like there are multiple >> problems here. >> >> Best, >> Matthias >> >> On Mon, Sep 26, 2022 at 6:25 AM ramkrishna vasudevan < >> ramvasu.fl...@gmail.com> wrote: >> >>> Hi all >>> >>> I have a simple job where we read for a given path in cloud storage to >>> watch for new files in a given fodler. While I setup my job there was some >>> permission issue on the folder. The job is STREAMING job. >>> The cluster is set in the session mode and is running on Kubernetes. >>> The job manager since then is failing to come back up and every time it >>> fails with the permission issue. But the point is how should i recover my >>> cluster in this case. Since JM is not there the UI is also not working and >>> how do I remove the bad job from the JM. >>> >>> Regards >>> Ram >>> >>
Message Message Starting execution of job WordCount (04ae99777ee2ed34c13fe8120e68436e) under job master id a6b99a98ea6483cedb49878b9af2465f. Starting split enumerator for source Source: file-input -> tokenizer. Closing SourceCoordinator for source Source: file-input -> tokenizer. Source coordinator for source Source: file-input -> tokenizer closed. Fatal error occurred in the cluster entrypoint. The RpcEndpoint jobmanager_2 failed. Enabling required built-in plugins Linking flink-azure-fs-hadoop-1.13.1-0.1.15.jar to plugin directory Successfully enabled flink-azure-fs-hadoop-1.13.1-0.1.15.jar sed: couldn't open temporary file /opt/flink/conf/sedWarRga: Read-only file system sed: couldn't open temporary file /opt/flink/conf/sed6VtlPa: Read-only file system /docker-entrypoint.sh: line 75: /opt/flink/conf/flink-conf.yaml: Permission denied sed: couldn't open temporary file /opt/flink/conf/sedWNUSh9: Read-only file system /docker-entrypoint.sh: line 90: /opt/flink/conf/flink-conf.yaml.tmp: Read-only file system Starting Job Manager Starting standalonesession as a console application on host 972ac3d8028e45fcafa9b8b7b7f1dafb-flink-jobmanager-56457c946f9f6. RESOURCE_PARAMS extraction logs: jvm_params: -Xmx1484783613 -Xms1484783613 -XX:MaxMetaspaceSize=268435456 dynamic_configs: -D jobmanager.memory.off-heap.size=134217728b -D jobmanager.memory.jvm-overhead.min=209715203b -D jobmanager.memory.jvm-metaspace.size=268435456b -D jobmanager.memory.heap.size=1484783613b -D jobmanager.memory.jvm-overhead.max=209715203b logs: WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance. INFO [] - Loading configuration property: blob.server.port, 6124 INFO [] - Loading configuration property: state.checkpoints.num-retained, 5 INFO [] - Loading configuration property: historyserver.web.port, 8082 INFO [] - Loading configuration property: rest.flamegraph.enabled, true INFO [] - Loading configuration property: jobmanager.rpc.address, flink-jobmanager-service INFO [] - Loading configuration property: fs.azure.account.auth.type, OAuth INFO [] - Loading configuration property: fs.azure.account.oauth.provider.type, org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider INFO [] - Loading configuration property: fs.azure.account.oauth2.client.id, 4fb013ac-9f6e-40f7-823b-721589d98db8 INFO [] - Loading configuration property: state.savepoints.dir, abfs://vrppestor...@vrppestore.dfs.core.windows.net/972ac3d8028e45fcafa9b8b7b7f1dafb/savepoints INFO [] - Loading configuration property: fs.azure.identity.transformer.service.principal.substitution.list, '*' INFO [] - Loading configuration property: kubernetes.cluster-id, flink-972ac3d8028e45fcafa9b8b7b7f1dafb INFO [] - Loading configuration property: high-availability.storageDir, abfs://vrppestor...@vrppestore.dfs.core.windows.net/972ac3d8028e45fcafa9b8b7b7f1dafb/ha INFO [] - Loading configuration property: parallelism.default, 2 INFO [] - Loading configuration property: kubernetes.namespace, 972ac3d8028e45fcafa9b8b7b7f1dafb INFO [] - Loading configuration property: taskmanager.numberOfTaskSlots, 2 INFO [] - Loading configuration property: metrics.reporters, prom INFO [] - Loading configuration property: metrics.reporter.prom.class, org.apache.flink.metrics.prometheus.PrometheusReporter INFO [] - Loading configuration property: historyserver.archive.fs.dir, abfs://vrppestor...@vrppestore.dfs.core.windows.net/completed-jobs INFO [] - Loading configuration property: taskmanager.memory.process.size, 2000mb INFO [] - Loading configuration property: jobmanager.archive.fs.dir, abfs://vrppestor...@vrppestore.dfs.core.windows.net/completed-jobs INFO [] - Loading configuration property: state.backend.incremental, true INFO [] - Loading configuration property: jobmanager.memory.process.size, 2000mb INFO [] - Loading configuration property: historyserver.archive.fs.refresh-interval, 10000 INFO [] - Loading configuration property: jobmanager.rpc.port, 6123 INFO [] - Loading configuration property: taskmanager.rpc.port, 6122 INFO [] - Loading configuration property: fs.azure.identity.transformer.service.principal.id, b8fd29c0-507d-4d46-a1d7-aef1deb44258 INFO [] - Loading configuration property: execution.checkpointing.interval, 60000 INFO [] - Loading configuration property: fs.azure.account.oauth2.msi.tenant, 72f988bf-86f1-41af-91ab-2d7cd011db47 INFO [] - Loading configuration property: env.log.dir, /opt/flink/log INFO [] - Loading configuration property: queryable-state.proxy.ports, 6125 INFO [] - Loading configuration property: high-availability, org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory INFO [] - Loading configuration property: taskmanager.memory.managed.size, 1000 INFO [] - Loading configuration property: state.backend, rocksdb INFO [] - Loading configuration property: state.checkpoints.dir, abfs://vrppestor...@vrppestore.dfs.core.windows.net/972ac3d8028e45fcafa9b8b7b7f1dafb/checkpoints INFO [] - Final Master Memory configuration: INFO [] - Total Process Memory: 1.953gb (2097152000 bytes) INFO [] - Total Flink Memory: 1.508gb (1619001341 bytes) INFO [] - JVM Heap: 1.383gb (1484783613 bytes) INFO [] - Off-heap: 128.000mb (134217728 bytes) INFO [] - JVM Metaspace: 256.000mb (268435456 bytes) INFO [] - JVM Overhead: 200.000mb (209715203 bytes) org.apache.flink.util.FlinkException: JobMaster for job 04ae99777ee2ed34c13fe8120e68436e failed. at org.apache.flink.runtime.dispatcher.Dispatcher.jobMasterFailed(Dispatcher.java:873) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.dispatcher.Dispatcher.jobManagerRunnerFailed(Dispatcher.java:459) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$runJob$3(Dispatcher.java:418) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$Completion.run(Unknown Source) ~[?:?] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:440) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:208) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.actor.Actor.aroundReceive(Actor.scala:517) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.actor.Actor.aroundReceive$(Actor.scala:515) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.actor.ActorCell.invoke(ActorCell.scala:561) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.dispatch.Mailbox.run(Mailbox.scala:225) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] Caused by: org.apache.flink.runtime.jobmaster.JobMasterException: Could not start the JobMaster. at org.apache.flink.runtime.jobmaster.JobMaster.onStart(JobMaster.java:385) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStart(RpcEndpoint.java:181) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.start(AkkaRpcActor.java:605) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:180) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] ... 13 more Caused by: org.apache.flink.util.FlinkRuntimeException: Failed to start the operator coordinators at org.apache.flink.runtime.scheduler.DefaultOperatorCoordinatorHandler.startAllOperatorCoordinators(DefaultOperatorCoordinatorHandler.java:90) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.scheduler.SchedulerBase.startScheduling(SchedulerBase.java:592) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.jobmaster.JobMaster.startScheduling(JobMaster.java:955) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.jobmaster.JobMaster.startJobExecution(JobMaster.java:873) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.jobmaster.JobMaster.onStart(JobMaster.java:383) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStart(RpcEndpoint.java:181) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.start(AkkaRpcActor.java:605) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:180) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] ... 13 more org.apache.flink.runtime.rpc.akka.exceptions.AkkaRpcException: Could not start RpcEndpoint jobmanager_2. at org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.start(AkkaRpcActor.java:610) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:180) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.actor.Actor.aroundReceive(Actor.scala:517) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.actor.Actor.aroundReceive$(Actor.scala:515) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.dispatch.Mailbox.run(Mailbox.scala:225) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] Caused by: org.apache.flink.runtime.jobmaster.JobMasterException: Could not start the JobMaster. at org.apache.flink.runtime.jobmaster.JobMaster.onStart(JobMaster.java:385) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStart(RpcEndpoint.java:181) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.start(AkkaRpcActor.java:605) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] ... 20 more Caused by: org.apache.flink.util.FlinkRuntimeException: Failed to start the operator coordinators at org.apache.flink.runtime.scheduler.DefaultOperatorCoordinatorHandler.startAllOperatorCoordinators(DefaultOperatorCoordinatorHandler.java:90) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.scheduler.SchedulerBase.startScheduling(SchedulerBase.java:592) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.jobmaster.JobMaster.startScheduling(JobMaster.java:955) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.jobmaster.JobMaster.startJobExecution(JobMaster.java:873) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.jobmaster.JobMaster.onStart(JobMaster.java:383) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStart(RpcEndpoint.java:181) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.start(AkkaRpcActor.java:605) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] ... 20 more Caused by: org.apache.flink.util.FlinkRuntimeException: Could not enumerate file splits at org.apache.flink.connector.file.src.AbstractFileSource.createEnumerator(AbstractFileSource.java:144) ~[flink-table-blink_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.source.coordinator.SourceCoordinator.start(SourceCoordinator.java:124) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator$DeferrableCoordinator.applyCall(RecreateOnResetOperatorCoordinator.java:291) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator.start(RecreateOnResetOperatorCoordinator.java:70) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder.start(OperatorCoordinatorHolder.java:194) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.scheduler.DefaultOperatorCoordinatorHandler.startAllOperatorCoordinators(DefaultOperatorCoordinatorHandler.java:85) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.scheduler.SchedulerBase.startScheduling(SchedulerBase.java:592) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.jobmaster.JobMaster.startScheduling(JobMaster.java:955) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.jobmaster.JobMaster.startJobExecution(JobMaster.java:873) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.jobmaster.JobMaster.onStart(JobMaster.java:383) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStart(RpcEndpoint.java:181) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.start(AkkaRpcActor.java:605) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] ... 20 more Caused by: java.io.FileNotFoundException: File x...@yyyy.dfs.core.windows.net/wordcountdata does not exist or the user running Flink ('flink') has insufficient permissions to access it. at org.apache.flink.core.fs.local.LocalFileSystem.getFileStatus(LocalFileSystem.java:106) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.connector.file.src.enumerate.NonSplittingRecursiveEnumerator.enumerateSplits(NonSplittingRecursiveEnumerator.java:83) ~[flink-table-blink_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.connector.file.src.AbstractFileSource.createEnumerator(AbstractFileSource.java:142) ~[flink-table-blink_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.source.coordinator.SourceCoordinator.start(SourceCoordinator.java:124) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator$DeferrableCoordinator.applyCall(RecreateOnResetOperatorCoordinator.java:291) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator.start(RecreateOnResetOperatorCoordinator.java:70) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder.start(OperatorCoordinatorHolder.java:194) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.scheduler.DefaultOperatorCoordinatorHandler.startAllOperatorCoordinators(DefaultOperatorCoordinatorHandler.java:85) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.scheduler.SchedulerBase.startScheduling(SchedulerBase.java:592) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.jobmaster.JobMaster.startScheduling(JobMaster.java:955) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.jobmaster.JobMaster.startJobExecution(JobMaster.java:873) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.jobmaster.JobMaster.onStart(JobMaster.java:383) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStart(RpcEndpoint.java:181) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.start(AkkaRpcActor.java:605) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] ... 20 more org.apache.flink.util.FlinkException: JobMaster for job 04ae99777ee2ed34c13fe8120e68436e failed. at org.apache.flink.runtime.dispatcher.Dispatcher.jobMasterFailed(Dispatcher.java:873) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.dispatcher.Dispatcher.jobManagerRunnerFailed(Dispatcher.java:459) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$runJob$3(Dispatcher.java:418) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$Completion.run(Unknown Source) ~[?:?] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:440) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:208) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.actor.Actor.aroundReceive(Actor.scala:517) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.actor.Actor.aroundReceive$(Actor.scala:515) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.actor.ActorCell.invoke(ActorCell.scala:561) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.dispatch.Mailbox.run(Mailbox.scala:225) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] Caused by: org.apache.flink.runtime.jobmaster.JobMasterException: Could not start the JobMaster. at org.apache.flink.runtime.jobmaster.JobMaster.onStart(JobMaster.java:385) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStart(RpcEndpoint.java:181) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.start(AkkaRpcActor.java:605) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:180) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] ... 13 more Caused by: org.apache.flink.util.FlinkRuntimeException: Failed to start the operator coordinators at org.apache.flink.runtime.scheduler.DefaultOperatorCoordinatorHandler.startAllOperatorCoordinators(DefaultOperatorCoordinatorHandler.java:90) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.scheduler.SchedulerBase.startScheduling(SchedulerBase.java:592) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.jobmaster.JobMaster.startScheduling(JobMaster.java:955) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.jobmaster.JobMaster.startJobExecution(JobMaster.java:873) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.jobmaster.JobMaster.onStart(JobMaster.java:383) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStart(RpcEndpoint.java:181) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.start(AkkaRpcActor.java:605) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:180) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] ... 13 more Caused by: org.apache.flink.util.FlinkRuntimeException: Could not enumerate file splits at org.apache.flink.connector.file.src.AbstractFileSource.createEnumerator(AbstractFileSource.java:144) ~[flink-table-blink_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.source.coordinator.SourceCoordinator.start(SourceCoordinator.java:124) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator$DeferrableCoordinator.applyCall(RecreateOnResetOperatorCoordinator.java:291) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator.start(RecreateOnResetOperatorCoordinator.java:70) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder.start(OperatorCoordinatorHolder.java:194) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.scheduler.DefaultOperatorCoordinatorHandler.startAllOperatorCoordinators(DefaultOperatorCoordinatorHandler.java:85) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.scheduler.SchedulerBase.startScheduling(SchedulerBase.java:592) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.jobmaster.JobMaster.startScheduling(JobMaster.java:955) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.jobmaster.JobMaster.startJobExecution(JobMaster.java:873) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.jobmaster.JobMaster.onStart(JobMaster.java:383) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStart(RpcEndpoint.java:181) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.start(AkkaRpcActor.java:605) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:180) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] ... 13 more Caused by: java.io.FileNotFoundException: File x...@vrppestore.dfs.core.windows.net/wordcountdata does not exist or the user running Flink ('flink') has insufficient permissions to access it. at org.apache.flink.core.fs.local.LocalFileSystem.getFileStatus(LocalFileSystem.java:106) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.connector.file.src.enumerate.NonSplittingRecursiveEnumerator.enumerateSplits(NonSplittingRecursiveEnumerator.java:83) ~[flink-table-blink_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.connector.file.src.AbstractFileSource.createEnumerator(AbstractFileSource.java:142) ~[flink-table-blink_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.source.coordinator.SourceCoordinator.start(SourceCoordinator.java:124) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator$DeferrableCoordinator.applyCall(RecreateOnResetOperatorCoordinator.java:291) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator.start(RecreateOnResetOperatorCoordinator.java:70) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder.start(OperatorCoordinatorHolder.java:194) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.scheduler.DefaultOperatorCoordinatorHandler.startAllOperatorCoordinators(DefaultOperatorCoordinatorHandler.java:85) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.scheduler.SchedulerBase.startScheduling(SchedulerBase.java:592) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.jobmaster.JobMaster.startScheduling(JobMaster.java:955) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.jobmaster.JobMaster.startJobExecution(JobMaster.java:873) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.jobmaster.JobMaster.onStart(JobMaster.java:383) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStart(RpcEndpoint.java:181) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.start(AkkaRpcActor.java:605) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:180) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.12-1.13.1-0.1.15.jar:1.13.1-0.1.15] ... 13 more 2022-09-23 10:37:43.376 [] StandaloneSessionClusterEntrypoint shutdown hook INFO flink apache.flink.runtime.entrypoint.ClusterEntrypoint 481 Shutting StandaloneSessionClusterEntrypoint down with application status UNKNOWN. Diagnostics Cluster entrypoint has been closed externally..