[
https://issues.apache.org/jira/browse/FLINK-19067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17189861#comment-17189861
]
JieFang.He edited comment on FLINK-19067 at 9/7/20, 3:00 AM:
-------------------------------------------------------------
[~rmetzger]
After testing again, this should have nothing to do with standby JobManager,
When it happens，all node submit job fail. The last time it failed only on the
Standby node seems to be accidental.
Also, Not every time you submit a job will fail, failure is probabilistic.
The log shows that there is no blob file on the server, but the wordcount
example seems no need for these files, the successful job did not generate
these files.
My resource_manager_lock is on Standby JobManager, Others on the Active
JobManager, I don't know if this has any impact.
{code:java}
2020-09-03 19:15:11,566 INFO [main-SendThread(deployer-hejiefang03:2181)]
[ClientCnxn]: Socket connection established, initiating session, client:
/172.17.0.8:45164, server: deployer-hejiefang03/172.17.0.9:2181
2020-09-03 19:15:11,573 INFO [main-SendThread(deployer-hejiefang03:2181)]
[ClientCnxn]: Session establishment complete on server
deployer-hejiefang03/172.17.0.9:2181, sessionid = 0x3000e154ba9000a, negotiated
timeout = 60000
2020-09-03 19:15:11,573 INFO [main-EventThread] [ConnectionStateManager]: State
change: CONNECTED
2020-09-03 19:15:11,578 INFO [main-EventThread] [EnsembleTracker]: New config
event received: {server.1=deployer-hejiefang01:2888:3888:participant,
version=0, server.3=deployer-hejiefang03:2888:3888:participant,
server.2=deployer-hejiefang02:2888:3888:participant}
2020-09-03 19:15:11,578 ERROR [main-EventThread] [EnsembleTracker]: Invalid
config event received: {server.1=deployer-hejiefang01:2888:3888:participant,
version=0, server.3=deployer-hejiefang03:2888:3888:participant,
server.2=deployer-hejiefang02:2888:3888:participant}
2020-09-03 19:15:11,579 INFO [main-EventThread] [EnsembleTracker]: New config
event received: {server.1=deployer-hejiefang01:2888:3888:participant,
version=0, server.3=deployer-hejiefang03:2888:3888:participant,
server.2=deployer-hejiefang02:2888:3888:participant}
2020-09-03 19:15:11,579 ERROR [main-EventThread] [EnsembleTracker]: Invalid
config event received: {server.1=deployer-hejiefang01:2888:3888:participant,
version=0, server.3=deployer-hejiefang03:2888:3888:participant,
server.2=deployer-hejiefang02:2888:3888:participant}
2020-09-03 19:15:11,668 INFO [Curator-Framework-0] [CuratorFrameworkImpl]:
backgroundOperationsLoop exiting
2020-09-03 19:15:11,772 INFO [ForkJoinPool.commonPool-worker-57] [ZooKeeper]:
Session: 0x3000e154ba9000a closed
2020-09-03 19:15:11,772 INFO [main-EventThread] [ClientCnxn]: EventThread shut
down for session: 0x3000e154ba9000a
2020-09-03 19:15:11,775 ERROR [main] [CliFrontend]: Error while running the
command.
org.apache.flink.client.program.ProgramInvocationException: The main method
caused an error: org.apache.flink.client.program.ProgramInvocationException:
Job failed (JobID: 8b242015d603f46e82a5f1299399b669)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:302)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:149)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:699)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:232)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:916)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:992)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_163]
at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_163]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
[flink-shaded-hadoop-3-uber-3.2.1-11.0.jar:?]
at
org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
[flink-dist_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:992)
[flink-dist_2.11-1.11.1.jar:1.11.1]
Caused by: java.util.concurrent.ExecutionException:
org.apache.flink.client.program.ProgramInvocationException: Job failed (JobID:
8b242015d603f46e82a5f1299399b669)
at
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
~[?:1.8.0_163]
at
org.apache.flink.client.program.ContextEnvironment.getJobExecutionResult(ContextEnvironment.java:112)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:76)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:873)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.api.java.DataSet.collect(DataSet.java:413)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.api.java.DataSet.print(DataSet.java:1652)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:96)
~[?:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_163]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_163]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_163]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_163]
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:288)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
... 11 more
Caused by: org.apache.flink.client.program.ProgramInvocationException: Job
failed (JobID: 8b242015d603f46e82a5f1299399b669)
at
org.apache.flink.client.deployment.ClusterClientJobClientAdapter.lambda$null$6(ClusterClientJobClientAdapter.java:116)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
~[?:1.8.0_163]
at
org.apache.flink.client.program.rest.RestClusterClient.lambda$pollResourceAsync$22(RestClusterClient.java:602)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
~[?:1.8.0_163]
at
org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$8(FutureUtils.java:309)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:561)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:929)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
~[?:1.8.0_163]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_163]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_163]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_163]
Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution
failed.
at
org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:147)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.client.deployment.ClusterClientJobClientAdapter.lambda$null$6(ClusterClientJobClientAdapter.java:114)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
~[?:1.8.0_163]
at
org.apache.flink.client.program.rest.RestClusterClient.lambda$pollResourceAsync$22(RestClusterClient.java:602)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
~[?:1.8.0_163]
at
org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$8(FutureUtils.java:309)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:561)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:929)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
~[?:1.8.0_163]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_163]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_163]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_163]
Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by
NoRestartBackoffTimeStrategy
at
org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:116)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:78)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:192)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:185)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:179)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:503)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:386)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_163]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_163]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_163]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_163]
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:284)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:199)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at akka.actor.ActorCell.invoke(ActorCell.scala:561)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
Caused by: java.io.IOException: Failed to fetch BLOB
8b242015d603f46e82a5f1299399b669/p-02e225df937959b03b8d26238f027e81191d1e2a-1535900a0910d61d04f56133aceb60c1
from deployer-hejiefang01/172.17.0.8:30419 and store it under
/data2/flink/tmp/blobStore-caaf1bc1-3d37-4e69-9ecd-b6cf51974c73/incoming/temp-00000001
at
org.apache.flink.runtime.blob.BlobClient.downloadFromBlobServer(BlobClient.java:169)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.AbstractBlobCache.getFileInternal(AbstractBlobCache.java:185)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.PermanentBlobCache.getFile(PermanentBlobCache.java:202)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.createUserCodeClassLoader(BlobLibraryCacheManager.java:234)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.getOrResolveClassLoader(BlobLibraryCacheManager.java:218)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.access$1200(BlobLibraryCacheManager.java:193)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$DefaultClassLoaderLease.getOrResolveClassLoader(BlobLibraryCacheManager.java:312)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:929)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:609)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_163]
Caused by: java.io.IOException: GET operation failed: Server side error:
/data2/flink/storageDir/default/blob/job_8b242015d603f46e82a5f1299399b669/blob_p-02e225df937959b03b8d26238f027e81191d1e2a-1535900a0910d61d04f56133aceb60c1
(No such file or directory)
at
org.apache.flink.runtime.blob.BlobClient.getInternal(BlobClient.java:231)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.BlobClient.downloadFromBlobServer(BlobClient.java:144)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.AbstractBlobCache.getFileInternal(AbstractBlobCache.java:185)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.PermanentBlobCache.getFile(PermanentBlobCache.java:202)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.createUserCodeClassLoader(BlobLibraryCacheManager.java:234)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.getOrResolveClassLoader(BlobLibraryCacheManager.java:218)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.access$1200(BlobLibraryCacheManager.java:193)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$DefaultClassLoaderLease.getOrResolveClassLoader(BlobLibraryCacheManager.java:312)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:929)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:609)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_163]
Caused by: java.io.IOException: Server side error:
/data2/flink/storageDir/default/blob/job_8b242015d603f46e82a5f1299399b669/blob_p-02e225df937959b03b8d26238f027e81191d1e2a-1535900a0910d61d04f56133aceb60c1
(No such file or directory)
at
org.apache.flink.runtime.blob.BlobClient.receiveAndCheckGetResponse(BlobClient.java:284)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.BlobClient.getInternal(BlobClient.java:225)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.BlobClient.downloadFromBlobServer(BlobClient.java:144)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.AbstractBlobCache.getFileInternal(AbstractBlobCache.java:185)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.PermanentBlobCache.getFile(PermanentBlobCache.java:202)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.createUserCodeClassLoader(BlobLibraryCacheManager.java:234)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.getOrResolveClassLoader(BlobLibraryCacheManager.java:218)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.access$1200(BlobLibraryCacheManager.java:193)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$DefaultClassLoaderLease.getOrResolveClassLoader(BlobLibraryCacheManager.java:312)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:929)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:609)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_163]
Caused by: java.io.FileNotFoundException:
/data2/flink/storageDir/default/blob/job_8b242015d603f46e82a5f1299399b669/blob_p-02e225df937959b03b8d26238f027e81191d1e2a-1535900a0910d61d04f56133aceb60c1
(No such file or directory)
at java.io.FileInputStream.open0(Native Method) ~[?:1.8.0_163]
at java.io.FileInputStream.open(FileInputStream.java:195) ~[?:1.8.0_163]
at java.io.FileInputStream.<init>(FileInputStream.java:138)
~[?:1.8.0_163]
at
org.apache.flink.core.fs.local.LocalDataInputStream.<init>(LocalDataInputStream.java:50)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.core.fs.local.LocalFileSystem.open(LocalFileSystem.java:143)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.FileSystemBlobStore.get(FileSystemBlobStore.java:105)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.FileSystemBlobStore.get(FileSystemBlobStore.java:87)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:501)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
{code}
was (Author: hejiefang):
[~rmetzger]
After testing again, this should have nothing to do with standby JobManager,
When it happens，all node submit job fail. The last time it failed only on the
Standby node seems to be accidental.
Also, Not every time you submit a job will fail, failure is probabilistic.
The log shows that there is no blob file on the server, but the wordcount
example seems no need for these files, the successful job did not generate
these files.
Also, if set --output to HDFS, it never fail.
My resource_manager_lock is on Standby JobManager, Others on the Active
JobManager, I don't know if this has any impact.
{code:java}
2020-09-03 19:15:11,566 INFO [main-SendThread(deployer-hejiefang03:2181)]
[ClientCnxn]: Socket connection established, initiating session, client:
/172.17.0.8:45164, server: deployer-hejiefang03/172.17.0.9:2181
2020-09-03 19:15:11,573 INFO [main-SendThread(deployer-hejiefang03:2181)]
[ClientCnxn]: Session establishment complete on server
deployer-hejiefang03/172.17.0.9:2181, sessionid = 0x3000e154ba9000a, negotiated
timeout = 60000
2020-09-03 19:15:11,573 INFO [main-EventThread] [ConnectionStateManager]: State
change: CONNECTED
2020-09-03 19:15:11,578 INFO [main-EventThread] [EnsembleTracker]: New config
event received: {server.1=deployer-hejiefang01:2888:3888:participant,
version=0, server.3=deployer-hejiefang03:2888:3888:participant,
server.2=deployer-hejiefang02:2888:3888:participant}
2020-09-03 19:15:11,578 ERROR [main-EventThread] [EnsembleTracker]: Invalid
config event received: {server.1=deployer-hejiefang01:2888:3888:participant,
version=0, server.3=deployer-hejiefang03:2888:3888:participant,
server.2=deployer-hejiefang02:2888:3888:participant}
2020-09-03 19:15:11,579 INFO [main-EventThread] [EnsembleTracker]: New config
event received: {server.1=deployer-hejiefang01:2888:3888:participant,
version=0, server.3=deployer-hejiefang03:2888:3888:participant,
server.2=deployer-hejiefang02:2888:3888:participant}
2020-09-03 19:15:11,579 ERROR [main-EventThread] [EnsembleTracker]: Invalid
config event received: {server.1=deployer-hejiefang01:2888:3888:participant,
version=0, server.3=deployer-hejiefang03:2888:3888:participant,
server.2=deployer-hejiefang02:2888:3888:participant}
2020-09-03 19:15:11,668 INFO [Curator-Framework-0] [CuratorFrameworkImpl]:
backgroundOperationsLoop exiting
2020-09-03 19:15:11,772 INFO [ForkJoinPool.commonPool-worker-57] [ZooKeeper]:
Session: 0x3000e154ba9000a closed
2020-09-03 19:15:11,772 INFO [main-EventThread] [ClientCnxn]: EventThread shut
down for session: 0x3000e154ba9000a
2020-09-03 19:15:11,775 ERROR [main] [CliFrontend]: Error while running the
command.
org.apache.flink.client.program.ProgramInvocationException: The main method
caused an error: org.apache.flink.client.program.ProgramInvocationException:
Job failed (JobID: 8b242015d603f46e82a5f1299399b669)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:302)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:149)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:699)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:232)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:916)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:992)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_163]
at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_163]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
[flink-shaded-hadoop-3-uber-3.2.1-11.0.jar:?]
at
org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
[flink-dist_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:992)
[flink-dist_2.11-1.11.1.jar:1.11.1]
Caused by: java.util.concurrent.ExecutionException:
org.apache.flink.client.program.ProgramInvocationException: Job failed (JobID:
8b242015d603f46e82a5f1299399b669)
at
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
~[?:1.8.0_163]
at
org.apache.flink.client.program.ContextEnvironment.getJobExecutionResult(ContextEnvironment.java:112)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:76)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:873)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.api.java.DataSet.collect(DataSet.java:413)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.api.java.DataSet.print(DataSet.java:1652)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:96)
~[?:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_163]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_163]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_163]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_163]
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:288)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
... 11 more
Caused by: org.apache.flink.client.program.ProgramInvocationException: Job
failed (JobID: 8b242015d603f46e82a5f1299399b669)
at
org.apache.flink.client.deployment.ClusterClientJobClientAdapter.lambda$null$6(ClusterClientJobClientAdapter.java:116)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
~[?:1.8.0_163]
at
org.apache.flink.client.program.rest.RestClusterClient.lambda$pollResourceAsync$22(RestClusterClient.java:602)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
~[?:1.8.0_163]
at
org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$8(FutureUtils.java:309)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:561)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:929)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
~[?:1.8.0_163]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_163]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_163]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_163]
Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution
failed.
at
org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:147)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.client.deployment.ClusterClientJobClientAdapter.lambda$null$6(ClusterClientJobClientAdapter.java:114)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
~[?:1.8.0_163]
at
org.apache.flink.client.program.rest.RestClusterClient.lambda$pollResourceAsync$22(RestClusterClient.java:602)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
~[?:1.8.0_163]
at
org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$8(FutureUtils.java:309)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:561)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:929)
~[?:1.8.0_163]
at
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
~[?:1.8.0_163]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_163]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_163]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_163]
Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by
NoRestartBackoffTimeStrategy
at
org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:116)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:78)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:192)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:185)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:179)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:503)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:386)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_163]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_163]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_163]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_163]
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:284)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:199)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at akka.actor.ActorCell.invoke(ActorCell.scala:561)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
Caused by: java.io.IOException: Failed to fetch BLOB
8b242015d603f46e82a5f1299399b669/p-02e225df937959b03b8d26238f027e81191d1e2a-1535900a0910d61d04f56133aceb60c1
from deployer-hejiefang01/172.17.0.8:30419 and store it under
/data2/flink/tmp/blobStore-caaf1bc1-3d37-4e69-9ecd-b6cf51974c73/incoming/temp-00000001
at
org.apache.flink.runtime.blob.BlobClient.downloadFromBlobServer(BlobClient.java:169)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.AbstractBlobCache.getFileInternal(AbstractBlobCache.java:185)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.PermanentBlobCache.getFile(PermanentBlobCache.java:202)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.createUserCodeClassLoader(BlobLibraryCacheManager.java:234)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.getOrResolveClassLoader(BlobLibraryCacheManager.java:218)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.access$1200(BlobLibraryCacheManager.java:193)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$DefaultClassLoaderLease.getOrResolveClassLoader(BlobLibraryCacheManager.java:312)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:929)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:609)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_163]
Caused by: java.io.IOException: GET operation failed: Server side error:
/data2/flink/storageDir/default/blob/job_8b242015d603f46e82a5f1299399b669/blob_p-02e225df937959b03b8d26238f027e81191d1e2a-1535900a0910d61d04f56133aceb60c1
(No such file or directory)
at
org.apache.flink.runtime.blob.BlobClient.getInternal(BlobClient.java:231)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.BlobClient.downloadFromBlobServer(BlobClient.java:144)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.AbstractBlobCache.getFileInternal(AbstractBlobCache.java:185)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.PermanentBlobCache.getFile(PermanentBlobCache.java:202)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.createUserCodeClassLoader(BlobLibraryCacheManager.java:234)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.getOrResolveClassLoader(BlobLibraryCacheManager.java:218)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.access$1200(BlobLibraryCacheManager.java:193)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$DefaultClassLoaderLease.getOrResolveClassLoader(BlobLibraryCacheManager.java:312)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:929)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:609)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_163]
Caused by: java.io.IOException: Server side error:
/data2/flink/storageDir/default/blob/job_8b242015d603f46e82a5f1299399b669/blob_p-02e225df937959b03b8d26238f027e81191d1e2a-1535900a0910d61d04f56133aceb60c1
(No such file or directory)
at
org.apache.flink.runtime.blob.BlobClient.receiveAndCheckGetResponse(BlobClient.java:284)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.BlobClient.getInternal(BlobClient.java:225)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.BlobClient.downloadFromBlobServer(BlobClient.java:144)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.AbstractBlobCache.getFileInternal(AbstractBlobCache.java:185)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.PermanentBlobCache.getFile(PermanentBlobCache.java:202)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.createUserCodeClassLoader(BlobLibraryCacheManager.java:234)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.getOrResolveClassLoader(BlobLibraryCacheManager.java:218)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$LibraryCacheEntry.access$1200(BlobLibraryCacheManager.java:193)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager$DefaultClassLoaderLease.getOrResolveClassLoader(BlobLibraryCacheManager.java:312)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:929)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:609)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_163]
Caused by: java.io.FileNotFoundException:
/data2/flink/storageDir/default/blob/job_8b242015d603f46e82a5f1299399b669/blob_p-02e225df937959b03b8d26238f027e81191d1e2a-1535900a0910d61d04f56133aceb60c1
(No such file or directory)
at java.io.FileInputStream.open0(Native Method) ~[?:1.8.0_163]
at java.io.FileInputStream.open(FileInputStream.java:195) ~[?:1.8.0_163]
at java.io.FileInputStream.<init>(FileInputStream.java:138)
~[?:1.8.0_163]
at
org.apache.flink.core.fs.local.LocalDataInputStream.<init>(LocalDataInputStream.java:50)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.core.fs.local.LocalFileSystem.open(LocalFileSystem.java:143)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.FileSystemBlobStore.get(FileSystemBlobStore.java:105)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.FileSystemBlobStore.get(FileSystemBlobStore.java:87)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:501)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117)
~[flink-dist_2.11-1.11.1.jar:1.11.1]
{code}
> FileNotFoundException when run flink examples on standby JobManager
> -------------------------------------------------------------------
>
> Key: FLINK-19067
> URL: https://issues.apache.org/jira/browse/FLINK-19067
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.11.1
> Reporter: JieFang.He
> Priority: Major
>
> 1、When run examples/batch/WordCount.jar on standby JobManager，it will fail
> with the exception:
> Caused by: java.io.FileNotFoundException:
> /data2/storageDir/default/blob/job_d29414828f614d5466e239be4d3889ac/blob_p-a2ebe1c5aa160595f214b4bd0f39d80e42ee2e93-f458f1c12dc023e78d25f191de1d7c4b
> (No such file or directory)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.<init>(FileInputStream.java:138)
> at
> org.apache.flink.core.fs.local.LocalDataInputStream.<init>(LocalDataInputStream.java:50)
> at
> org.apache.flink.core.fs.local.LocalFileSystem.open(LocalFileSystem.java:143)
> at
> org.apache.flink.runtime.blob.FileSystemBlobStore.get(FileSystemBlobStore.java:105)
> at
> org.apache.flink.runtime.blob.FileSystemBlobStore.get(FileSystemBlobStore.java:87)
> at
> org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:501)
> at
> org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231)
> at
> org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117)
>
> 2、Run examples success on other nodes
> 3、After run success on the other node, it can run success on the Standby
> JobManager. But run again will fail
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
