Hi @all,
i have a yarn cluster with 5 Nodes with a running flink (0.10.2)
instance. Today we shut down one of the Yarn-Hosts due to maintance
reasons. After the restart we have some flink streaming routes in a
restarting status (see stacktrace below). Now I want to restart these
routes to continue their work from the last checkpoint. What can i do?
Greets
Dominique
Stacktrace
===================================================================================
java.io.IOException: Cannot get library with hash
8f15fe4a8137ca2f9fb348ec634f3703f4fd7317
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerReferenceToBlobKeyAndGetURL(BlobLibraryCacheManager.java:254)
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerTask(BlobLibraryCacheManager.java:114)
at
org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:710)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:471)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed to fetch BLOB
8f15fe4a8137ca2f9fb348ec634f3703f4fd7317 from /10.24.20.14:60485 and store it
under
/tmp/blobStore-efdeddf9-d096-440f-a4cb-9c79334ff92c/cache/blob_8f15fe4a8137ca2f9fb348ec634f3703f4fd7317
at org.apache.flink.runtime.blob.BlobCache.getURL(BlobCache.java:177)
at
org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerReferenceToBlobKeyAndGetURL(BlobLibraryCacheManager.java:245)
... 4 more
Caused by: java.io.IOException: GET operation failed: Server side error: Cannot
find required BLOB at
/tmp/blobStore-0f9a63e3-5700-4d47-aea7-310506c1496c/cache/blob_8f15fe4a8137ca2f9fb348ec634f3703f4fd7317
at org.apache.flink.runtime.blob.BlobClient.get(BlobClient.java:165)
at org.apache.flink.runtime.blob.BlobCache.getURL(BlobCache.java:125)
... 5 more
Caused by: java.io.IOException: Server side error: Cannot find required BLOB at
/tmp/blobStore-0f9a63e3-5700-4d47-aea7-310506c1496c/cache/blob_8f15fe4a8137ca2f9fb348ec634f3703f4fd7317
at
org.apache.flink.runtime.blob.BlobClient.receiveAndCheckResponse(BlobClient.java:213)
at org.apache.flink.runtime.blob.BlobClient.get(BlobClient.java:159)
... 6 more
Caused by: java.io.IOException: Cannot find required BLOB at
/tmp/blobStore-0f9a63e3-5700-4d47-aea7-310506c1496c/cache/blob_8f15fe4a8137ca2f9fb348ec634f3703f4fd7317
at
org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:202)
at
org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:112)
