Zhenqiu Huang created FLINK-37246:
-------------------------------------
Summary: Delegation Token expire after enable delegation token
framework
Key: FLINK-37246
URL: https://issues.apache.org/jira/browse/FLINK-37246
Project: Flink
Issue Type: Bug
Components: Deployment / YARN
Affects Versions: 1.19.1, 1.20.0, 1.18.1
Reporter: Zhenqiu Huang
Fix For: 1.20.2
We see delegation token expire after enabling the delegation token in Flink
1.18.
2025-01-30 13:51:20.967 [Checkpoint Timer] WARN
org.apache.flink.runtime.checkpoint.CheckpointFailureManager - Failed to
trigger or complete checkpoint 24406 for job 757e9207f2c8cf3f0954e578438f7f01.
(0 consecutive failed attempts so far)
org.apache.flink.runtime.checkpoint.CheckpointException: An Exception occurred
while triggering the checkpoint. IO-problem detected.
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.getCheckpointException(CheckpointCoordinator.java:2226)
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.onTriggerFailure(CheckpointCoordinator.java:989)
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.onTriggerFailure(CheckpointCoordinator.java:967)
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.lambda$startTriggeringCheckpoint$7(CheckpointCoordinator.java:695)
at
java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
at
java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
at
java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.hadoop.ipc.RemoteException: token (HDFS_DELEGATION_TOKEN
token 46908908 for rt_data) can't be found in cache
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1519)
at org.apache.hadoop.ipc.Client.call(Client.java:1457)
at org.apache.hadoop.ipc.Client.call(Client.java:1367)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at jdk.proxy2/jdk.proxy2.$Proxy22.mkdirs(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:598)
at jdk.internal.reflect.GeneratedMethodAccessor80.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:411)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:348)
at jdk.proxy2/jdk.proxy2.$Proxy23.mkdirs(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2446)
at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2421)
at
org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1191)
at
org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1188)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1188)
at
org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1180)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1943)
at
org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.mkdirs(HadoopFileSystem.java:183)
at
org.apache.flink.runtime.state.filesystem.FsCheckpointStorageAccess.initializeLocationForCheckpoint(FsCheckpointStorageAccess.java:135)
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.initializeCheckpointLocation(CheckpointCoordinator.java:827)
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.lambda$startTriggeringCheckpoint$4(CheckpointCoordinator.java:618)
at
java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
at
java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
... 3 common frames omitted
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
