Hi,
I'm running the sql-runner example in the flink-kuberenetes-operator repo
and hitting an error. I've tried building the image from different releases
of the repo and that does not help. The pods run (status stays at Running)
and it is able to build the execution graph successfully, however I see the
error below (the important part seems to be `The file STDOUT does not exist
on the TaskExecutor`). I've also tried building the python example that has
a similar sql query and I get the same error. I'm running this example on
minikube, building everything as-is from the OSS flink-kubernetes-operator
repo. Any idea why I'm getting this error?
Full Error Logs:
2023-05-04 19:36:29,983 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
orders[1] -> Sink: print_table[2] (1/1)
(fa0a897ca85acd1d2b7719bb2c929d2f_cbc357ccb763df2852fee8c4fc7d55f2_0_0)
switched from SCHEDULED to DEPLOYING.
2023-05-04 19:36:30,092 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
Deploying Source: orders[1] -> Sink: print_table[2] (1/1) (attempt #0) with
attempt id
fa0a897ca85acd1d2b7719bb2c929d2f_cbc357ccb763df2852fee8c4fc7d55f2_0_0 and
vertex id cbc357ccb763df2852fee8c4fc7d55f2_0 to
python-example-taskmanager-1-1 @ 10.244.0.40 (dataPort=39441) with
allocation id fe50815410ace91f428c4aee34bce7ab
2023-05-04 19:36:30,388 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
orders[1] -> Sink: print_table[2] (1/1)
(fa0a897ca85acd1d2b7719bb2c929d2f_cbc357ccb763df2852fee8c4fc7d55f2_0_0)
switched from DEPLOYING to INITIALIZING.
2023-05-04 19:36:30,496 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
orders[1] -> Sink: print_table[2] (1/1)
(fa0a897ca85acd1d2b7719bb2c929d2f_cbc357ccb763df2852fee8c4fc7d55f2_0_0)
switched from INITIALIZING to RUNNING.
2023-05-04 19:39:23,567 ERROR
org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerStdoutFileHandler
[] - Failed to transfer file from TaskExecutor
python-example-taskmanager-1-1.
java.util.concurrent.CompletionException:
org.apache.flink.util.FlinkException: The file STDOUT does not exist on the
TaskExecutor.
at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
~[?:?]
at
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
~[?:?]
at
java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:704)
~[?:?]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
~[?:?]
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
~[?:?]
at
org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandler.java:261)
~[?:?]
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
~[?:?]
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
~[?:?]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
~[?:?]
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
~[?:?]
at
org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1275)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$null$1(ClassLoadingUtils.java:93)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
~[?:?]
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
~[?:?]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
~[?:?]
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
~[?:?]
at
org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$1.onComplete(AkkaFutureUtils.java:45)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at akka.dispatch.OnComplete.internal(Future.scala:299)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at akka.dispatch.OnComplete.internal(Future.scala:297)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:224)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:221)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$DirectExecutionContext.execute(AkkaFutureUtils.java:65)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:621)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:25)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:23)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at scala.concurrent.Future.$anonfun$andThen$1(Future.scala:532)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
[?:?]
at
java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
[?:?]
at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
[?:?]
at
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) [?:?]
at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
[?:?]
Caused by: org.apache.flink.util.FlinkException: The file STDOUT does not
exist on the TaskExecutor.
at
org.apache.flink.runtime.taskexecutor.TaskExecutor.lambda$requestFileUploadByFilePath$24(TaskExecutor.java:2154)
~[flink-dist-1.16.1.jar:1.16.1]
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
2023-05-04 19:39:23,576 ERROR
org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerStdoutFileHandler
[] - Unhandled exception.
org.apache.flink.util.FlinkException: The file STDOUT does not exist on the
TaskExecutor.
at
org.apache.flink.runtime.taskexecutor.TaskExecutor.lambda$requestFileUploadByFilePath$24(TaskExecutor.java:2154)
~[flink-dist-1.16.1.jar:1.16.1]
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
