Hi Team,
We are using Apache Flink 1.16.1 configured as a standalone Kubernetes pod
,for one of our applications to read from confluent Kafka topics to do
event correlation. We are using the flink's Table API join for the same (in
SQL format).We are able to submit the job using the flink's UI. For our DEV
environment , we implemented a jenkins pipeline, which downloads the jar
that is required to submit the job and also creates the flink kubernetes
pods and copy the downloaded jar to the flink pod's folder and uses the
flink's run command to submit the job.The deployment step happens through
the spinnaker webhook. We use a docker file to create the kubernetes pods,
also have a docker-entrypoin.sh which has the flink run command to submit
the job.
Everything works fine, but when the job is getting submitted , we get the
below exception.
The flink run command used is
*flink run /opt/flink/lib/application-0.0.1.jar*
Any help is appreciated.
------------------------------------------------------------
The program finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The main
method caused an error: Failed to execute sql
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
at
org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:98)
at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:843)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:240)
at
org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1087)
at
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1165)
at
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1165)
Caused by: org.apache.flink.table.api.TableException: Failed to execute sql
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:867)
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:827)
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:918)
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:730)
at com.sample.SampleStreamingApp.main(SampleStreamingApp.java:157)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
... 8 more
Caused by: org.apache.flink.util.FlinkException: Failed to execute job
'insert-into_default_catalog.default_database.Sample'.
at
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2203)
at
org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:206)
at
org.apache.flink.table.planner.delegation.DefaultExecutor.executeAsync(DefaultExecutor.java:95)
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:850)
... 17 more
Caused by: org.apache.flink.runtime.client.JobSubmissionException:
Failed to submit JobGraph.
15:06:44.964 [main] ERROR org.apache.flink.client.cli.CliFrontend -
Error while running the command.
org.apache.flink.client.program.ProgramInvocationException: The main
method caused an error: Failed to execute sql
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
~[application-0.0.1.jar:?]
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
~[application-0.0.1.jar:?]
at
org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:98)
~[application-0.0.1.jar:?]
at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:843)
~[application-0.0.1.jar:?]
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:240)
~[application-0.0.1.jar:?]
at
org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1087)
~[application-0.0.1.jar:?]
at
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1165)
~[application-0.0.1.jar:?]
at
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
[application-0.0.1.jar:?]
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1165)
[application-0.0.1.jar:?]
aused by: org.apache.flink.table.api.TableException: Failed to execute sql
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:867)
~[application-0.0.1.jar:?]
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:827)
~[application-0.0.1.jar:?]
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:918)
~[application-0.0.1.jar:?]
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:730)
~[application-0.0.1.jar:?]
at com.sample.SampleStreamingApp.main(SampleStreamingApp.java:157)
~[application-0.0.1.jar:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown
Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source) ~[?:?]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
~[application-0.0.1.jar:?]
... 8 more
Caused by: org.apache.flink.util.FlinkException: Failed to execute job
'insert-into_default_catalog.default_database.Sample'.
at
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2203)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:206)
~[application-0.0.1.jar:?]
at
org.apache.flink.table.planner.delegation.DefaultExecutor.executeAsync(DefaultExecutor.java:95)
~[?:?]
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:850)
~[application-0.0.1.jar:?]
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:827)
~[application-0.0.1.jar:?]
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:918)
~[application-0.0.1.jar:?]
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:730)
~[application-0.0.1.jar:?]
at com.sample.SampleStreamingApp.main(SampleStreamingApp.java:157)
~[application-0.0.1.jar:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown
Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source) ~[?:?]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
~[application-0.0.1.jar:?]
... 8 more
Caused by: org.apache.flink.runtime.client.JobSubmissionException:
Failed to submit JobGraph.
at
org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$11(RestClusterClient.java:448)
~[application-0.0.1.jar:?]
at java.util.concurrent.CompletableFuture.uniExceptionally(Unknown
Source) ~[?:?]
at
java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(Unknown
Source) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(Unknown
Source) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown
Source) ~[?:?]
at
org.apache.flink.util.concurrent.FutureUtils.lambda$retryOperationWithDelay$6(FutureUtils.java:271)
~[application-0.0.1.jar:?]
at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown
Source) ~[?:?]
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown
Source) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(Unknown
Source) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown
Source) ~[?:?]
at
org.apache.flink.util.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1125)
~[application-0.0.1.jar:?]
at
org.apache.flink.util.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
~[application-0.0.1.jar:?]
at
org.apache.flink.util.concurrent.FutureUtils.lambda$orTimeout$12(FutureUtils.java:489)
~[application-0.0.1.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source) ~[?:?]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source) ~[?:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: java.util.concurrent.TimeoutException
at
org.apache.flink.util.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1125)
~[application-0.0.1.jar:?]
at
org.apache.flink.util.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
~[application-0.0.1.jar:?]
at
org.apache.flink.util.concurrent.FutureUtils.lambda$orTimeout$12(FutureUtils.java:489)
~[application-0.0.1.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source) ~[?:?]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source) ~[?:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
at
org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$11(RestClusterClient.java:448)
at
java.base/java.util.concurrent.CompletableFuture.uniExceptionally(Unknown
Source)
at
java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(Unknown
Source)
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown
Source)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown
Source)
at
org.apache.flink.util.concurrent.FutureUtils.lambda$retryOperationWithDelay$6(FutureUtils.java:271)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown
Source)
at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown
Source)
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown
Source)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown
Source)
at
org.apache.flink.util.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1125)
at
org.apache.flink.util.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
at
org.apache.flink.util.concurrent.FutureUtils.lambda$orTimeout$12(FutureUtils.java:489)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.util.concurrent.TimeoutException
... 9 more
