Hi Everyone
I am facing some issues while running the batch job on a Flink cluster
using Adaptive Batch Scheduler. I have deployed a flink cluster on
Kubernetes using the flink Kubernetes operator and submitted a job to the
cluster using Apache beam FlinkRunner. I am using Flink version 1.16. I
want to figure out two things.
*1.* I am trying to run one batch job with an adaptive batch scheduler. I
referred to the elastic scaling document [1] and based on the document I
have added the below config to my FlinkDeployment. But the job is failing
with JobInitializationException: Could not start the JobMaster. I have
attached the job manager log below. *Can someone explain why the job is
failing despite using the recommended configuration?*
jobmanager.scheduler: AdaptiveBatch
parallelism.default: "-1"
taskmanager.network.memory.buffers-per-channel: "0"
execution.batch-shuffle-mode: "ALL_EXCHANGES_BLOCKING"
*2. *The document has listed some limitations of elastic scaling. One of
them is about the input file format. *My code is written in Apache Beam,
how to check, how the input is being read in Flink?*
- FileInputFormat sources are not supported: FileInputFormat sources are
not supported, including StreamExecutionEnvironment#readFile(...)
StreamExecutionEnvironment#readTextFile(...) and
StreamExecutionEnvironment#createInput(FileInputFormat,
...). Users should use the new sources(FileSystem DataStream Connector
<https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/datastream/filesystem/>
or FileSystem SQL Connector
<https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/filesystem/>)
to read files when using the Adaptive Batch Scheduler.
Let me know if additional information is needed to debug the issue.
[1]
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/elastic_scaling/#adaptive-batch-scheduler
Regards
Dipak
Enabling required built-in plugins
Linking flink-gs-fs-hadoop-1.16.3.jar to plugin directory
Successfully enabled flink-gs-fs-hadoop-1.16.3.jar
sed: couldn't open temporary file /opt/flink/conf/sedqH57Xz: Read-only file
system
sed: couldn't open temporary file /opt/flink/conf/sedw7wkdz: Read-only file
system
/docker-entrypoint.sh: line 73: /opt/flink/conf/flink-conf.yaml: Read-only file
system
/docker-entrypoint.sh: line 89: /opt/flink/conf/flink-conf.yaml.tmp: Read-only
file system
Starting kubernetes-session as a console application on host
flink-deployment-fl-7fcf489865-kcrxl.
2024-03-28 16:06:35,858 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
--------------------------------------------------------------------------------
2024-03-28 16:06:35,863 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
Preconfiguration:
2024-03-28 16:06:35,864 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
RESOURCE_PARAMS extraction logs:
jvm_params: -Xmx9034530801 -Xms9034530801 -XX:MaxMetaspaceSize=268435456
dynamic_configs: -D jobmanager.memory.off-heap.size=134217728b -D
jobmanager.memory.jvm-overhead.min=1048576015b -D
jobmanager.memory.jvm-metaspace.size=268435456b -D
jobmanager.memory.heap.size=9034530801b -D
jobmanager.memory.jvm-overhead.max=1048576015b
logs: WARNING: sun.reflect.Reflection.getCallerClass is not supported. This
will impact performance.
INFO [] - Loading configuration property: blob.server.port, 6124
INFO [] - Loading configuration property: kubernetes.jobmanager.annotations,
flinkdeployment.flink.apache.org/generation:8
INFO [] - Loading configuration property: kubernetes.jobmanager.replicas, 1
INFO [] - Loading configuration property: jobmanager.rpc.address,
flink-deployment-fl.flink
INFO [] - Loading configuration property: kubernetes.taskmanager.cpu, 4.0
INFO [] - Loading configuration property: kubernetes.service-account, flink
INFO [] - Loading configuration property: kubernetes.cluster-id,
flink-deployment-fl
INFO [] - Loading configuration property:
taskmanager.memory.framework.off-heap.size, 512m
INFO [] - Loading configuration property: kubernetes.container.image,
flink:1.16
INFO [] - Loading configuration property: parallelism.default, -1
INFO [] - Loading configuration property: kubernetes.namespace, flink
INFO [] - Loading configuration property: taskmanager.numberOfTaskSlots, 1
INFO [] - Loading configuration property:
kubernetes.rest-service.exposed.type, ClusterIP
INFO [] - Loading configuration property:
kubernetes.jobmanager.owner.reference,
uid:d17af556-505a-4084-9ae5-eea79aca86c6,name:flink-deployment-fl,controller:false,blockOwnerDeletion:true,apiVersion:flink.apache.org/v1beta1,kind:FlinkDeployment
INFO [] - Loading configuration property: execution.batch-shuffle-mode,
ALL_EXCHANGES_BLOCKING
INFO [] - Loading configuration property: taskmanager.memory.process.size,
8192m
INFO [] - Loading configuration property:
kubernetes.internal.jobmanager.entrypoint.class,
org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint
INFO [] - Loading configuration property:
kubernetes.pod-template-file.taskmanager,
/tmp/flink_op_generated_podTemplate_17087270823600889192.yaml
INFO [] - Loading configuration property: web.cancel.enable, false
INFO [] - Loading configuration property:
taskmanager.network.memory.buffers-per-channel, 0
INFO [] - Loading configuration property: execution.target, kubernetes-session
INFO [] - Loading configuration property: jobmanager.memory.process.size,
10000m
INFO [] - Loading configuration property: taskmanager.rpc.port, 6122
INFO [] - Loading configuration property: internal.cluster.execution-mode,
NORMAL
INFO [] - Loading configuration property: jobmanager.scheduler, AdaptiveBatch
INFO [] - Loading configuration property: kubernetes.jobmanager.cpu, 4.0
INFO [] - Loading configuration property: $internal.flink.version, v1_16
INFO [] - Loading configuration property:
kubernetes.pod-template-file.jobmanager,
/tmp/flink_op_generated_podTemplate_1781475540500341703.yaml
INFO [] - Final Master Memory configuration:
INFO [] - Total Process Memory: 9.766gb (10485760000 bytes)
INFO [] - Total Flink Memory: 8.539gb (9168748529 bytes)
INFO [] - JVM Heap: 8.414gb (9034530801 bytes)
INFO [] - Off-heap: 128.000mb (134217728 bytes)
INFO [] - JVM Metaspace: 256.000mb (268435456 bytes)
INFO [] - JVM Overhead: 1000.000mb (1048576015 bytes)
2024-03-28 16:06:35,865 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
--------------------------------------------------------------------------------
2024-03-28 16:06:35,865 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting
KubernetesSessionClusterEntrypoint (Version: 1.16.3, Scala: 2.12, Rev:5df8156,
Date:2023-11-10T11:00:33+01:00)
2024-03-28 16:06:35,867 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - OS current
user: flink
2024-03-28 16:06:35,869 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Current
Hadoop/Kerberos user: <no hadoop dependency found>
2024-03-28 16:06:35,870 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM: OpenJDK
64-Bit Server VM - Eclipse Adoptium - 11/11.0.22+7
2024-03-28 16:06:35,870 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Arch: amd64
2024-03-28 16:06:35,870 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Maximum heap
size: 8616 MiBytes
2024-03-28 16:06:35,871 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JAVA_HOME:
/opt/java/openjdk
2024-03-28 16:06:35,873 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - No Hadoop
Dependency available
2024-03-28 16:06:35,875 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM Options:
2024-03-28 16:06:35,876 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Xmx9034530801
2024-03-28 16:06:35,876 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Xms9034530801
2024-03-28 16:06:35,876 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-XX:MaxMetaspaceSize=268435456
2024-03-28 16:06:35,877 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Dlog.file=/opt/flink/log/flink--kubernetes-session-0-flink-deployment-fl-7fcf489865-kcrxl.log
2024-03-28 16:06:35,877 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
2024-03-28 16:06:35,877 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties
2024-03-28 16:06:35,877 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
-Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
2024-03-28 16:06:35,878 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Program
Arguments:
2024-03-28 16:06:35,880 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2024-03-28 16:06:35,880 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
jobmanager.memory.off-heap.size=134217728b
2024-03-28 16:06:35,880 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2024-03-28 16:06:35,880 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
jobmanager.memory.jvm-overhead.min=1048576015b
2024-03-28 16:06:35,880 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2024-03-28 16:06:35,880 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
jobmanager.memory.jvm-metaspace.size=268435456b
2024-03-28 16:06:35,881 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2024-03-28 16:06:35,881 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
jobmanager.memory.heap.size=9034530801b
2024-03-28 16:06:35,881 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D
2024-03-28 16:06:35,881 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
jobmanager.memory.jvm-overhead.max=1048576015b
2024-03-28 16:06:35,881 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Classpath:
/opt/flink/lib/flink-cep-1.16.3.jar:/opt/flink/lib/flink-connector-files-1.16.3.jar:/opt/flink/lib/flink-csv-1.16.3.jar:/opt/flink/lib/flink-json-1.16.3.jar:/opt/flink/lib/flink-scala_2.12-1.16.3.jar:/opt/flink/lib/flink-shaded-zookeeper-3.5.9.jar:/opt/flink/lib/flink-table-api-java-uber-1.16.3.jar:/opt/flink/lib/flink-table-planner-loader-1.16.3.jar:/opt/flink/lib/flink-table-runtime-1.16.3.jar:/opt/flink/lib/log4j-1.2-api-2.17.1.jar:/opt/flink/lib/log4j-api-2.17.1.jar:/opt/flink/lib/log4j-core-2.17.1.jar:/opt/flink/lib/log4j-slf4j-impl-2.17.1.jar:/opt/flink/lib/flink-dist-1.16.3.jar::::
2024-03-28 16:06:35,881 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -
--------------------------------------------------------------------------------
2024-03-28 16:06:35,884 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Registered
UNIX signal handlers for [TERM, HUP, INT]
2024-03-28 16:06:35,917 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: blob.server.port, 6124
2024-03-28 16:06:35,918 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.jobmanager.annotations,
flinkdeployment.flink.apache.org/generation:8
2024-03-28 16:06:35,918 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.jobmanager.replicas, 1
2024-03-28 16:06:35,918 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.rpc.address, flink-deployment-fl.flink
2024-03-28 16:06:35,919 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.taskmanager.cpu, 4.0
2024-03-28 16:06:35,919 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.service-account, flink
2024-03-28 16:06:35,919 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.cluster-id, flink-deployment-fl
2024-03-28 16:06:35,919 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.memory.framework.off-heap.size, 512m
2024-03-28 16:06:35,920 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.container.image, flink:1.16
2024-03-28 16:06:35,920 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: parallelism.default, -1
2024-03-28 16:06:35,920 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.namespace, flink
2024-03-28 16:06:35,920 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.numberOfTaskSlots, 1
2024-03-28 16:06:35,920 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.rest-service.exposed.type, ClusterIP
2024-03-28 16:06:35,921 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.jobmanager.owner.reference,
uid:d17af556-505a-4084-9ae5-eea79aca86c6,name:flink-deployment-fl,controller:false,blockOwnerDeletion:true,apiVersion:flink.apache.org/v1beta1,kind:FlinkDeployment
2024-03-28 16:06:35,921 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: execution.batch-shuffle-mode, ALL_EXCHANGES_BLOCKING
2024-03-28 16:06:35,921 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.memory.process.size, 8192m
2024-03-28 16:06:35,921 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.internal.jobmanager.entrypoint.class,
org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint
2024-03-28 16:06:35,922 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.pod-template-file.taskmanager,
/tmp/flink_op_generated_podTemplate_17087270823600889192.yaml
2024-03-28 16:06:35,922 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: web.cancel.enable, false
2024-03-28 16:06:35,922 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.network.memory.buffers-per-channel, 0
2024-03-28 16:06:35,922 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: execution.target, kubernetes-session
2024-03-28 16:06:35,922 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.memory.process.size, 10000m
2024-03-28 16:06:35,923 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.rpc.port, 6122
2024-03-28 16:06:35,923 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: internal.cluster.execution-mode, NORMAL
2024-03-28 16:06:35,923 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.scheduler, AdaptiveBatch
2024-03-28 16:06:35,923 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.jobmanager.cpu, 4.0
2024-03-28 16:06:35,923 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: $internal.flink.version, v1_16
2024-03-28 16:06:35,924 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.pod-template-file.jobmanager,
/tmp/flink_op_generated_podTemplate_1781475540500341703.yaml
2024-03-28 16:06:35,924 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
dynamic configuration property: jobmanager.memory.off-heap.size, 134217728b
2024-03-28 16:06:35,924 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
dynamic configuration property: jobmanager.memory.jvm-overhead.min, 1048576015b
2024-03-28 16:06:35,924 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
dynamic configuration property: jobmanager.memory.jvm-metaspace.size, 268435456b
2024-03-28 16:06:35,925 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
dynamic configuration property: jobmanager.memory.heap.size, 9034530801b
2024-03-28 16:06:35,925 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
dynamic configuration property: jobmanager.memory.jvm-overhead.max, 1048576015b
2024-03-28 16:06:36,500 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting
KubernetesSessionClusterEntrypoint.
2024-03-28 16:06:36,621 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install
default filesystem.
2024-03-28 16:06:36,630 INFO org.apache.flink.core.fs.FileSystem
[] - Hadoop is not in the classpath/dependencies. The extended set
of supported File Systems via Hadoop is not available.
2024-03-28 16:06:36,870 INFO org.apache.flink.fs.gs.GSFileSystemFactory
[] - Creating GSFileSystemFactory
2024-03-28 16:06:36,939 INFO org.apache.flink.fs.gs.GSFileSystemFactory
[] - Configuring GSFileSystemFactory with Flink configuration
{blob.server.port=6124,
kubernetes.jobmanager.annotations=flinkdeployment.flink.apache.org/generation:8,
kubernetes.jobmanager.replicas=1,
jobmanager.rpc.address=flink-deployment-fl.flink,
jobmanager.memory.jvm-overhead.min=1048576015b, kubernetes.taskmanager.cpu=4.0,
kubernetes.service-account=flink, kubernetes.cluster-id=flink-deployment-fl,
taskmanager.memory.framework.off-heap.size=512m,
kubernetes.container.image=flink:1.16, parallelism.default=-1,
kubernetes.namespace=flink, taskmanager.numberOfTaskSlots=1,
kubernetes.rest-service.exposed.type=ClusterIP,
kubernetes.jobmanager.owner.reference=uid:d17af556-505a-4084-9ae5-eea79aca86c6,name:flink-deployment-fl,controller:false,blockOwnerDeletion:true,apiVersion:flink.apache.org/v1beta1,kind:FlinkDeployment,
execution.batch-shuffle-mode=ALL_EXCHANGES_BLOCKING,
taskmanager.memory.process.size=8192m,
kubernetes.internal.jobmanager.entrypoint.class=org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint,
kubernetes.pod-template-file.taskmanager=/tmp/flink_op_generated_podTemplate_17087270823600889192.yaml,
jobmanager.memory.off-heap.size=134217728b, web.cancel.enable=false,
taskmanager.network.memory.buffers-per-channel=0,
execution.target=kubernetes-session, jobmanager.memory.process.size=10000m,
web.tmpdir=/tmp/flink-web-0287dafa-b2ad-46c0-b982-c5a46d919abf,
taskmanager.rpc.port=6122, internal.cluster.execution-mode=NORMAL,
jobmanager.scheduler=AdaptiveBatch, kubernetes.jobmanager.cpu=4.0,
$internal.flink.version=v1_16, jobmanager.memory.jvm-metaspace.size=268435456b,
kubernetes.pod-template-file.jobmanager=/tmp/flink_op_generated_podTemplate_1781475540500341703.yaml,
jobmanager.memory.heap.size=9034530801b,
jobmanager.memory.jvm-overhead.max=1048576015b}
2024-03-28 16:06:37,246 INFO org.apache.flink.fs.gs.GSFileSystemFactory
[] - Using Hadoop configuration {"properties":[]}
2024-03-28 16:06:37,251 INFO org.apache.flink.fs.gs.GSFileSystemFactory
[] - Using file system options
GSFileSystemOptions{writerTemporaryBucketName=Optional.empty,
writerChunkSize=Optional.empty}
2024-03-28 16:06:37,254 INFO org.apache.flink.fs.gs.utils.ConfigUtils
[] - GSRecoverableWriter is using GOOGLE_APPLICATION_CREDENTIALS at
/app/secret/ml-gcp-service-account.json
2024-03-28 16:06:37,256 INFO org.apache.flink.fs.gs.utils.ConfigUtils
[] - Creating GSRecoverableWriter using credentials from
/app/secret/ml-gcp-service-account.json
2024-03-28 16:06:37,744 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install
security context.
2024-03-28 16:06:37,772 INFO
org.apache.flink.runtime.security.modules.HadoopModuleFactory [] - Cannot
create Hadoop Security Module because Hadoop cannot be found in the Classpath.
2024-03-28 16:06:37,783 INFO
org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file
will be created as /tmp/jaas-8825585220084806408.conf.
2024-03-28 16:06:37,808 INFO
org.apache.flink.runtime.security.contexts.HadoopSecurityContextFactory [] -
Cannot install HadoopSecurityContext because Hadoop cannot be found in the
Classpath.
2024-03-28 16:06:37,814 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Initializing
cluster services.
2024-03-28 16:06:37,837 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Using working
directory: WorkingDirectory(/tmp/jm_eb670ab8a7defba7a17aca40de781e0f).
2024-03-28 16:06:38,844 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to
start actor system, external address flink-deployment-fl.flink:6123, bind
address 0.0.0.0:6123.
2024-03-28 16:06:40,738 INFO akka.event.slf4j.Slf4jLogger
[] - Slf4jLogger started
2024-03-28 16:06:40,836 INFO akka.remote.RemoteActorRefProvider
[] - Akka Cluster not in use - enabling unsafe features anyway
because `akka.remote.use-unsafe-remote-features-outside-cluster` has been
enabled.
2024-03-28 16:06:40,838 INFO akka.remote.Remoting
[] - Starting remoting
2024-03-28 16:06:41,146 INFO akka.remote.Remoting
[] - Remoting started; listening on addresses
:[akka.tcp://flink@flink-deployment-fl.flink:6123]
2024-03-28 16:06:41,564 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system
started at akka.tcp://flink@flink-deployment-fl.flink:6123
2024-03-28 16:06:41,629 INFO org.apache.flink.configuration.Configuration
[] - Config uses fallback configuration key 'jobmanager.rpc.address'
instead of key 'rest.address'
2024-03-28 16:06:41,646 INFO org.apache.flink.runtime.blob.BlobServer
[] - Created BLOB server storage directory
/tmp/jm_eb670ab8a7defba7a17aca40de781e0f/blobStorage
2024-03-28 16:06:41,658 INFO org.apache.flink.runtime.blob.BlobServer
[] - Started BLOB server at 0.0.0.0:6124 - max concurrent requests:
50 - max backlog: 1000
2024-03-28 16:06:41,668 INFO
org.apache.flink.runtime.security.token.KerberosDelegationTokenManagerFactory
[] - Cannot use kerberos delegation token manager because Hadoop cannot be
found in the Classpath.
2024-03-28 16:06:41,688 INFO
org.apache.flink.runtime.metrics.MetricRegistryImpl [] - No metrics
reporter configured, no metrics will be exposed/reported.
2024-03-28 16:06:41,696 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to
start actor system, external address flink-deployment-fl.flink:0, bind address
0.0.0.0:0.
2024-03-28 16:06:41,737 INFO akka.event.slf4j.Slf4jLogger
[] - Slf4jLogger started
2024-03-28 16:06:41,752 INFO akka.remote.RemoteActorRefProvider
[] - Akka Cluster not in use - enabling unsafe features anyway
because `akka.remote.use-unsafe-remote-features-outside-cluster` has been
enabled.
2024-03-28 16:06:41,753 INFO akka.remote.Remoting
[] - Starting remoting
2024-03-28 16:06:41,786 INFO akka.remote.Remoting
[] - Remoting started; listening on addresses
:[akka.tcp://flink-metrics@flink-deployment-fl.flink:33587]
2024-03-28 16:06:41,805 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system
started at akka.tcp://flink-metrics@flink-deployment-fl.flink:33587
2024-03-28 16:06:41,834 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
[] - Starting RPC endpoint for
org.apache.flink.runtime.metrics.dump.MetricQueryService at
akka://flink-metrics/user/rpc/MetricQueryService .
2024-03-28 16:06:41,870 INFO
org.apache.flink.runtime.dispatcher.FileExecutionGraphInfoStore [] -
Initializing FileExecutionGraphInfoStore: Storage directory
/tmp/executionGraphStore-6ed7b419-06b6-410d-9e73-dd4f45e54f06, expiration time
3600000, maximum cache size 52428800 bytes.
2024-03-28 16:06:41,932 INFO org.apache.flink.configuration.Configuration
[] - Config uses fallback configuration key 'jobmanager.rpc.address'
instead of key 'rest.address'
2024-03-28 16:06:41,933 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - Upload
directory /tmp/flink-web-0287dafa-b2ad-46c0-b982-c5a46d919abf/flink-web-upload
does not exist.
2024-03-28 16:06:41,934 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - Created
directory /tmp/flink-web-0287dafa-b2ad-46c0-b982-c5a46d919abf/flink-web-upload
for file uploads.
2024-03-28 16:06:41,936 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - Starting rest
endpoint.
2024-03-28 16:06:42,213 INFO
org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined
location of main cluster component log file:
/opt/flink/log/flink--kubernetes-session-0-flink-deployment-fl-7fcf489865-kcrxl.log
2024-03-28 16:06:42,214 INFO
org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined
location of main cluster component stdout file:
/opt/flink/log/flink--kubernetes-session-0-flink-deployment-fl-7fcf489865-kcrxl.out
2024-03-28 16:06:42,521 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - Rest endpoint
listening at flink-deployment-fl.flink:8081
2024-03-28 16:06:42,526 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] -
http://flink-deployment-fl.flink:8081 was granted leadership with
leaderSessionID=00000000-0000-0000-0000-000000000000
2024-03-28 16:06:42,528 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint [] - Web frontend
listening at http://flink-deployment-fl.flink:8081.
2024-03-28 16:06:42,596 INFO
org.apache.flink.runtime.dispatcher.runner.DefaultDispatcherRunner [] -
DefaultDispatcherRunner was granted leadership with leader id
00000000-0000-0000-0000-000000000000. Creating new DispatcherLeaderProcess.
2024-03-28 16:06:42,610 INFO
org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] -
Start SessionDispatcherLeaderProcess.
2024-03-28 16:06:42,616 INFO
org.apache.flink.runtime.resourcemanager.ResourceManagerServiceImpl [] -
Starting resource manager service.
2024-03-28 16:06:42,621 INFO
org.apache.flink.runtime.resourcemanager.ResourceManagerServiceImpl [] -
Resource manager service is granted leadership with session id
00000000-0000-0000-0000-000000000000.
2024-03-28 16:06:42,631 INFO
org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] -
Recover all persisted job graphs that are not finished, yet.
2024-03-28 16:06:42,631 INFO
org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] -
Successfully recovered 0 persisted job graphs.
2024-03-28 16:06:42,664 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
[] - Starting RPC endpoint for
org.apache.flink.runtime.dispatcher.StandaloneDispatcher at
akka://flink/user/rpc/dispatcher_0 .
2024-03-28 16:06:43,328 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: blob.server.port, 6124
2024-03-28 16:06:43,329 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.jobmanager.annotations,
flinkdeployment.flink.apache.org/generation:8
2024-03-28 16:06:43,329 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.jobmanager.replicas, 1
2024-03-28 16:06:43,329 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.rpc.address, flink-deployment-fl.flink
2024-03-28 16:06:43,329 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.taskmanager.cpu, 4.0
2024-03-28 16:06:43,329 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.service-account, flink
2024-03-28 16:06:43,329 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.cluster-id, flink-deployment-fl
2024-03-28 16:06:43,330 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.memory.framework.off-heap.size, 512m
2024-03-28 16:06:43,331 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.container.image, flink:1.16
2024-03-28 16:06:43,331 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: parallelism.default, -1
2024-03-28 16:06:43,331 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.namespace, flink
2024-03-28 16:06:43,332 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.numberOfTaskSlots, 1
2024-03-28 16:06:43,333 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.rest-service.exposed.type, ClusterIP
2024-03-28 16:06:43,334 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.jobmanager.owner.reference,
uid:d17af556-505a-4084-9ae5-eea79aca86c6,name:flink-deployment-fl,controller:false,blockOwnerDeletion:true,apiVersion:flink.apache.org/v1beta1,kind:FlinkDeployment
2024-03-28 16:06:43,334 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: execution.batch-shuffle-mode, ALL_EXCHANGES_BLOCKING
2024-03-28 16:06:43,334 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.memory.process.size, 8192m
2024-03-28 16:06:43,334 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.internal.jobmanager.entrypoint.class,
org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint
2024-03-28 16:06:43,335 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.pod-template-file.taskmanager,
/tmp/flink_op_generated_podTemplate_17087270823600889192.yaml
2024-03-28 16:06:43,335 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: web.cancel.enable, false
2024-03-28 16:06:43,335 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.network.memory.buffers-per-channel, 0
2024-03-28 16:06:43,336 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: execution.target, kubernetes-session
2024-03-28 16:06:43,336 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.memory.process.size, 10000m
2024-03-28 16:06:43,336 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.rpc.port, 6122
2024-03-28 16:06:43,336 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: internal.cluster.execution-mode, NORMAL
2024-03-28 16:06:43,336 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.scheduler, AdaptiveBatch
2024-03-28 16:06:43,336 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.jobmanager.cpu, 4.0
2024-03-28 16:06:43,337 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: $internal.flink.version, v1_16
2024-03-28 16:06:43,337 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.pod-template-file.jobmanager,
/tmp/flink_op_generated_podTemplate_1781475540500341703.yaml
2024-03-28 16:06:43,346 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
[] - Starting RPC endpoint for
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager at
akka://flink/user/rpc/resourcemanager_1 .
2024-03-28 16:06:43,372 INFO
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
Starting the resource manager.
2024-03-28 16:06:45,414 INFO
org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Recovered 0
pods from previous attempts, current attempt id is 1.
2024-03-28 16:06:45,419 INFO
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
Recovered 0 workers from previous attempt.
2024-03-28 16:10:27,600 WARN
org.apache.flink.runtime.webmonitor.handlers.JarRunHandler [] - Configuring
the job submission via query parameters is deprecated. Please migrate to
submitting a JSON request instead.
2024-03-28 16:10:27,896 INFO org.apache.flink.client.ClientUtils
[] - Starting program (detached: true)
2024-03-28 16:10:29,543 WARN
software.amazon.awssdk.regions.internal.util.EC2MetadataUtils [] - Unable to
retrieve the requested metadata.
2024-03-28 16:10:30,720 INFO org.apache.beam.runners.flink.FlinkPipelineRunner
[] - Translating pipeline to Flink program.
2024-03-28 16:10:30,963 INFO
org.apache.beam.runners.flink.FlinkExecutionEnvironments [] - Creating a
Batch Execution Environment.
2024-03-28 16:10:30,963 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: blob.server.port, 6124
2024-03-28 16:10:30,964 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.jobmanager.annotations,
flinkdeployment.flink.apache.org/generation:8
2024-03-28 16:10:30,964 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.jobmanager.replicas, 1
2024-03-28 16:10:30,964 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.rpc.address, flink-deployment-fl.flink
2024-03-28 16:10:30,964 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.taskmanager.cpu, 4.0
2024-03-28 16:10:30,964 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.service-account, flink
2024-03-28 16:10:30,964 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.cluster-id, flink-deployment-fl
2024-03-28 16:10:30,965 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.memory.framework.off-heap.size, 512m
2024-03-28 16:10:30,965 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.container.image, flink:1.16
2024-03-28 16:10:30,965 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: parallelism.default, -1
2024-03-28 16:10:30,965 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.namespace, flink
2024-03-28 16:10:30,965 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.numberOfTaskSlots, 1
2024-03-28 16:10:30,965 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.rest-service.exposed.type, ClusterIP
2024-03-28 16:10:30,965 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.jobmanager.owner.reference,
uid:d17af556-505a-4084-9ae5-eea79aca86c6,name:flink-deployment-fl,controller:false,blockOwnerDeletion:true,apiVersion:flink.apache.org/v1beta1,kind:FlinkDeployment
2024-03-28 16:10:30,965 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: execution.batch-shuffle-mode, ALL_EXCHANGES_BLOCKING
2024-03-28 16:10:30,966 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.memory.process.size, 8192m
2024-03-28 16:10:30,966 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.internal.jobmanager.entrypoint.class,
org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint
2024-03-28 16:10:30,966 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.pod-template-file.taskmanager,
/tmp/flink_op_generated_podTemplate_17087270823600889192.yaml
2024-03-28 16:10:30,966 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: web.cancel.enable, false
2024-03-28 16:10:30,966 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.network.memory.buffers-per-channel, 0
2024-03-28 16:10:30,966 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: execution.target, kubernetes-session
2024-03-28 16:10:30,966 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.memory.process.size, 10000m
2024-03-28 16:10:30,966 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.rpc.port, 6122
2024-03-28 16:10:30,966 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: internal.cluster.execution-mode, NORMAL
2024-03-28 16:10:30,967 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.scheduler, AdaptiveBatch
2024-03-28 16:10:30,967 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.jobmanager.cpu, 4.0
2024-03-28 16:10:30,967 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: $internal.flink.version, v1_16
2024-03-28 16:10:30,967 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.pod-template-file.jobmanager,
/tmp/flink_op_generated_podTemplate_1781475540500341703.yaml
2024-03-28 16:10:31,048 WARN
software.amazon.awssdk.regions.internal.util.EC2MetadataUtils [] - Unable to
retrieve the requested metadata.
2024-03-28 16:10:31,630 INFO org.apache.flink.api.java.utils.PlanGenerator
[] - The job has 0 registered types and 0 default Kryo serializers
2024-03-28 16:10:31,723 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: blob.server.port, 6124
2024-03-28 16:10:31,723 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.jobmanager.annotations,
flinkdeployment.flink.apache.org/generation:8
2024-03-28 16:10:31,723 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.jobmanager.replicas, 1
2024-03-28 16:10:31,723 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.rpc.address, flink-deployment-fl.flink
2024-03-28 16:10:31,724 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.taskmanager.cpu, 4.0
2024-03-28 16:10:31,725 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.service-account, flink
2024-03-28 16:10:31,725 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.cluster-id, flink-deployment-fl
2024-03-28 16:10:31,725 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.memory.framework.off-heap.size, 512m
2024-03-28 16:10:31,725 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.container.image, flink:1.16
2024-03-28 16:10:31,725 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: parallelism.default, -1
2024-03-28 16:10:31,725 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.namespace, flink
2024-03-28 16:10:31,725 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.numberOfTaskSlots, 1
2024-03-28 16:10:31,725 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.rest-service.exposed.type, ClusterIP
2024-03-28 16:10:31,725 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.jobmanager.owner.reference,
uid:d17af556-505a-4084-9ae5-eea79aca86c6,name:flink-deployment-fl,controller:false,blockOwnerDeletion:true,apiVersion:flink.apache.org/v1beta1,kind:FlinkDeployment
2024-03-28 16:10:31,726 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: execution.batch-shuffle-mode, ALL_EXCHANGES_BLOCKING
2024-03-28 16:10:31,726 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.memory.process.size, 8192m
2024-03-28 16:10:31,726 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.internal.jobmanager.entrypoint.class,
org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint
2024-03-28 16:10:31,726 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.pod-template-file.taskmanager,
/tmp/flink_op_generated_podTemplate_17087270823600889192.yaml
2024-03-28 16:10:31,726 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: web.cancel.enable, false
2024-03-28 16:10:31,726 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.network.memory.buffers-per-channel, 0
2024-03-28 16:10:31,726 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: execution.target, kubernetes-session
2024-03-28 16:10:31,726 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.memory.process.size, 10000m
2024-03-28 16:10:31,726 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: taskmanager.rpc.port, 6122
2024-03-28 16:10:31,726 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: internal.cluster.execution-mode, NORMAL
2024-03-28 16:10:31,726 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: jobmanager.scheduler, AdaptiveBatch
2024-03-28 16:10:31,726 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.jobmanager.cpu, 4.0
2024-03-28 16:10:31,727 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: $internal.flink.version, v1_16
2024-03-28 16:10:31,727 INFO
org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: kubernetes.pod-template-file.jobmanager,
/tmp/flink_op_generated_podTemplate_1781475540500341703.yaml
2024-03-28 16:10:31,851 INFO
org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] -
Job 8e4a1d1ce1e7651fb1e504e134b2318a is submitted.
2024-03-28 16:10:31,851 INFO
org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] -
Submitting Job with JobId=8e4a1d1ce1e7651fb1e504e134b2318a.
2024-03-28 16:10:33,113 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Received
JobGraph submission 'BeamApp-flink-0328161029-d5130474'
(8e4a1d1ce1e7651fb1e504e134b2318a).
2024-03-28 16:10:33,115 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Submitting
job 'BeamApp-flink-0328161029-d5130474' (8e4a1d1ce1e7651fb1e504e134b2318a).
2024-03-28 16:10:33,132 INFO
org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner [] -
JobMasterServiceLeadershipRunner for job 8e4a1d1ce1e7651fb1e504e134b2318a was
granted leadership with leader id 00000000-0000-0000-0000-000000000000.
Creating new JobMasterServiceProcess.
2024-03-28 16:10:33,148 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
[] - Starting RPC endpoint for
org.apache.flink.runtime.jobmaster.JobMaster at
akka://flink/user/rpc/jobmanager_2 .
2024-03-28 16:10:33,156 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Initializing job 'BeamApp-flink-0328161029-d5130474'
(8e4a1d1ce1e7651fb1e504e134b2318a).
2024-03-28 16:10:33,190 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Job
8e4a1d1ce1e7651fb1e504e134b2318a reached terminal state FAILED.
org.apache.flink.runtime.client.JobInitializationException: Could not start the
JobMaster.
at
org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown
Source)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.util.concurrent.CompletionException:
java.lang.IllegalStateException: At the moment, adaptive batch scheduler
requires batch workloads to be executed with types of all edges being BLOCKING.
To do that, you need to configure 'execution.batch-shuffle-mode' to
'ALL_EXCHANGES_BLOCKING'.
at
java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)
at
java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown
Source)
... 4 more
Caused by: java.lang.IllegalStateException: At the moment, adaptive batch
scheduler requires batch workloads to be executed with types of all edges being
BLOCKING. To do that, you need to configure 'execution.batch-shuffle-mode' to
'ALL_EXCHANGES_BLOCKING'.
at
org.apache.flink.util.Preconditions.checkState(Preconditions.java:193)
at
org.apache.flink.runtime.scheduler.adaptivebatch.AdaptiveBatchSchedulerFactory.checkAllExchangesBlocking(AdaptiveBatchSchedulerFactory.java:236)
at
org.apache.flink.runtime.scheduler.adaptivebatch.AdaptiveBatchSchedulerFactory.createInstance(AdaptiveBatchSchedulerFactory.java:104)
at
org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:119)
at
org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:369)
at
org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:346)
at
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:123)
at
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:95)
at
org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112)
... 4 more
2024-03-28 16:10:33,210 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Job
8e4a1d1ce1e7651fb1e504e134b2318a has been registered for cleanup in the
JobResultStore after reaching a terminal state.
2024-03-28 16:10:33,212 WARN
org.apache.flink.client.deployment.application.DetachedApplicationRunner [] -
Could not execute application:
org.apache.flink.client.program.ProgramInvocationException: The main method
caused an error: Job
BeamApp-flink-0328161029-d5130474_b567045d-42ce-44ce-80ff-14113db9ba2f failed.
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:98)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:70)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:107)
~[flink-dist-1.16.3.jar:1.16.3]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown
Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.lang.RuntimeException: Job
BeamApp-flink-0328161029-d5130474_b567045d-42ce-44ce-80ff-14113db9ba2f failed.
at
org.apache.beam.runners.flink.FlinkPipelineRunner.main(FlinkPipelineRunner.java:212)
~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source) ~[?:?]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
~[flink-dist-1.16.3.jar:1.16.3]
... 7 more
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.RuntimeException:
org.apache.flink.runtime.client.JobInitializationException: Could not start the
JobMaster.
at
org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.api.java.ExecutionEnvironment.executeAsync(ExecutionEnvironment.java:1061)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.client.program.ContextEnvironment.executeAsync(ContextEnvironment.java:132)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:70)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.beam.runners.flink.FlinkBatchPortablePipelineTranslator$BatchTranslationContext.execute(FlinkBatchPortablePipelineTranslator.java:195)
~[?:?]
at
org.apache.beam.runners.flink.FlinkPipelineRunner.runPipelineWithTranslator(FlinkPipelineRunner.java:132)
~[?:?]
at
org.apache.beam.runners.flink.FlinkPipelineRunner.run(FlinkPipelineRunner.java:99)
~[?:?]
at
org.apache.beam.runners.flink.FlinkPipelineRunner.main(FlinkPipelineRunner.java:210)
~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source) ~[?:?]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
~[flink-dist-1.16.3.jar:1.16.3]
... 7 more
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException:
org.apache.flink.runtime.client.JobInitializationException: Could not start the
JobMaster.
at java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
~[?:?]
at java.util.concurrent.CompletableFuture.get(Unknown Source) ~[?:?]
at
org.apache.flink.api.java.ExecutionEnvironment.executeAsync(ExecutionEnvironment.java:1056)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.client.program.ContextEnvironment.executeAsync(ContextEnvironment.java:132)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:70)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.beam.runners.flink.FlinkBatchPortablePipelineTranslator$BatchTranslationContext.execute(FlinkBatchPortablePipelineTranslator.java:195)
~[?:?]
at
org.apache.beam.runners.flink.FlinkPipelineRunner.runPipelineWithTranslator(FlinkPipelineRunner.java:132)
~[?:?]
at
org.apache.beam.runners.flink.FlinkPipelineRunner.run(FlinkPipelineRunner.java:99)
~[?:?]
at
org.apache.beam.runners.flink.FlinkPipelineRunner.main(FlinkPipelineRunner.java:210)
~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source) ~[?:?]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
~[flink-dist-1.16.3.jar:1.16.3]
... 7 more
Caused by: java.lang.RuntimeException:
org.apache.flink.runtime.client.JobInitializationException: Could not start the
JobMaster.
at
org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:75)
~[flink-dist-1.16.3.jar:1.16.3]
at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown
Source) ~[?:?]
at java.util.concurrent.CompletableFuture$Completion.exec(Unknown
Source) ~[?:?]
at java.util.concurrent.ForkJoinTask.doExec(Unknown Source) ~[?:?]
at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown
Source) ~[?:?]
at java.util.concurrent.ForkJoinPool.scan(Unknown Source) ~[?:?]
at java.util.concurrent.ForkJoinPool.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source) ~[?:?]
Caused by: org.apache.flink.runtime.client.JobInitializationException: Could
not start the JobMaster.
at
org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97)
~[flink-dist-1.16.3.jar:1.16.3]
at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown
Source) ~[?:?]
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)
~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
~[?:?]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown
Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
~[?:?]
... 1 more
Caused by: java.util.concurrent.CompletionException:
java.lang.IllegalStateException: At the moment, adaptive batch scheduler
requires batch workloads to be executed with types of all edges being BLOCKING.
To do that, you need to configure 'execution.batch-shuffle-mode' to
'ALL_EXCHANGES_BLOCKING'.
at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown
Source) ~[?:?]
at java.util.concurrent.CompletableFuture.completeThrowable(Unknown
Source) ~[?:?]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown
Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
~[?:?]
... 1 more
Caused by: java.lang.IllegalStateException: At the moment, adaptive batch
scheduler requires batch workloads to be executed with types of all edges being
BLOCKING. To do that, you need to configure 'execution.batch-shuffle-mode' to
'ALL_EXCHANGES_BLOCKING'.
at
org.apache.flink.util.Preconditions.checkState(Preconditions.java:193)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.runtime.scheduler.adaptivebatch.AdaptiveBatchSchedulerFactory.checkAllExchangesBlocking(AdaptiveBatchSchedulerFactory.java:236)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.runtime.scheduler.adaptivebatch.AdaptiveBatchSchedulerFactory.createInstance(AdaptiveBatchSchedulerFactory.java:104)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:119)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:369)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:346)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:123)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:95)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112)
~[flink-dist-1.16.3.jar:1.16.3]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown
Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
~[?:?]
... 1 more
2024-03-28 16:10:33,228 ERROR
org.apache.flink.runtime.webmonitor.handlers.JarRunHandler [] - Exception
occurred in REST handler: Could not execute application.
2024-03-29 07:36:22,604 WARN
org.apache.flink.runtime.webmonitor.handlers.JarRunHandler [] - Configuring
the job submission via query parameters is deprecated. Please migrate to
submitting a JSON request instead.
2024-03-29 07:36:22,991 INFO org.apache.flink.client.ClientUtils
[] - Starting program (detached: true)
2024-03-29 07:36:25,277 WARN
software.amazon.awssdk.regions.internal.util.EC2MetadataUtils [] - Unable to
retrieve the requested metadata.
2024-03-29 07:36:25,555 WARN
org.apache.beam.runners.jobsubmission.JobServerDriver [] - Exception
during job server creation
java.io.IOException: Failed to bind to address 0.0.0.0/0.0.0.0:8098
at
org.apache.beam.vendor.grpc.v1p48p1.io.grpc.netty.NettyServer.start(NettyServer.java:328)
~[1ef4c38f-8d3d-4e78-b241-a9f2fb579744_beam.jar:?]
at
org.apache.beam.vendor.grpc.v1p48p1.io.grpc.internal.ServerImpl.start(ServerImpl.java:183)
~[1ef4c38f-8d3d-4e78-b241-a9f2fb579744_beam.jar:?]
at
org.apache.beam.vendor.grpc.v1p48p1.io.grpc.internal.ServerImpl.start(ServerImpl.java:92)
~[1ef4c38f-8d3d-4e78-b241-a9f2fb579744_beam.jar:?]
at
org.apache.beam.sdk.fn.server.ServerFactory$InetSocketAddressServerFactory.createServer(ServerFactory.java:162)
~[1ef4c38f-8d3d-4e78-b241-a9f2fb579744_beam.jar:?]
at
org.apache.beam.sdk.fn.server.ServerFactory$InetSocketAddressServerFactory.create(ServerFactory.java:145)
~[1ef4c38f-8d3d-4e78-b241-a9f2fb579744_beam.jar:?]
at
org.apache.beam.sdk.fn.server.GrpcFnServer.create(GrpcFnServer.java:110)
~[1ef4c38f-8d3d-4e78-b241-a9f2fb579744_beam.jar:?]
at
org.apache.beam.runners.jobsubmission.JobServerDriver.createArtifactStagingService(JobServerDriver.java:257)
~[1ef4c38f-8d3d-4e78-b241-a9f2fb579744_beam.jar:?]
at
org.apache.beam.runners.jobsubmission.JobServerDriver.createJobService(JobServerDriver.java:58)
~[1ef4c38f-8d3d-4e78-b241-a9f2fb579744_beam.jar:?]
at
org.apache.beam.runners.jobsubmission.JobServerDriver.createJobServer(JobServerDriver.java:229)
~[1ef4c38f-8d3d-4e78-b241-a9f2fb579744_beam.jar:?]
at
org.apache.beam.runners.jobsubmission.JobServerDriver.run(JobServerDriver.java:176)
~[1ef4c38f-8d3d-4e78-b241-a9f2fb579744_beam.jar:?]
at
org.apache.beam.runners.flink.FlinkJobServerDriver.main(FlinkJobServerDriver.java:77)
~[1ef4c38f-8d3d-4e78-b241-a9f2fb579744_beam.jar:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source) ~[?:?]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:98)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:70)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:107)
~[flink-dist-1.16.3.jar:1.16.3]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown
Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by:
org.apache.beam.vendor.grpc.v1p48p1.io.netty.channel.unix.Errors$NativeIoException:
bind(..) failed: Address already in use
2024-03-29 07:36:25,558 ERROR
org.apache.flink.runtime.webmonitor.handlers.JarRunHandler [] - Exception
occurred in REST handler: No jobs included in application.
2024-03-29 07:36:26,218 WARN
org.apache.flink.runtime.webmonitor.handlers.JarRunHandler [] - Configuring
the job submission via query parameters is deprecated. Please migrate to
submitting a JSON request instead.
2024-03-29 07:36:26,246 INFO org.apache.flink.client.ClientUtils
[] - Starting program (detached: true)
2024-03-29 07:36:27,648 WARN
software.amazon.awssdk.regions.internal.util.EC2MetadataUtils [] - Unable to
retrieve the requested metadata.
2024-03-29 07:36:27,905 WARN
org.apache.beam.runners.jobsubmission.JobServerDriver [] - Exception
during job server creation
java.io.IOException: Failed to bind to address 0.0.0.0/0.0.0.0:8098
at
org.apache.beam.vendor.grpc.v1p48p1.io.grpc.netty.NettyServer.start(NettyServer.java:328)
~[1ef4c38f-8d3d-4e78-b241-a9f2fb579744_beam.jar:?]
at
org.apache.beam.vendor.grpc.v1p48p1.io.grpc.internal.ServerImpl.start(ServerImpl.java:183)
~[1ef4c38f-8d3d-4e78-b241-a9f2fb579744_beam.jar:?]
at
org.apache.beam.vendor.grpc.v1p48p1.io.grpc.internal.ServerImpl.start(ServerImpl.java:92)
~[1ef4c38f-8d3d-4e78-b241-a9f2fb579744_beam.jar:?]
at
org.apache.beam.sdk.fn.server.ServerFactory$InetSocketAddressServerFactory.createServer(ServerFactory.java:162)
~[1ef4c38f-8d3d-4e78-b241-a9f2fb579744_beam.jar:?]
at
org.apache.beam.sdk.fn.server.ServerFactory$InetSocketAddressServerFactory.create(ServerFactory.java:145)
~[1ef4c38f-8d3d-4e78-b241-a9f2fb579744_beam.jar:?]
at
org.apache.beam.sdk.fn.server.GrpcFnServer.create(GrpcFnServer.java:110)
~[1ef4c38f-8d3d-4e78-b241-a9f2fb579744_beam.jar:?]
at
org.apache.beam.runners.jobsubmission.JobServerDriver.createArtifactStagingService(JobServerDriver.java:257)
~[1ef4c38f-8d3d-4e78-b241-a9f2fb579744_beam.jar:?]
at
org.apache.beam.runners.jobsubmission.JobServerDriver.createJobService(JobServerDriver.java:58)
~[1ef4c38f-8d3d-4e78-b241-a9f2fb579744_beam.jar:?]
at
org.apache.beam.runners.jobsubmission.JobServerDriver.createJobServer(JobServerDriver.java:229)
~[1ef4c38f-8d3d-4e78-b241-a9f2fb579744_beam.jar:?]
at
org.apache.beam.runners.jobsubmission.JobServerDriver.run(JobServerDriver.java:176)
~[1ef4c38f-8d3d-4e78-b241-a9f2fb579744_beam.jar:?]
at
org.apache.beam.runners.flink.FlinkJobServerDriver.main(FlinkJobServerDriver.java:77)
~[1ef4c38f-8d3d-4e78-b241-a9f2fb579744_beam.jar:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source) ~[?:?]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:98)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:70)
~[flink-dist-1.16.3.jar:1.16.3]
at
org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:107)
~[flink-dist-1.16.3.jar:1.16.3]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown
Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by:
org.apache.beam.vendor.grpc.v1p48p1.io.netty.channel.unix.Errors$NativeIoException:
bind(..) failed: Address already in use
2024-03-29 07:36:27,907 ERROR
org.apache.flink.runtime.webmonitor.handlers.JarRunHandler [] - Exception
occurred in REST handler: No jobs included in application.
