Hi,
I've recently migrated from Flink 1.8 to Flink 1.10
And when I start the job using YarnClusterDescriptor.deployJobCluster
method everything works fine.
However when I start the job from shell script, the job fails with messages:
*Shell script reports:*
Cluster specification: ClusterSpecification{masterMemoryMB=1024,
taskManagerMemoryMB=12000, slotsPerTaskManager=3}
YarnClusterDescriptor Deployment took more than 60 seconds. Please check
if the requested resources are available in the YARN cluster
*1. The server has twice more than that, and on flink 1.8 this
configuration works, why when switching to 1.10 it is not enough resources?*
*yarn log content of the job reports:*
2020-04-08 14:31:02,558 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting
YarnJobClusterEntrypoint (Version: 1.10.0, Rev:aa4eb8f, Date:07.02.202
0 @ 19:18:19 CET)
2020-04-08 14:31:02,558 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS
current user: yarn
2020-04-08 14:31:03,092 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Current
Hadoop/Kerberos user: erm_user
2020-04-08 14:31:03,092 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: Java
HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.112-b15
2020-04-08 14:31:03,092 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum
heap size: 406 MiBytes
2020-04-08 14:31:03,092 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
JAVA_HOME: /usr/jdk64/jdk1.8.0_112
2020-04-08 14:31:03,093 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Hadoop
version: 2.7.5
2020-04-08 14:31:03,093 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM
Options:
*2020-04-08 14:31:03,093 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
-Xms424m2020-04-08 14:31:03,094 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
-Xmx424m*
2020-04-08 14:31:03,094 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
-Dlog.file=/hadoop/yarn/log/application_1586286375485_0025/container_e82_1586286375485_0025_01_000001/jobmanager.log
2020-04-08 14:31:03,094 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
-Dlog4j.configuration=file:log4j.properties
2020-04-08 14:31:03,094 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Program
Arguments: (none)
*2. Why ClusterEnterpoint reports -xmx424m ?*
*3. When I start the job YarnClusterDescriptor.deployJobCluster it reports
the amount of memory assigned to the task manager, *
* ClusterEnterpoint reports -xmx424m is responsible for?*
Second suspicious message in log is:
2020-04-08 16:28:50,840 WARN org.apache.zookeeper.ClientCnxn
- *SASL configuration failed:
javax.security.auth.login.LoginException: No JAAS configuration section
named 'Client' was found in specified JAAS configuration file*:
'/hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026/jaas-1348005722200054084.conf'.
Will continue connection to Zookeeper server without SASL authentication,
if Zookeeper server allows it.
*4. What leads to this exception and how am I supposed to configure JAAS
section named Client?*
Third suspicious message, though most likely it is an outcome of something
being incorrectly configured:
2020-04-08 16:28:52,115 INFO
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot
serve slot request, no ResourceManager connected. Adding as pending request
[SlotRequestId{7a4da93f1f0bed92ccdbd707dfb47b7f}]
2020-04-08 16:28:52,120 INFO
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot
serve slot request, no ResourceManager connected. Adding as pending request
[SlotRequestId{3642e93314a78205854b7dfee80ea1a7}]
2020-04-08 16:28:52,121 INFO
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot
serve slot request, no ResourceManager connected. Adding as pending request
[SlotRequestId{c632edc6775f52762cb5a981a4109b89}]
2020-04-08 16:28:52,125 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Connecting to ResourceManager
akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal
:39331/user/resourcemanager(00000000000000000000000000000000)
2020-04-08 16:28:52,130 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Could not resolve ResourceManager address
akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager,
retrying in 10000 ms: Could not connect to rpc endpoint under address
akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal
:39331/user/resourcemanager..
5. *What can be the reason of failures to connect ResourceManager (if with
flink 1.8 the job didn't have such issues, it's not a firewall issue or
lack of resources)*?
PS
The whole yarn log is attached.
Container: container_e82_1586286375485_0026_01_000001 on ip-172-31-65-130.eu-central-1.compute.internal_45454
LogAggregationType: AGGREGATED
=============================================================================================================
LogType:directory.info
LogLastModifiedTime:Wed Apr 08 16:28:47 +0000 2020
LogLength:3598
LogContents:
ls -l:
total 20
-rw-r--r-- 1 yarn hadoop 69 Apr 8 16:28 container_tokens
-rwx------ 1 yarn hadoop 657 Apr 8 16:28 default_container_executor_session.sh
-rwx------ 1 yarn hadoop 711 Apr 8 16:28 default_container_executor.sh
lrwxrwxrwx 1 yarn hadoop 128 Apr 8 16:28 erm-monthlyperformance-production.jar -> /hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026/filecache/19/erm-monthlyperformance-production.jar
lrwxrwxrwx 1 yarn hadoop 160 Apr 8 16:28 flink-conf.yaml -> /hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026/filecache/11/application_1586286375485_0026-flink-conf.yaml1289844950966534911.tmp
lrwxrwxrwx 1 yarn hadoop 117 Apr 8 16:28 flink-dist_2.12-1.10.0.jar -> /hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026/filecache/15/flink-dist_2.12-1.10.0.jar
lrwxrwxrwx 1 yarn hadoop 144 Apr 8 16:28 job.graph -> /hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026/filecache/14/application_1586286375485_00261000897891880198941.tmp
-rwx------ 1 yarn hadoop 7601 Apr 8 16:28 launch_container.sh
drwxr-xr-x 2 yarn hadoop 138 Apr 8 16:28 lib
lrwxrwxrwx 1 yarn hadoop 107 Apr 8 16:28 log4j.properties -> /hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026/filecache/16/log4j.properties
drwxr-xr-x 2 yarn hadoop 24 Apr 8 16:28 plugins
drwx--x--- 2 yarn hadoop 6 Apr 8 16:28 tmp
find -L . -maxdepth 5 -ls:
1078728690 4 drwx--x--- 5 yarn hadoop 4096 Apr 8 16:28 .
3221353006 0 drwx--x--- 2 yarn hadoop 6 Apr 8 16:28 ./tmp
1078728700 4 -rw-r--r-- 1 yarn hadoop 69 Apr 8 16:28 ./container_tokens
1078744099 4 -rw-r--r-- 1 yarn hadoop 12 Apr 8 16:28 ./.container_tokens.crc
1078747968 8 -rwx------ 1 yarn hadoop 7601 Apr 8 16:28 ./launch_container.sh
1078760307 4 -rw-r--r-- 1 yarn hadoop 68 Apr 8 16:28 ./.launch_container.sh.crc
1078762482 4 -rwx------ 1 yarn hadoop 657 Apr 8 16:28 ./default_container_executor_session.sh
1078767543 4 -rw-r--r-- 1 yarn hadoop 16 Apr 8 16:28 ./.default_container_executor_session.sh.crc
1078768452 4 -rwx------ 1 yarn hadoop 711 Apr 8 16:28 ./default_container_executor.sh
1078771057 4 -rw-r--r-- 1 yarn hadoop 16 Apr 8 16:28 ./.default_container_executor.sh.crc
891954 0 drwxr-xr-x 2 yarn hadoop 138 Apr 8 16:28 ./lib
2149767169 17532 -r-x------ 1 yarn hadoop 17951938 Apr 8 16:28 ./lib/flink-table_2.12-1.10.0.jar
566226 12 -r-x------ 1 yarn hadoop 9931 Apr 8 16:28 ./lib/slf4j-log4j12-1.7.15.jar
2149767176 480 -r-x------ 1 yarn hadoop 489884 Apr 8 16:28 ./lib/log4j-1.2.17.jar
568204 19624 -r-x------ 1 yarn hadoop 20091194 Apr 8 16:28 ./lib/flink-table-blink_2.12-1.10.0.jar
891942 4 -r-x------ 1 yarn hadoop 1939 Apr 8 16:28 ./log4j.properties
568210 224 -r-x------ 1 yarn hadoop 225530 Apr 8 16:28 ./job.graph
2149767179 138168 -r-x------ 1 yarn hadoop 141483649 Apr 8 16:28 ./erm-monthlyperformance-production.jar
2149767173 98796 -r-x------ 1 yarn hadoop 101167027 Apr 8 16:28 ./flink-dist_2.12-1.10.0.jar
1078777596 0 drwxr-xr-x 2 yarn hadoop 24 Apr 8 16:28 ./plugins
891945 4 -r-x------ 1 yarn hadoop 654 Apr 8 16:28 ./plugins/README.txt
2149767038 4 -r-x------ 1 yarn hadoop 808 Apr 8 16:28 ./flink-conf.yaml
broken symlinks(find -L . -maxdepth 5 -type l -ls):
End of LogType:directory.info
*******************************************************************************
Container: container_e82_1586286375485_0026_01_000001 on ip-172-31-65-130.eu-central-1.compute.internal_45454
LogAggregationType: AGGREGATED
=============================================================================================================
LogType:prelaunch.out
LogLastModifiedTime:Wed Apr 08 16:28:47 +0000 2020
LogLength:100
LogContents:
Setting up env variables
Setting up job resources
Copying debugging information
Launching container
End of LogType:prelaunch.out
******************************************************************************
End of LogType:jobmanager.out
*******************************************************************************
End of LogType:prelaunch.err
******************************************************************************
Container: container_e82_1586286375485_0026_01_000001 on ip-172-31-65-130.eu-central-1.compute.internal_45454
LogAggregationType: AGGREGATED
=============================================================================================================
LogType:jobmanager.err
LogLastModifiedTime:Wed Apr 08 16:28:47 +0000 2020
LogLength:732
LogContents:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026/filecache/19/erm-monthlyperformance-production.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026/filecache/10/slf4j-log4j12-1.7.15.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
End of LogType:jobmanager.err
*******************************************************************************
Container: container_e82_1586286375485_0026_01_000001 on ip-172-31-65-130.eu-central-1.compute.internal_45454
LogAggregationType: AGGREGATED
=============================================================================================================
LogType:jobmanager.log
LogLastModifiedTime:Wed Apr 08 16:33:10 +0000 2020
LogLength:72924
LogContents:
2020-04-08 16:28:47,446 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
2020-04-08 16:28:47,447 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnJobClusterEntrypoint (Version: 1.10.0, Rev:aa4eb8f, Date:07.02.2020 @ 19:18:19 CET)
2020-04-08 16:28:47,447 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS current user: yarn
2020-04-08 16:28:47,831 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-04-08 16:28:47,918 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Current Hadoop/Kerberos user: erm_user
2020-04-08 16:28:47,919 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.112-b15
2020-04-08 16:28:47,919 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum heap size: 406 MiBytes
2020-04-08 16:28:47,919 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JAVA_HOME: /usr/jdk64/jdk1.8.0_112
2020-04-08 16:28:47,920 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Hadoop version: 2.7.5
2020-04-08 16:28:47,920 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM Options:
2020-04-08 16:28:47,920 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xms424m
2020-04-08 16:28:47,920 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xmx424m
2020-04-08 16:28:47,920 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog.file=/hadoop/yarn/log/application_1586286375485_0026/container_e82_1586286375485_0026_01_000001/jobmanager.log
2020-04-08 16:28:47,920 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog4j.configuration=file:log4j.properties
2020-04-08 16:28:47,920 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Program Arguments: (none)
2020-04-08 16:28:47,921 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Classpath: :erm-monthlyperformance-production.jar:lib/flink-table-blink_2.12-1.10.0.jar:lib/flink-table_2.12-1.10.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.15.jar:flink-dist_2.12-1.10.0.jar:flink-conf.yaml:job.graph::/usr/hdp/2.6.5.0-292/hadoop/conf:/usr/hdp/2.6.5.0-292/hadoop/azure-data-lake-store-sdk-2.2.5.jar:/usr/hdp/2.6.5.0-292/hadoop/gcs-connector-1.8.1.2.6.5.0-292-shaded.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-annotations-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-annotations.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-auth-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-auth.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-aws-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-aws.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-azure-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-azure-datalake-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-azure-datalake.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-azure.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-common-2.7.3.2.6.5.0-292-tests.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-common-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-common-tests.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-common.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-nfs-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-nfs.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/ojdbc6.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/nimbus-jose-jwt-4.41.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/ranger-hdfs-plugin-shim-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-annotations-2.2.3.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/ranger-plugin-classloader-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-core-2.2.3.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/ranger-yarn-plugin-shim-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/xmlenc-0.52.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/activation-1.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jettison-1.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jetty-sslengine-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/api-util-1.0.0-M20.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/asm-3.2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/xz-1.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/avro-1.7.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-databind-2.2.3.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/aws-java-sdk-core-1.10.6.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/aws-java-sdk-kms-1.10.6.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/joda-time-2.9.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/azure-keyvault-core-0.8.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jsch-0.1.54.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/azure-storage-5.4.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/json-smart-1.3.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-cli-1.2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jsp-api-2.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-codec-1.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-collections-3.2.2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jsr305-3.0.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-compress-1.4.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-configuration-1.6.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/junit-4.11.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-digester-1.8.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-io-2.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/log4j-1.2.17.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-lang-2.6.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/mockito-all-1.8.5.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-lang3-3.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/netty-3.6.2.Final.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-logging-1.1.3.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/paranamer-2.3.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-math3-3.1.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-net-3.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/curator-client-2.7.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/servlet-api-2.5.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/curator-framework-2.7.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-api-1.7.10.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/curator-recipes-2.7.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/gson-2.2.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/guava-11.0.2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/hamcrest-core-1.3.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/httpclient-4.5.2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/httpcore-4.4.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/stax-api-1.0-2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jcip-annotations-1.0-1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jersey-core-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jersey-json-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/zookeeper-3.4.6.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jersey-server-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jets3t-0.9.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/hadoop-lzo-0.6.0.2.6.5.0-292-javadoc.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/hadoop-lzo-0.6.0.2.6.5.0-292-sources.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/hadoop-lzo-0.6.0.2.6.5.0-292.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.3.2.6.5.0-292-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs.jar:/usr/hdp/current/hadoop-hdfs-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-daemon-1.0.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-hdfs-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-hdfs-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-all-4.0.52.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okhttp-2.7.5.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okio-1.6.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xercesImpl-2.9.1.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xml-apis-1.3.04.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-pluginstorage-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-pluginstorage.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy.jar:/usr/hdp/current/hadoop-yarn-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/aopalliance-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsch-0.1.54.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-sslengine-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/azure-keyvault-core-0.8.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/azure-storage-5.4.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/json-smart-1.3.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/javassist-3.18.1-GA.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/javax.inject-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-guice-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/metrics-core-3.0.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-lang3-3.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/nimbus-jose-jwt-4.41.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/objenesis-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/fst-2.24.jar:/usr/hdp/current/hadoop-yarn-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-servlet-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-yarn-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpclient-4.5.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpcore-4.4.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/xz-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jcip-annotations-1.0-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-client-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.6.5.0-292-tests.jar:/usr/hdp/current/ext/hadoop/*
2020-04-08 16:28:47,921 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
2020-04-08 16:28:47,922 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT]
2020-04-08 16:28:47,925 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - YARN daemon is running as: erm_user Yarn client user obtainer: erm_user
2020-04-08 16:28:47,928 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.process.size, 12000m
2020-04-08 16:28:47,928 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: web.timeout, 600000
2020-04-08 16:28:47,929 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.execution.failover-strategy, region
2020-04-08 16:28:47,929 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, application_1586286375485_0026
2020-04-08 16:28:47,929 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, localhost
2020-04-08 16:28:47,929 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: execution.target, yarn-per-job
2020-04-08 16:28:47,929 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.network.request-backoff.max, 30000
2020-04-08 16:28:47,929 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123
2020-04-08 16:28:47,929 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: execution.savepoint.ignore-unclaimed-state, false
2020-04-08 16:28:47,930 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: execution.attached, true
2020-04-08 16:28:47,930 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: internal.cluster.execution-mode, NORMAL
2020-04-08 16:28:47,930 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: execution.shutdown-on-attached-exit, false
2020-04-08 16:28:47,930 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: pipeline.jars, [file:/home/erm_user/erm/mprnew/erm-monthlyperformance-production.jar]
2020-04-08 16:28:47,930 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 3
2020-04-08 16:28:47,930 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 3
2020-04-08 16:28:47,930 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: akka.ask.timeout, 600s
2020-04-08 16:28:47,933 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.application.name, Intraregional Volume Job - Flink session cluster
2020-04-08 16:28:47,933 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: $internal.yarn.log-config-file, /home/erm_user/erm/flink/conf/log4j.properties
2020-04-08 16:28:47,934 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m
2020-04-08 16:28:47,951 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'web.port' instead of proper key 'rest.bind-port'
2020-04-08 16:28:47,958 INFO org.apache.flink.runtime.clusterframework.BootstrapTools - Setting directories for temporary files to: /hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026
2020-04-08 16:28:47,972 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnJobClusterEntrypoint.
2020-04-08 16:28:47,972 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem.
2020-04-08 16:28:48,109 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to erm_user (auth:SIMPLE)
2020-04-08 16:28:48,118 INFO org.apache.flink.runtime.security.modules.JaasModule - Jaas file will be created as /hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026/jaas-1348005722200054084.conf.
2020-04-08 16:28:48,125 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services.
2020-04-08 16:28:48,141 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start actor system at ip-172-31-65-130.eu-central-1.compute.internal:0
2020-04-08 16:28:49,156 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2020-04-08 16:28:49,185 INFO akka.remote.Remoting - Starting remoting
2020-04-08 16:28:49,329 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331]
2020-04-08 16:28:49,580 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system started at akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331
2020-04-08 16:28:49,595 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'web.port' instead of proper key 'rest.port'
2020-04-08 16:28:49,603 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026/blobStore-57f1ca36-bdc2-4d49-b6fa-845536cd453e
2020-04-08 16:28:49,605 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:41429 - max concurrent requests: 50 - max backlog: 1000
2020-04-08 16:28:49,616 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported.
2020-04-08 16:28:49,619 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start actor system at ip-172-31-65-130.eu-central-1.compute.internal:0
2020-04-08 16:28:49,643 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2020-04-08 16:28:49,647 INFO akka.remote.Remoting - Starting remoting
2020-04-08 16:28:49,655 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink-metr...@ip-172-31-65-130.eu-central-1.compute.internal:34831]
2020-04-08 16:28:49,702 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system started at akka.tcp://flink-metr...@ip-172-31-65-130.eu-central-1.compute.internal:34831
2020-04-08 16:28:49,708 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.metrics.dump.MetricQueryService at akka://flink-metrics/user/MetricQueryService .
2020-04-08 16:28:49,791 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'web.port' instead of proper key 'rest.bind-port'
2020-04-08 16:28:49,792 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Upload directory /tmp/flink-web-77e0318d-c686-4922-9299-a86496ed0d0d/flink-web-upload does not exist.
2020-04-08 16:28:49,794 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Created directory /tmp/flink-web-77e0318d-c686-4922-9299-a86496ed0d0d/flink-web-upload for file uploads.
2020-04-08 16:28:49,811 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Starting rest endpoint.
2020-04-08 16:28:50,061 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component log file: /hadoop/yarn/log/application_1586286375485_0026/container_e82_1586286375485_0026_01_000001/jobmanager.log
2020-04-08 16:28:50,061 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component stdout file: /hadoop/yarn/log/application_1586286375485_0026/container_e82_1586286375485_0026_01_000001/jobmanager.out
2020-04-08 16:28:50,209 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Rest endpoint listening at ip-172-31-65-130.eu-central-1.compute.internal:46266
2020-04-08 16:28:50,210 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - http://ip-172-31-65-130.eu-central-1.compute.internal:46266 was granted leadership with leaderSessionID=00000000-0000-0000-0000-000000000000
2020-04-08 16:28:50,210 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Web frontend listening at http://ip-172-31-65-130.eu-central-1.compute.internal:46266.
2020-04-08 16:28:50,299 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.yarn.YarnResourceManager at akka://flink/user/resourcemanager .
2020-04-08 16:28:50,317 INFO org.apache.flink.runtime.clusterframework.TaskExecutorProcessUtils - The derived from fraction jvm overhead memory (1.172gb (1258291218 bytes)) is greater than its max value 1024.000mb (1073741824 bytes), max value will be used instead
2020-04-08 16:28:50,318 INFO org.apache.flink.runtime.clusterframework.TaskExecutorProcessUtils - The derived from fraction network memory (1.063gb (1140850705 bytes)) is greater than its max value 1024.000mb (1073741824 bytes), max value will be used instead
2020-04-08 16:28:50,321 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.process.size, 12000m
2020-04-08 16:28:50,321 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: web.timeout, 600000
2020-04-08 16:28:50,321 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.execution.failover-strategy, region
2020-04-08 16:28:50,322 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, application_1586286375485_0026
2020-04-08 16:28:50,322 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, localhost
2020-04-08 16:28:50,322 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: execution.target, yarn-per-job
2020-04-08 16:28:50,322 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.network.request-backoff.max, 30000
2020-04-08 16:28:50,322 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123
2020-04-08 16:28:50,322 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: execution.savepoint.ignore-unclaimed-state, false
2020-04-08 16:28:50,322 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: execution.attached, true
2020-04-08 16:28:50,322 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: internal.cluster.execution-mode, NORMAL
2020-04-08 16:28:50,322 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: execution.shutdown-on-attached-exit, false
2020-04-08 16:28:50,322 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: pipeline.jars, [file:/home/erm_user/erm/mprnew/erm-monthlyperformance-production.jar]
2020-04-08 16:28:50,323 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 3
2020-04-08 16:28:50,323 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 3
2020-04-08 16:28:50,323 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: akka.ask.timeout, 600s
2020-04-08 16:28:50,323 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.application.name, Intraregional Volume Job - Flink session cluster
2020-04-08 16:28:50,323 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: $internal.yarn.log-config-file, /home/erm_user/erm/flink/conf/log4j.properties
2020-04-08 16:28:50,323 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m
2020-04-08 16:28:50,413 INFO org.apache.flink.runtime.dispatcher.runner.JobDispatcherLeaderProcess - Start JobDispatcherLeaderProcess.
2020-04-08 16:28:50,419 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.MiniDispatcher at akka://flink/user/dispatcher .
2020-04-08 16:28:50,459 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/jobmanager_0 .
2020-04-08 16:28:50,467 INFO org.apache.flink.runtime.jobmaster.JobMaster - Initializing job IntraregionalVolumeJob_month_2019-11_at_2019-12-05T00:00:00Z_by_server-console (6a20ebe846763f2aebd14dfaa3a75f3c).
2020-04-08 16:28:50,548 INFO org.apache.flink.runtime.jobmaster.JobMaster - Using restart back off time strategy NoRestartBackoffTimeStrategy for IntraregionalVolumeJob_month_2019-11_at_2019-12-05T00:00:00Z_by_server-console (6a20ebe846763f2aebd14dfaa3a75f3c).
2020-04-08 16:28:50,595 INFO org.apache.flink.runtime.jobmaster.JobMaster - Running initialization on master for job IntraregionalVolumeJob_month_2019-11_at_2019-12-05T00:00:00Z_by_server-console (6a20ebe846763f2aebd14dfaa3a75f3c).
2020-04-08 16:28:50,617 INFO org.upu.erm.calculation.mpr.sources.HBaseResultInputFormat - Configuring.
2020-04-08 16:28:50,660 INFO org.upu.erm.calculation.common.services.ServiceLocator - Adding Hadoop configuration resource: 'conf-hdp/aws-int/core-site.xml'.
2020-04-08 16:28:50,660 INFO org.upu.erm.calculation.common.services.ServiceLocator - Adding Hadoop configuration resource: 'conf-hdp/aws-int/hdfs-site.xml'.
2020-04-08 16:28:50,660 INFO org.upu.erm.calculation.common.services.ServiceLocator - Adding Hadoop configuration resource: 'conf-hdp/aws-int/yarn-site.xml'.
2020-04-08 16:28:50,660 INFO org.upu.erm.calculation.common.services.ServiceLocator - Adding Hadoop configuration resource: 'conf-hdp/aws-int/capacity-scheduler.xml'.
2020-04-08 16:28:50,660 INFO org.upu.erm.calculation.common.services.ServiceLocator - Adding Hadoop configuration resource: 'conf-hdp/aws-int/mapred-site.xml'.
2020-04-08 16:28:50,661 INFO org.upu.erm.calculation.common.services.ServiceLocator - Adding Hadoop configuration resource: 'conf-hdp/aws-int/hbase-site.xml'.
2020-04-08 16:28:50,661 INFO org.upu.erm.calculation.common.services.ServiceLocator - Adding Hadoop configuration resource: 'conf-hdp/aws-int/hbase-policy.xml'.
2020-04-08 16:28:50,813 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Process identifier=hconnection-0x7dd9e145 connecting to ZooKeeper ensemble=zk1.hdp-int.aws.paranor.ch:2181,zk2.hdp-int.aws.paranor.ch:2181,zk3.hdp-int.aws.paranor.ch:2181
2020-04-08 16:28:50,821 INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2020-04-08 16:28:50,821 INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=ip-172-31-65-130.eu-central-1.compute.internal
2020-04-08 16:28:50,821 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_112
2020-04-08 16:28:50,821 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
2020-04-08 16:28:50,821 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/jdk64/jdk1.8.0_112/jre
2020-04-08 16:28:50,821 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=:erm-monthlyperformance-production.jar:lib/flink-table-blink_2.12-1.10.0.jar:lib/flink-table_2.12-1.10.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.15.jar:flink-dist_2.12-1.10.0.jar:flink-conf.yaml:job.graph::/usr/hdp/2.6.5.0-292/hadoop/conf:/usr/hdp/2.6.5.0-292/hadoop/azure-data-lake-store-sdk-2.2.5.jar:/usr/hdp/2.6.5.0-292/hadoop/gcs-connector-1.8.1.2.6.5.0-292-shaded.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-annotations-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-annotations.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-auth-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-auth.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-aws-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-aws.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-azure-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-azure-datalake-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-azure-datalake.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-azure.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-common-2.7.3.2.6.5.0-292-tests.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-common-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-common-tests.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-common.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-nfs-2.7.3.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/hadoop-nfs.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/ojdbc6.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/nimbus-jose-jwt-4.41.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/ranger-hdfs-plugin-shim-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-annotations-2.2.3.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/ranger-plugin-classloader-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-core-2.2.3.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/ranger-yarn-plugin-shim-0.7.0.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/xmlenc-0.52.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/activation-1.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jettison-1.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jetty-sslengine-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/api-util-1.0.0-M20.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/asm-3.2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/xz-1.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/avro-1.7.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-databind-2.2.3.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/aws-java-sdk-core-1.10.6.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/aws-java-sdk-kms-1.10.6.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/joda-time-2.9.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/azure-keyvault-core-0.8.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jsch-0.1.54.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/azure-storage-5.4.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/json-smart-1.3.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-cli-1.2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jsp-api-2.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-codec-1.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-collections-3.2.2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jsr305-3.0.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-compress-1.4.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-configuration-1.6.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/junit-4.11.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-digester-1.8.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-io-2.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/log4j-1.2.17.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-lang-2.6.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/mockito-all-1.8.5.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-lang3-3.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/netty-3.6.2.Final.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-logging-1.1.3.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/paranamer-2.3.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-math3-3.1.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/commons-net-3.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/curator-client-2.7.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/servlet-api-2.5.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/curator-framework-2.7.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-api-1.7.10.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/curator-recipes-2.7.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/gson-2.2.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/guava-11.0.2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/hamcrest-core-1.3.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/httpclient-4.5.2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/httpcore-4.4.4.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/stax-api-1.0-2.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jcip-annotations-1.0-1.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jersey-core-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jersey-json-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/zookeeper-3.4.6.2.6.5.0-292.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jersey-server-1.9.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/jets3t-0.9.0.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/hadoop-lzo-0.6.0.2.6.5.0-292-javadoc.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/hadoop-lzo-0.6.0.2.6.5.0-292-sources.jar:/usr/hdp/2.6.5.0-292/hadoop/lib/hadoop-lzo-0.6.0.2.6.5.0-292.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.3.2.6.5.0-292-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs.jar:/usr/hdp/current/hadoop-hdfs-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-daemon-1.0.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-hdfs-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-hdfs-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-all-4.0.52.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okhttp-2.7.5.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okio-1.6.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xercesImpl-2.9.1.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xml-apis-1.3.04.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-pluginstorage-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-pluginstorage.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy-2.7.3.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy.jar:/usr/hdp/current/hadoop-yarn-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/aopalliance-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsch-0.1.54.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-sslengine-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/azure-keyvault-core-0.8.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/azure-storage-5.4.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/json-smart-1.3.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/javassist-3.18.1-GA.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/javax.inject-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-guice-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/metrics-core-3.0.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-lang3-3.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/nimbus-jose-jwt-4.41.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/objenesis-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/fst-2.24.jar:/usr/hdp/current/hadoop-yarn-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-servlet-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-yarn-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpclient-4.5.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpcore-4.4.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/xz-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.6.5.0-292.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jcip-annotations-1.0-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-client-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.6.5.0-292-tests.jar:/usr/hdp/current/ext/hadoop/*
2020-04-08 16:28:50,821 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2020-04-08 16:28:50,821 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
2020-04-08 16:28:50,821 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
2020-04-08 16:28:50,821 INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2020-04-08 16:28:50,822 INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2020-04-08 16:28:50,822 INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=3.10.0-862.3.2.el7.x86_64
2020-04-08 16:28:50,822 INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=yarn
2020-04-08 16:28:50,822 INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/yarn
2020-04-08 16:28:50,822 INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026/container_e82_1586286375485_0026_01_000001
2020-04-08 16:28:50,823 INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zk1.hdp-int.aws.paranor.ch:2181,zk2.hdp-int.aws.paranor.ch:2181,zk3.hdp-int.aws.paranor.ch:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@6611fb89
2020-04-08 16:28:50,840 WARN org.apache.zookeeper.ClientCnxn - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026/jaas-1348005722200054084.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2020-04-08 16:28:50,842 INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server ip-172-31-65-62.eu-central-1.compute.internal/172.31.65.62:2181
2020-04-08 16:28:50,844 INFO org.apache.zookeeper.ClientCnxn - Socket connection established to ip-172-31-65-62.eu-central-1.compute.internal/172.31.65.62:2181, initiating session
2020-04-08 16:28:50,852 INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server ip-172-31-65-62.eu-central-1.compute.internal/172.31.65.62:2181, sessionid = 0x1715121cba703e2, negotiated timeout = 60000
2020-04-08 16:28:51,409 WARN org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
2020-04-08 16:28:51,437 INFO org.upu.erm.calculation.mpr.sources.HBaseResultInputFormat - Configuring.
2020-04-08 16:28:51,461 INFO org.upu.erm.calculation.common.services.ServiceLocator - Adding Hadoop configuration resource: 'conf-hdp/aws-int/core-site.xml'.
2020-04-08 16:28:51,461 INFO org.upu.erm.calculation.common.services.ServiceLocator - Adding Hadoop configuration resource: 'conf-hdp/aws-int/hdfs-site.xml'.
2020-04-08 16:28:51,461 INFO org.upu.erm.calculation.common.services.ServiceLocator - Adding Hadoop configuration resource: 'conf-hdp/aws-int/yarn-site.xml'.
2020-04-08 16:28:51,461 INFO org.upu.erm.calculation.common.services.ServiceLocator - Adding Hadoop configuration resource: 'conf-hdp/aws-int/capacity-scheduler.xml'.
2020-04-08 16:28:51,461 INFO org.upu.erm.calculation.common.services.ServiceLocator - Adding Hadoop configuration resource: 'conf-hdp/aws-int/mapred-site.xml'.
2020-04-08 16:28:51,461 INFO org.upu.erm.calculation.common.services.ServiceLocator - Adding Hadoop configuration resource: 'conf-hdp/aws-int/hbase-site.xml'.
2020-04-08 16:28:51,461 INFO org.upu.erm.calculation.common.services.ServiceLocator - Adding Hadoop configuration resource: 'conf-hdp/aws-int/hbase-policy.xml'.
2020-04-08 16:28:51,495 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Process identifier=hconnection-0x1fc9ce23 connecting to ZooKeeper ensemble=zk1.hdp-int.aws.paranor.ch:2181,zk2.hdp-int.aws.paranor.ch:2181,zk3.hdp-int.aws.paranor.ch:2181
2020-04-08 16:28:51,496 INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zk1.hdp-int.aws.paranor.ch:2181,zk2.hdp-int.aws.paranor.ch:2181,zk3.hdp-int.aws.paranor.ch:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@3d971a08
2020-04-08 16:28:51,497 WARN org.apache.zookeeper.ClientCnxn - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026/jaas-1348005722200054084.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2020-04-08 16:28:51,497 INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server ip-172-31-65-62.eu-central-1.compute.internal/172.31.65.62:2181
2020-04-08 16:28:51,498 INFO org.apache.zookeeper.ClientCnxn - Socket connection established to ip-172-31-65-62.eu-central-1.compute.internal/172.31.65.62:2181, initiating session
2020-04-08 16:28:51,502 INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server ip-172-31-65-62.eu-central-1.compute.internal/172.31.65.62:2181, sessionid = 0x1715121cba703e3, negotiated timeout = 60000
2020-04-08 16:28:51,522 INFO org.apache.flink.runtime.jobmaster.JobMaster - Successfully ran initialization on master in 927 ms.
2020-04-08 16:28:51,935 INFO org.apache.flink.addons.hbase.AbstractTableInputFormat - Created 2 splits
2020-04-08 16:28:51,935 INFO org.apache.flink.addons.hbase.AbstractTableInputFormat - created split (this=org.upu.erm.calculation.mpr.sources.HBaseResultInputFormat@5b7e0d1c)[0|[ip-172-31-65-130.eu-central-1.compute.internal:16020]|CM20191101|CM201911061520EG705996860JP]
2020-04-08 16:28:51,935 INFO org.apache.flink.addons.hbase.AbstractTableInputFormat - created split (this=org.upu.erm.calculation.mpr.sources.HBaseResultInputFormat@5b7e0d1c)[1|[ip-172-31-73-119.eu-central-1.compute.internal:16020]|CM201911061520EG705996860JP|CM20191201]
2020-04-08 16:28:52,060 INFO org.apache.flink.addons.hbase.AbstractTableInputFormat - Created 1 splits
2020-04-08 16:28:52,060 INFO org.apache.flink.addons.hbase.AbstractTableInputFormat - created split (this=org.upu.erm.calculation.mpr.sources.HBaseResultInputFormat@54ea73ea)[0|[ip-172-31-65-130.eu-central-1.compute.internal:16020]|DM20191101|DM20191201]
2020-04-08 16:28:52,074 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionStrategy - Start building failover regions.
2020-04-08 16:28:52,075 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionStrategy - Created 1 failover regions.
2020-04-08 16:28:52,076 INFO org.apache.flink.runtime.jobmaster.JobMaster - Using failover strategy org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionStrategy@73a3bc7c for IntraregionalVolumeJob_month_2019-11_at_2019-12-05T00:00:00Z_by_server-console (6a20ebe846763f2aebd14dfaa3a75f3c).
2020-04-08 16:28:52,081 INFO org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl - JobManager runner for job IntraregionalVolumeJob_month_2019-11_at_2019-12-05T00:00:00Z_by_server-console (6a20ebe846763f2aebd14dfaa3a75f3c) was granted leadership with session id 00000000-0000-0000-0000-000000000000 at akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/jobmanager_0.
2020-04-08 16:28:52,087 INFO org.apache.flink.runtime.jobmaster.JobMaster - Starting execution of job IntraregionalVolumeJob_month_2019-11_at_2019-12-05T00:00:00Z_by_server-console (6a20ebe846763f2aebd14dfaa3a75f3c) under job master id 00000000000000000000000000000000.
2020-04-08 16:28:52,089 INFO org.apache.flink.runtime.jobmaster.JobMaster - Starting scheduling with scheduling strategy [org.apache.flink.runtime.scheduler.strategy.LazyFromSourcesSchedulingStrategy]
2020-04-08 16:28:52,089 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job IntraregionalVolumeJob_month_2019-11_at_2019-12-05T00:00:00Z_by_server-console (6a20ebe846763f2aebd14dfaa3a75f3c) switched from state CREATED to RUNNING.
2020-04-08 16:28:52,100 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - CHAIN DataSource (MailEvent; EMC; 2019-11-01 - 2019-12-01; null - 1575504000000) -> FlatMap (SplitDuplicate) -> FlatMap (Create MailEvent) -> Filter (EventDateTimeRangeFilter) -> Filter (TrackingStatusesFilter) -> FlatMap (Get mail item by EMC event) -> Map (Map IntraregionalVolumeItem data set from EMC events) (1/3) (d7490088b7138689912daca0af5d25d9) switched from CREATED to SCHEDULED.
2020-04-08 16:28:52,100 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - CHAIN DataSource (MailEvent; EMC; 2019-11-01 - 2019-12-01; null - 1575504000000) -> FlatMap (SplitDuplicate) -> FlatMap (Create MailEvent) -> Filter (EventDateTimeRangeFilter) -> Filter (TrackingStatusesFilter) -> FlatMap (Get mail item by EMC event) -> Map (Map IntraregionalVolumeItem data set from EMC events) (2/3) (0554a0d7b895b211bb337b35198078fb) switched from CREATED to SCHEDULED.
2020-04-08 16:28:52,100 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - CHAIN DataSource (MailEvent; EMC; 2019-11-01 - 2019-12-01; null - 1575504000000) -> FlatMap (SplitDuplicate) -> FlatMap (Create MailEvent) -> Filter (EventDateTimeRangeFilter) -> Filter (TrackingStatusesFilter) -> FlatMap (Get mail item by EMC event) -> Map (Map IntraregionalVolumeItem data set from EMC events) (3/3) (a508b3876dc368f6e6efd01c6fac025b) switched from CREATED to SCHEDULED.
2020-04-08 16:28:52,100 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - CHAIN DataSource (MailEvent; EMD; 2019-11-01 - 2019-12-01; null - 1575504000000) -> FlatMap (SplitDuplicate) -> FlatMap (Create MailEvent) -> Filter (EventDateTimeRangeFilter) -> Filter (TrackingStatusesFilter) -> FlatMap (Get mail item by EMD event) -> Map (Map IntraregionalVolumeItem data set from EMD events) (1/3) (6bbd1ef730c4f88bed8bf9a14858a380) switched from CREATED to SCHEDULED.
2020-04-08 16:28:52,100 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - CHAIN DataSource (MailEvent; EMD; 2019-11-01 - 2019-12-01; null - 1575504000000) -> FlatMap (SplitDuplicate) -> FlatMap (Create MailEvent) -> Filter (EventDateTimeRangeFilter) -> Filter (TrackingStatusesFilter) -> FlatMap (Get mail item by EMD event) -> Map (Map IntraregionalVolumeItem data set from EMD events) (2/3) (4059581e386b5b251f2488c303705c94) switched from CREATED to SCHEDULED.
2020-04-08 16:28:52,100 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - CHAIN DataSource (MailEvent; EMD; 2019-11-01 - 2019-12-01; null - 1575504000000) -> FlatMap (SplitDuplicate) -> FlatMap (Create MailEvent) -> Filter (EventDateTimeRangeFilter) -> Filter (TrackingStatusesFilter) -> FlatMap (Get mail item by EMD event) -> Map (Map IntraregionalVolumeItem data set from EMD events) (3/3) (1e3821066e4f549ca7a373b2842dfce6) switched from CREATED to SCHEDULED.
2020-04-08 16:28:52,115 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{7a4da93f1f0bed92ccdbd707dfb47b7f}]
2020-04-08 16:28:52,120 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{3642e93314a78205854b7dfee80ea1a7}]
2020-04-08 16:28:52,121 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{c632edc6775f52762cb5a981a4109b89}]
2020-04-08 16:28:52,125 INFO org.apache.flink.runtime.jobmaster.JobMaster - Connecting to ResourceManager akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager(00000000000000000000000000000000)
2020-04-08 16:28:52,130 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:29:02,143 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:29:12,164 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:29:22,183 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:29:32,204 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:29:42,224 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:29:52,244 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:30:02,264 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:30:12,284 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:30:22,304 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:30:32,324 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:30:42,344 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:30:52,364 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:31:02,384 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:31:12,404 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:31:22,424 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:31:32,444 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:31:42,463 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:31:52,484 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:32:02,504 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:32:12,524 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:32:22,543 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:32:32,564 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:32:42,584 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:32:52,604 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:33:02,623 INFO org.apache.flink.runtime.jobmaster.JobMaster - Could not resolve ResourceManager address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://fl...@ip-172-31-65-130.eu-central-1.compute.internal:39331/user/resourcemanager..
2020-04-08 16:33:10,621 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.
2020-04-08 16:33:10,627 INFO org.apache.flink.runtime.blob.BlobServer - Stopped BLOB server at 0.0.0.0:41429
End of LogType:jobmanager.log
*******************************************************************************
Container: container_e82_1586286375485_0026_01_000001 on ip-172-31-65-130.eu-central-1.compute.internal_45454
LogAggregationType: AGGREGATED
=============================================================================================================
LogType:launch_container.sh
LogLastModifiedTime:Wed Apr 08 16:28:47 +0000 2020
LogLength:7601
LogContents:
#!/bin/bash
set -o pipefail -e
export PRELAUNCH_OUT="/hadoop/yarn/log/application_1586286375485_0026/container_e82_1586286375485_0026_01_000001/prelaunch.out"
exec >"${PRELAUNCH_OUT}"
export PRELAUNCH_ERR="/hadoop/yarn/log/application_1586286375485_0026/container_e82_1586286375485_0026_01_000001/prelaunch.err"
exec 2>"${PRELAUNCH_ERR}"
echo "Setting up env variables"
export PATH="/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent"
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/usr/hdp/2.6.5.0-292/hadoop/conf"}
export MAX_APP_ATTEMPTS="1"
export JAVA_HOME=${JAVA_HOME:-"/usr/jdk64/jdk1.8.0_112"}
export _CLIENT_HOME_DIR="hdfs://erm-int/user/erm_user"
export LANG="en_US.UTF-8"
export APP_SUBMIT_TIME_ENV="1586363326034"
export NM_HOST="ip-172-31-65-130.eu-central-1.compute.internal"
export _APP_ID="application_1586286375485_0026"
export HADOOP_USER_NAME="erm_user"
export LOGNAME="erm_user"
export JVM_PID="$$"
export PWD="/hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026/container_e82_1586286375485_0026_01_000001"
export LOCAL_DIRS="/hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026"
export APPLICATION_WEB_PROXY_BASE="/proxy/application_1586286375485_0026"
export NM_HTTP_PORT="8042"
export LOG_DIRS="/hadoop/yarn/log/application_1586286375485_0026/container_e82_1586286375485_0026_01_000001"
export NM_AUX_SERVICE_mapreduce_shuffle="AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
"
export _CLIENT_SHIP_FILES="log4j.properties=hdfs://erm-int/user/erm_user/.flink/application_1586286375485_0026/log4j.properties,lib/flink-table-blink_2.12-1.10.0.jar=hdfs://erm-int/user/erm_user/.flink/application_1586286375485_0026/lib/flink-table-blink_2.12-1.10.0.jar,lib/flink-table_2.12-1.10.0.jar=hdfs://erm-int/user/erm_user/.flink/application_1586286375485_0026/lib/flink-table_2.12-1.10.0.jar,lib/slf4j-log4j12-1.7.15.jar=hdfs://erm-int/user/erm_user/.flink/application_1586286375485_0026/lib/slf4j-log4j12-1.7.15.jar,lib/log4j-1.2.17.jar=hdfs://erm-int/user/erm_user/.flink/application_1586286375485_0026/lib/log4j-1.2.17.jar,plugins/README.txt=hdfs://erm-int/user/erm_user/.flink/application_1586286375485_0026/plugins/README.txt,erm-monthlyperformance-production.jar=hdfs://erm-int/user/erm_user/.flink/application_1586286375485_0026/erm-monthlyperformance-production.jar,flink-conf.yaml=hdfs://erm-int/user/erm_user/.flink/application_1586286375485_0026/application_1586286375485_0026-flink-conf.yaml1289844950966534911.tmp,"
export NM_PORT="45454"
export USER="erm_user"
export HADOOP_YARN_HOME=${HADOOP_YARN_HOME:-"/usr/hdp/2.6.5.0-292/hadoop-yarn"}
export CLASSPATH=":erm-monthlyperformance-production.jar:lib/flink-table-blink_2.12-1.10.0.jar:lib/flink-table_2.12-1.10.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.15.jar:flink-dist_2.12-1.10.0.jar:flink-conf.yaml:job.graph::/usr/hdp/2.6.5.0-292/hadoop/conf:/usr/hdp/2.6.5.0-292/hadoop/*:/usr/hdp/2.6.5.0-292/hadoop/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:/usr/hdp/current/ext/hadoop/*"
export _FLINK_YARN_FILES="hdfs://erm-int/user/erm_user/.flink/application_1586286375485_0026"
export _ZOOKEEPER_NAMESPACE="application_1586286375485_0026"
export HADOOP_TOKEN_FILE_LOCATION="/hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026/container_e82_1586286375485_0026_01_000001/container_tokens"
export NM_AUX_SERVICE_spark_shuffle=""
export _FLINK_JAR_PATH="hdfs://erm-int/user/erm_user/.flink/application_1586286375485_0026/flink-dist_2.12-1.10.0.jar"
export LOCAL_USER_DIRS="/hadoop/yarn/local/usercache/erm_user/"
export HADOOP_HOME="/usr/hdp/2.6.5.0-292/hadoop"
export _FLINK_CLASSPATH=":erm-monthlyperformance-production.jar:lib/flink-table-blink_2.12-1.10.0.jar:lib/flink-table_2.12-1.10.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.15.jar:flink-dist_2.12-1.10.0.jar:flink-conf.yaml:job.graph:"
export HOME="/home/"
export NM_AUX_SERVICE_spark2_shuffle=""
export CONTAINER_ID="container_e82_1586286375485_0026_01_000001"
export MALLOC_ARENA_MAX="4"
echo "Setting up job resources"
mkdir -p lib
ln -sf "/hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026/filecache/13/flink-table_2.12-1.10.0.jar" "lib/flink-table_2.12-1.10.0.jar"
ln -sf "/hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026/filecache/16/log4j.properties" "log4j.properties"
ln -sf "/hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026/filecache/14/application_1586286375485_00261000897891880198941.tmp" "job.graph"
ln -sf "/hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026/filecache/19/erm-monthlyperformance-production.jar" "erm-monthlyperformance-production.jar"
ln -sf "/hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026/filecache/15/flink-dist_2.12-1.10.0.jar" "flink-dist_2.12-1.10.0.jar"
mkdir -p lib
ln -sf "/hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026/filecache/10/slf4j-log4j12-1.7.15.jar" "lib/slf4j-log4j12-1.7.15.jar"
mkdir -p plugins
ln -sf "/hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026/filecache/18/README.txt" "plugins/README.txt"
ln -sf "/hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026/filecache/11/application_1586286375485_0026-flink-conf.yaml1289844950966534911.tmp" "flink-conf.yaml"
mkdir -p lib
ln -sf "/hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026/filecache/17/log4j-1.2.17.jar" "lib/log4j-1.2.17.jar"
mkdir -p lib
ln -sf "/hadoop/yarn/local/usercache/erm_user/appcache/application_1586286375485_0026/filecache/12/flink-table-blink_2.12-1.10.0.jar" "lib/flink-table-blink_2.12-1.10.0.jar"
echo "Copying debugging information"
# Creating copy of launch script
cp "launch_container.sh" "/hadoop/yarn/log/application_1586286375485_0026/container_e82_1586286375485_0026_01_000001/launch_container.sh"
chmod 640 "/hadoop/yarn/log/application_1586286375485_0026/container_e82_1586286375485_0026_01_000001/launch_container.sh"
# Determining directory contents
echo "ls -l:" 1>"/hadoop/yarn/log/application_1586286375485_0026/container_e82_1586286375485_0026_01_000001/directory.info"
ls -l 1>>"/hadoop/yarn/log/application_1586286375485_0026/container_e82_1586286375485_0026_01_000001/directory.info"
echo "find -L . -maxdepth 5 -ls:" 1>>"/hadoop/yarn/log/application_1586286375485_0026/container_e82_1586286375485_0026_01_000001/directory.info"
find -L . -maxdepth 5 -ls 1>>"/hadoop/yarn/log/application_1586286375485_0026/container_e82_1586286375485_0026_01_000001/directory.info"
echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/hadoop/yarn/log/application_1586286375485_0026/container_e82_1586286375485_0026_01_000001/directory.info"
find -L . -maxdepth 5 -type l -ls 1>>"/hadoop/yarn/log/application_1586286375485_0026/container_e82_1586286375485_0026_01_000001/directory.info"
echo "Launching container"
exec /bin/bash -c "$JAVA_HOME/bin/java -Xms424m -Xmx424m -Dlog.file="/hadoop/yarn/log/application_1586286375485_0026/container_e82_1586286375485_0026_01_000001/jobmanager.log" -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint 1> /hadoop/yarn/log/application_1586286375485_0026/container_e82_1586286375485_0026_01_000001/jobmanager.out 2> /hadoop/yarn/log/application_1586286375485_0026/container_e82_1586286375485_0026_01_000001/jobmanager.err"
End of LogType:launch_container.sh
************************************************************************************
