Hi,
This is Qiong Wei, from Microsoft. I’m trying to create a Flink streaming job
which consumer the data from Kafka and then write the data into Azure Blob
Storage. I have a Hadoop cluster (yarn cluster), but it failed when I run the
streaming job, I’m not sure if anybody can help me to solve the problem. Here
is the full log file. Really appreciate 😊
Log Type: jobmanager.log
Log Upload Time: Mon Apr 23 19:18:11 +0000 2018
Log Length: 30602
2018-04-23 19:18:08,232 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
-
--------------------------------------------------------------------------------
2018-04-23 19:18:08,234 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- Starting YARN ApplicationMaster / ResourceManager / JobManager
(Version: <unknown>, Rev:04e4c85, Date:28.02.2018 @ 20:53:14 CST)
2018-04-23 19:18:08,234 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- OS current user: yarn
2018-04-23 19:18:08,650 WARN org.apache.hadoop.util.NativeCodeLoader
- Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2018-04-23 19:18:08,851 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- Current Hadoop/Kerberos user: sshuser
2018-04-23 19:18:08,851 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- JVM: OpenJDK 64-Bit Server VM - Oracle Corporation -
1.8/25.162-b12
2018-04-23 19:18:08,851 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- Maximum heap size: 406 MiBytes
2018-04-23 19:18:08,851 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
2018-04-23 19:18:08,853 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- Hadoop version: 2.4.1
2018-04-23 19:18:08,853 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- JVM Options:
2018-04-23 19:18:08,853 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- -Xmx424m
2018-04-23 19:18:08,853 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
-
-Dlog.file=/mnt/resource/hadoop/yarn/log/application_1523909520695_0017/container_1523909520695_0017_01_000001/jobmanager.log
2018-04-23 19:18:08,853 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- -Dlogback.configurationFile=file:logback.xml
2018-04-23 19:18:08,853 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- -Dlog4j.configuration=file:log4j.properties
2018-04-23 19:18:08,853 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- Program Arguments: (none)
2018-04-23 19:18:08,854 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- Classpath:
kafka-flink-1.0-SNAPSHOT.jar:lib/flink-dist_2.11-1.4.2.jar:lib/flink-python_2.11-1.4.2.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/usr/hdp/current/hadoop-client/conf:/usr/hdp/current/hadoop-client/hadoop-azure-2.7.3.2.6.2.25-1.jar:/usr/hdp/current/hadoop-client/hadoop-aws-2.7.3.2.6.2.25-1.jar:/usr/hdp/current/hadoop-client/hadoop-azure-datalake-2.7.3.2.6.2.25-1.jar:/usr/hdp/current/hadoop-client/hadoop-common-2.7.3.2.6.2.25-1.jar:/usr/hdp/current/hadoop-client/hadoop-aws.jar:/usr/hdp/current/hadoop-client/hadoop-common.jar:/usr/hdp/current/hadoop-client/hadoop-auth.jar:/usr/hdp/current/hadoop-client/hadoop-azure.jar:/usr/hdp/current/hadoop-client/hadoop-annotations-2.7.3.2.6.2.25-1.jar:/usr/hdp/current/hadoop-client/hadoop-azure-datalake.jar:/usr/hdp/current/hadoop-client/hadoop-common-2.7.3.2.6.2.25-1-tests.jar:/usr/hdp/current/hadoop-client/hadoop-common-tests.jar:/usr/hdp/current/hadoop-client/hadoop-annotations.jar:/usr/hdp/current/hadoop-client/azure-data-lake-store-sdk-2.1.4.jar:/usr/hdp/current/hadoop-client/hadoop-auth-2.7.3.2.6.2.25-1.jar:/usr/hdp/current/hadoop-client/hadoop-nfs-2.7.3.2.6.2.25-1.jar:/usr/hdp/current/hadoop-client/hadoop-nfs.jar:/usr/hdp/current/hadoop-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-client/lib/httpcore-4.4.4.jar:/usr/hdp/current/hadoop-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/zookeeper-3.4.6.2.6.2.25-1.jar:/usr/hdp/current/hadoop-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/ranger-yarn-plugin-shim-0.7.0.2.6.2.25-1.jar:/usr/hdp/current/hadoop-client/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/current/hadoop-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-client/lib/json-smart-1.1.1.jar:/usr/hdp/current/hadoop-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.6.2.25-1-javadoc.jar:/usr/hdp/current/hadoop-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-client/lib/httpclient-4.5.2.jar:/usr/hdp/current/hadoop-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-client/lib/jcip-annotations-1.0.jar:/usr/hdp/current/hadoop-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/commons-lang3-3.4.jar:/usr/hdp/current/hadoop-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.6.2.25-1.jar:/usr/hdp/current/hadoop-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/microsoft-log4j-etwappender-1.0.jar:/usr/hdp/current/hadoop-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-client/lib/azure-keyvault-core-0.8.0.jar:/usr/hdp/current/hadoop-client/lib/jetty-sslengine-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/ranger-hdfs-plugin-shim-0.7.0.2.6.2.25-1.jar:/usr/hdp/current/hadoop-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-client/lib/json-simple-1.1.jar:/usr/hdp/current/hadoop-client/lib/azure-storage-5.3.0.jar:/usr/hdp/current/hadoop-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-client/lib/junit-4.11.jar:/usr/hdp/current/hadoop-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-client/lib/joda-time-2.9.4.jar:/usr/hdp/current/hadoop-client/lib/xz-1.0.jar:/usr/hdp/current/hadoop-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.6.2.25-1-sources.jar:/usr/hdp/current/hadoop-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-client/lib/mdsdclient-1.0.jar:/usr/hdp/current/hadoop-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-client/lib/slf4j-api-1.7.10.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-client/lib/mockito-all-1.8.5.jar:/usr/hdp/current/hadoop-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/jsch-0.1.54.jar:/usr/hdp/current/hadoop-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-client/lib/json-20090211.jar:/usr/hdp/current/hadoop-client/lib/ojdbc6.jar:/usr/hdp/current/hadoop-client/lib/ranger-plugin-classloader-0.7.0.2.6.2.25-1.jar:/usr/hdp/current/hadoop-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-client/lib/hamcrest-core-1.3.jar:/usr/hdp/current/hadoop-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/nimbus-jose-jwt-3.9.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.3.2.6.2.25-1.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-2.7.3.2.6.2.25-1-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-hdfs-nfs-2.7.3.2.6.2.25-1.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xercesImpl-2.9.1.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-hdfs-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-hdfs-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xml-apis-1.3.04.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okhttp-2.4.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/adls2-oauth2-token-provider-1.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/adls2-oauth2-token-provider.jar:/usr/hdp/current/hadoop-hdfs-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-hdfs-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/microsoft-log4j-etwappender-1.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/json-simple-1.1.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-hdfs-client/lib/okio-1.4.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-hdfs-client/lib/mdsdclient-1.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/hadoop-azure-datalake.jar:/usr/hdp/current/hadoop-hdfs-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-all-4.0.23.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-daemon-1.0.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-hdfs-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/lib/json-20090211.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-hdfs-client/lib/azure-data-lake-store-sdk.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client-2.7.3.2.6.2.25-1.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api-2.7.3.2.6.2.25-1.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager-2.7.3.2.6.2.25-1.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-resourcemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-unmanaged-am-launcher-2.7.3.2.6.2.25-1.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy-2.7.3.2.6.2.25-1.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-pluginstorage-2.7.3.2.6.2.25-1.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager-2.7.3.2.6.2.25-1.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-api.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common-2.7.3.2.6.2.25-1.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-common.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-client.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager-2.7.3.2.6.2.25-1.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-applicationhistoryservice-2.7.3.2.6.2.25-1.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-sharedcachemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-common-2.7.3.2.6.2.25-1.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-nodemanager.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-timeline-pluginstorage.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-2.7.3.2.6.2.25-1.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-web-proxy.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-tests-2.7.3.2.6.2.25-1.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-registry-2.7.3.2.6.2.25-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-core-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpcore-4.4.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.6.2.25-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-guice-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-annotations-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.6.2.25-1-tests.jar:/usr/hdp/current/hadoop-yarn-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-net-3.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-servlet-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/json-smart-1.1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/asm-3.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/httpclient-4.5.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-codec-1.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/metrics-core-3.0.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jcip-annotations-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-yarn-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/fst-2.24.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-lang3-3.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/hdinsight-ats-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/microsoft-log4j-etwappender-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/paranamer-2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/javax.inject-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/azure-keyvault-core-0.8.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-sslengine-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsp-api-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/json-simple-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/objenesis-2.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/azure-storage-5.3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-pool2-2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-databind-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-client-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-dbcp2-2.0.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-digester-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-yarn-client/lib/xz-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hadoop-yarn-client/lib/mdsdclient-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/guava-11.0.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/jersey-json-1.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/guice-3.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/javassist-3.18.1-GA.jar:/usr/hdp/current/hadoop-yarn-client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-yarn-client/lib/stax-api-1.0-2.jar:/usr/hdp/current/hadoop-yarn-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-yarn-client/lib/activation-1.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsch-0.1.54.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-configuration-1.6.jar:/usr/hdp/current/hadoop-yarn-client/lib/aopalliance-1.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/json-20090211.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/lib/sqljdbc41.jar:/usr/hdp/current/hadoop-yarn-client/lib/nimbus-jose-jwt-3.9.jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-client/lib/api-util-1.0.0-M20.jar
2018-04-23 19:18:08,855 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
-
--------------------------------------------------------------------------------
2018-04-23 19:18:08,856 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- Registered UNIX signal handlers for [TERM, HUP, INT]
2018-04-23 19:18:08,863 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- remoteKeytabPrincipal obtained null
2018-04-23 19:18:08,863 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- YARN daemon is running as: sshuser Yarn client user obtainer:
sshuser
2018-04-23 19:18:08,871 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- Loading config from directory
/mnt/resource/hadoop/yarn/local/usercache/sshuser/appcache/application_1523909520695_0017/container_1523909520695_0017_01_000001
2018-04-23 19:18:08,875 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: web.port, 8081
2018-04-23 19:18:08,875 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: parallelism.default, 1
2018-04-23 19:18:08,875 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: high-availability.cluster-id,
application_1523909520695_0017
2018-04-23 19:18:08,875 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.address, localhost
2018-04-23 19:18:08,875 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.heap.mb, 1024
2018-04-23 19:18:08,875 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.numberOfTaskSlots, 5
2018-04-23 19:18:08,876 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.heap.mb, 1024
2018-04-23 19:18:08,876 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.port, 6123
2018-04-23 19:18:08,876 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.memory.preallocate, false
2018-04-23 19:18:08,954 WARN org.apache.hadoop.conf.Configuration
- /usr/hdp/current/hadoop-client/conf/core-site.xml:an attempt to
override final parameter: fs.defaultFS; Ignoring.
2018-04-23 19:18:08,957 WARN org.apache.hadoop.conf.Configuration
- /usr/hdp/current/hadoop-client/conf/hdfs-site.xml:an attempt to
override final parameter: dfs.datanode.data.dir; Ignoring.
2018-04-23 19:18:08,957 WARN org.apache.hadoop.conf.Configuration
- /usr/hdp/current/hadoop-client/conf/hdfs-site.xml:an attempt to
override final parameter: dfs.datanode.failed.volumes.tolerated; Ignoring.
2018-04-23 19:18:08,958 WARN org.apache.hadoop.conf.Configuration
- /usr/hdp/current/hadoop-client/conf/hdfs-site.xml:an attempt to
override final parameter: dfs.namenode.name.dir; Ignoring.
2018-04-23 19:18:08,958 WARN org.apache.hadoop.conf.Configuration
- /usr/hdp/current/hadoop-client/conf/hdfs-site.xml:an attempt to
override final parameter: dfs.support.append; Ignoring.
2018-04-23 19:18:08,958 WARN org.apache.hadoop.conf.Configuration
- /usr/hdp/current/hadoop-client/conf/hdfs-site.xml:an attempt to
override final parameter: dfs.webhdfs.enabled; Ignoring.
2018-04-23 19:18:08,961 WARN org.apache.hadoop.conf.Configuration
- /usr/hdp/2.6.2.25-1/hadoop/conf/core-site.xml:an attempt to
override final parameter: fs.defaultFS; Ignoring.
2018-04-23 19:18:08,964 WARN org.apache.hadoop.conf.Configuration
- /usr/hdp/2.6.2.25-1/hadoop/conf/hdfs-site.xml:an attempt to
override final parameter: dfs.datanode.data.dir; Ignoring.
2018-04-23 19:18:08,964 WARN org.apache.hadoop.conf.Configuration
- /usr/hdp/2.6.2.25-1/hadoop/conf/hdfs-site.xml:an attempt to
override final parameter: dfs.datanode.failed.volumes.tolerated; Ignoring.
2018-04-23 19:18:08,965 WARN org.apache.hadoop.conf.Configuration
- /usr/hdp/2.6.2.25-1/hadoop/conf/hdfs-site.xml:an attempt to
override final parameter: dfs.namenode.name.dir; Ignoring.
2018-04-23 19:18:08,965 WARN org.apache.hadoop.conf.Configuration
- /usr/hdp/2.6.2.25-1/hadoop/conf/hdfs-site.xml:an attempt to
override final parameter: dfs.support.append; Ignoring.
2018-04-23 19:18:08,965 WARN org.apache.hadoop.conf.Configuration
- /usr/hdp/2.6.2.25-1/hadoop/conf/hdfs-site.xml:an attempt to
override final parameter: dfs.webhdfs.enabled; Ignoring.
2018-04-23 19:18:08,968 WARN org.apache.hadoop.conf.Configuration
- /usr/hdp/2.6.2.25-1/hadoop/etc/hadoop/core-site.xml:an attempt to
override final parameter: fs.defaultFS; Ignoring.
2018-04-23 19:18:08,971 WARN org.apache.hadoop.conf.Configuration
- /usr/hdp/2.6.2.25-1/hadoop/etc/hadoop/hdfs-site.xml:an attempt to
override final parameter: dfs.datanode.data.dir; Ignoring.
2018-04-23 19:18:08,971 WARN org.apache.hadoop.conf.Configuration
- /usr/hdp/2.6.2.25-1/hadoop/etc/hadoop/hdfs-site.xml:an attempt to
override final parameter: dfs.datanode.failed.volumes.tolerated; Ignoring.
2018-04-23 19:18:08,971 WARN org.apache.hadoop.conf.Configuration
- /usr/hdp/2.6.2.25-1/hadoop/etc/hadoop/hdfs-site.xml:an attempt to
override final parameter: dfs.namenode.name.dir; Ignoring.
2018-04-23 19:18:08,972 WARN org.apache.hadoop.conf.Configuration
- /usr/hdp/2.6.2.25-1/hadoop/etc/hadoop/hdfs-site.xml:an attempt to
override final parameter: dfs.support.append; Ignoring.
2018-04-23 19:18:08,972 WARN org.apache.hadoop.conf.Configuration
- /usr/hdp/2.6.2.25-1/hadoop/etc/hadoop/hdfs-site.xml:an attempt to
override final parameter: dfs.webhdfs.enabled; Ignoring.
2018-04-23 19:18:08,973 INFO
org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set
to sshuser (auth:SIMPLE)
2018-04-23 19:18:09,017 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- YARN assigned hostname for application master:
wn1-flinkc.zmzaiimzkxhe1jx024zqv0iqof.cx.internal.cloudapp.net
2018-04-23 19:18:09,023 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- TaskManagers will be created with 5 task slots
2018-04-23 19:18:09,023 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- TaskManagers will be started with container size 4096 MB, JVM
heap size 3072 MB, JVM direct memory limit -1 MB
2018-04-23 19:18:09,029 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- Trying to start actor system at
wn1-flinkc.zmzaiimzkxhe1jx024zqv0iqof.cx.internal.cloudapp.net:32865
2018-04-23 19:18:09,717 INFO akka.event.slf4j.Slf4jLogger
- Slf4jLogger started
2018-04-23 19:18:09,834 INFO akka.remote.Remoting
- Starting remoting
2018-04-23 19:18:10,060 INFO akka.remote.Remoting
- Remoting started; listening on addresses
:[akka.tcp://fl...@wn1-flinkc.zmzaiimzkxhe1jx024zqv0iqof.cx.internal.cloudapp.net:32865]
2018-04-23 19:18:10,071 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- Actor system started at
akka.tcp://fl...@wn1-flinkc.zmzaiimzkxhe1jx024zqv0iqof.cx.internal.cloudapp.net:32865
2018-04-23 19:18:10,071 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- Actor system bound to hostname
wn1-flinkc.zmzaiimzkxhe1jx024zqv0iqof.cx.internal.cloudapp.net.
2018-04-23 19:18:10,075 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- TM:remote keytab path obtained null
2018-04-23 19:18:10,075 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- TM:remote keytab principal obtained null
2018-04-23 19:18:10,075 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- TM:remote yarn conf path obtained null
2018-04-23 19:18:10,076 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
- TM:remote krb5 path obtained null
2018-04-23 19:18:10,250 INFO org.apache.hadoop.metrics2.impl.MetricsConfig
- loaded properties from
hadoop-metrics2-azure-file-system.properties
2018-04-23 19:18:10,259 INFO org.apache.hadoop.metrics2.sink.WasbAzureIaasSink
- Init starting.
2018-04-23 19:18:10,259 INFO org.apache.hadoop.metrics2.sink.AzureIaasSink
- Init starting. Initializing MdsLogger.
2018-04-23 19:18:10,266 INFO org.apache.hadoop.metrics2.sink.AzureIaasSink
- Init completed.
2018-04-23 19:18:10,266 INFO org.apache.hadoop.metrics2.sink.WasbAzureIaasSink
- Init completed.
2018-04-23 19:18:10,273 INFO
org.apache.hadoop.metrics2.impl.MetricsSinkAdapter - Sink azurefs2
started
2018-04-23 19:18:10,347 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl
- Scheduled snapshot period at 60 second(s).
2018-04-23 19:18:10,347 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl
- azure-file-system metrics system started
2018-04-23 19:18:10,358 ERROR org.apache.flink.yarn.YarnApplicationMasterRunner
- YARN Application Master initialization failed
java.lang.NoSuchMethodError:
org.apache.hadoop.conf.Configuration.getPassword(Ljava/lang/String;)[C
at
org.apache.hadoop.fs.azure.SimpleKeyProvider.getStorageAccountKey(SimpleKeyProvider.java:47)
at
org.apache.hadoop.fs.azure.ShellDecryptionKeyProvider.getStorageAccountKey(ShellDecryptionKeyProvider.java:40)
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.getAccountKeyFromConfiguration(AzureNativeFileSystemStore.java:934)
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:1023)
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.initialize(AzureNativeFileSystemStore.java:507)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.initialize(NativeAzureFileSystem.java:1281)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2397)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.flink.yarn.Utils.createTaskExecutorContext(Utils.java:398)
at
org.apache.flink.yarn.YarnApplicationMasterRunner.runApplicationMaster(YarnApplicationMasterRunner.java:316)
at
org.apache.flink.yarn.YarnApplicationMasterRunner$1.call(YarnApplicationMasterRunner.java:194)
at
org.apache.flink.yarn.YarnApplicationMasterRunner$1.call(YarnApplicationMasterRunner.java:191)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at
org.apache.flink.yarn.YarnApplicationMasterRunner.run(YarnApplicationMasterRunner.java:191)
at
org.apache.flink.yarn.YarnApplicationMasterRunner.main(YarnApplicationMasterRunner.java:124)
Best,
Qiong
