Hi, I’m using Flink 1.6.4 and recently I ran into a weird issue of rocksdb statebackend. A job that runs fine on a YARN cluster keeps failing on checkpoint after migrated to a new one (with almost everything the same but better machines), and even a clean restart doesn’t help. The root cause is IllegalStateException but with no error message. The stack trace shows that when the rocksdb statebackend is doing the async part of snapshots (runSnapshot), it finds that the local snapshot directory that is created by rocksdb earlier (takeSnapshot) does not exist. I tried to log more informations in RocksDBKeyedStateBackend (see attachment), and found that the local snapshot performed as expected and the .sst files were written, but when the async task accessed the directory, the whole snapshot directory was gone. What could possibly be the cause? Thanks a lot. Best, Paul Lam
|
2019-03-27 16:51:51,229 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - --------------------------------------------------------------------------------
2019-03-27 16:51:51,230 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Starting YARN TaskExecutor runner (Version: 1.6.4-GDC1.0.0, Rev:5ed70c6, Date:18.03.2019 @ 10:03:06 GMT)
2019-03-27 16:51:51,230 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - OS current user: paul
2019-03-27 16:51:51,543 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Current Hadoop/Kerberos user: paul
2019-03-27 16:51:51,543 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.151-b12
2019-03-27 16:51:51,544 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Maximum heap size: 1304 MiBytes
2019-03-27 16:51:51,544 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - JAVA_HOME: /home/hadoop/jdk8
2019-03-27 16:51:51,547 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Hadoop version: 2.6.5
2019-03-27 16:51:51,548 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - JVM Options:
2019-03-27 16:51:51,548 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - -Xms1304m
2019-03-27 16:51:51,548 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - -Xmx1304m
2019-03-27 16:51:51,549 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - -XX:MaxDirectMemorySize=744m
2019-03-27 16:51:51,549 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - -XX:+UseG1GC
2019-03-27 16:51:51,549 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - -Dlog.file.external=/home/hadoop/var/log/flink/paul/5c9b2b70a312953453ce2005/taskmanager
2019-03-27 16:51:51,550 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - -Dlog.file=/home/disk1/yarn/logs/application_1552032656212_0200/container_e05_1552032656212_0200_01_000002/taskmanager.log
2019-03-27 16:51:51,550 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - -Dlog4j.configuration=file:./log4j.properties
2019-03-27 16:51:51,550 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Program Arguments:
2019-03-27 16:51:51,550 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - --configDir
2019-03-27 16:51:51,551 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - .
2019-03-27 16:51:51,551 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Classpath: 7a7740525e220135e43e55b524657ccb:lib/flink-shaded-hadoop2-uber-1.6.4-GDC1.0.0.jar:lib/log4j-1.2.17.jar:lib/log4j-utils-1.0.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:flink.jar:flink-conf.yaml:job.graph::/home/hadoop/gdcconf/hadoop/cosmos:/home/hadoop/hadoop/share/hadoop/common/hadoop-common-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/common/hadoop-common-2.6.0-cdh5.6.0-tests.jar:/home/hadoop/hadoop/share/hadoop/common/hadoop-nfs-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/activation-1.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/xz-1.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/hadoop-auth-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/asm-3.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jetty-6.1.26.cloudera.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/home/hadoop/hadoop/share/hadoop/common/lib/logredactor-1.0.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/avro-1.7.6-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jsch-0.1.54.jar:/home/hadoop/hadoop/share/hadoop/common/lib/hadoop-annotations-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/hadoop/hadoop/share/hadoop/common/lib/htrace-core4-4.0.1-incubating.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-collections-3.2.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop/share/hadoop/common/lib/junit-4.11.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-el-1.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/httpcore-4.2.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/home/hadoop/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/httpclient-4.2.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/curator-client-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/curator-framework-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/home/hadoop/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jsr305-3.0.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/home/hadoop/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/zookeeper-3.4.5-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/hadoop/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/gson-2.2.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.6.0-cdh5.6.0-tests.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.cloudera.4.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.cloudera.4.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/htrace-core4-4.0.1-incubating.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0-cdh5.6.0-tests.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/avro-1.7.6-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce2/*:/home/hadoop/hadoop/share/hadoop/mapreduce2/lib/*:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/spark-2.0.2-yarn-shuffle.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-registry-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jetty-util-6.1.26.cloudera.4.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/activation-1.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jetty-6.1.26.cloudera.4.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jettison-1.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jline-2.11.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jackson-xc-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.5-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:
2019-03-27 16:51:51,554 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - --------------------------------------------------------------------------------
2019-03-27 16:51:51,556 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Registered UNIX signal handlers for [TERM, HUP, INT]
2019-03-27 16:51:51,559 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Current working/local Directory: /home/disk1/yarn/local/usercache/paul/appcache/application_1552032656212_0200
2019-03-27 16:51:51,559 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Current working Directory: /home/disk1/yarn/local/usercache/paul/appcache/application_1552032656212_0200/container_e05_1552032656212_0200_01_000002
2019-03-27 16:51:51,559 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - TM: remote keytab path obtained hdfs://cosmos/user/paul/.flink/application_1552032656212_0200/paul.keytab
2019-03-27 16:51:51,559 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - TM: remote keytab principal obtained paul/schedu...@nie.netease.com
2019-03-27 16:51:51,562 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: env.java.opts.jobmanager, -Dlog.file.external=/home/hadoop/var/log/flink/paul/5c9b2b70a312953453ce2005/jobmanager
2019-03-27 16:51:51,562 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: restart-strategy.failure-rate.max-failures-per-interval, 5
2019-03-27 16:51:51,562 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.jvm-exit-on-oom, false
2019-03-27 16:51:51,562 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: env.java.opts, -XX:+UseG1GC
2019-03-27 16:51:51,562 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, application_1552032656212_0200
2019-03-27 16:51:51,563 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, gdc-dn54-cosmos.i.nease.net
2019-03-27 16:51:51,563 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.scope.task, taskmanager.<job_name>.<task_name>.<subtask_index>
2019-03-27 16:51:51,563 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: containerized.taskmanager.env.LD_LIBRARY_PATH, /home/hadoop/hadoop/lib/native
2019-03-27 16:51:51,563 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: security.kerberos.login.contexts, Client,KafkaClient
2019-03-27 16:51:51,563 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: containerized.master.env.LD_LIBRARY_PATH, home/hadoop/hadoop/lib/native
2019-03-27 16:51:51,563 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.path.root, /flink
2019-03-27 16:51:51,563 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: mode, new
2019-03-27 16:51:51,564 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.scope.tm, taskmanager
2019-03-27 16:51:51,564 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.storageDir, hdfs://cosmos/flink/paul/ha/5c9b2b70a312953453ce2005
2019-03-27 16:51:51,564 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.off-heap, false
2019-03-27 16:51:51,564 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1
2019-03-27 16:51:51,564 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.application-master.env.LD_LIBRARY_PATH, home/hadoop/hadoop/lib/native
2019-03-27 16:51:51,564 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: restart-strategy.failure-rate.failure-rate-interval, 5min
2019-03-27 16:51:51,565 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: historyserver.archive.fs.dir, hdfs://cosmos/flink/default/completed-jobs/
2019-03-27 16:51:51,565 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: web.port, 0
2019-03-27 16:51:51,565 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.backend.incremental, true
2019-03-27 16:51:51,565 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: web.tmpdir, /tmp/flink-web-0b9e63fd-5464-458f-8281-b59410dca40c
2019-03-27 16:51:51,565 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 21967
2019-03-27 16:51:51,565 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.preallocate, false
2019-03-27 16:51:51,565 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: security.kerberos.login.principal, paul/schedu...@nie.netease.com
2019-03-27 16:51:51,565 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.port, 0
2019-03-27 16:51:51,566 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.quorum, cosmos.zk.gdc.x.netease.com:2181
2019-03-27 16:51:51,566 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.reporter.jmx.class, org.apache.flink.metrics.jmx.JMXReporter
2019-03-27 16:51:51,566 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: zookeeper.sasl.login-context-name, Client
2019-03-27 16:51:51,566 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: web.submit.enable, false
2019-03-27 16:51:51,566 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.address, gdc-dn54-cosmos.i.nease.net
2019-03-27 16:51:51,566 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.backend, rocksdb
2019-03-27 16:51:51,566 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: security.kerberos.login.keytab, /home/disk1/yarn/local/usercache/paul/appcache/application_1552032656212_0200/container_e05_1552032656212_0200_01_000001/krb5.keytab
2019-03-27 16:51:51,567 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: containerized.master.env.JAVA_HOME, /home/hadoop/jdk8
2019-03-27 16:51:51,567 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: historyserver.web.address, 0.0.0.0
2019-03-27 16:51:51,567 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: historyserver.web.port, 8082
2019-03-27 16:51:51,567 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: yarn.taskmanager.env.LD_LIBRARY_PATH, /home/hadoop/hadoop/lib/native
2019-03-27 16:51:51,567 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.scope.operator, taskmanager.<job_name>.<operator_name>.<subtask_index>
2019-03-27 16:51:51,568 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: env.java.home, /home/hadoop/jdk8
2019-03-27 16:51:51,568 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: fs.default-scheme, hdfs://cosmos
2019-03-27 16:51:51,568 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.reporter.jmx.port, 8800-8850
2019-03-27 16:51:51,568 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.savepoints.dir, hdfs://cosmos/flink/paul/savepoints/5c9b2b70a312953453ce2005
2019-03-27 16:51:51,568 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: security.kerberos.login.use-ticket-cache, true
2019-03-27 16:51:51,568 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: containerized.taskmanager.env.JAVA_HOME, /home/hadoop/jdk8
2019-03-27 16:51:51,568 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2019-03-27 16:51:51,568 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.scope.jm.job, jobmanager.<job_name>
2019-03-27 16:51:51,568 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: akka.ask.timeout, 10 s
2019-03-27 16:51:51,569 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: akka.jvm-exit-on-fatal-error, true
2019-03-27 16:51:51,569 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.mb, 2048
2019-03-27 16:51:51,569 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.archive.fs.dir, hdfs://cosmos/flink/paul/completed-jobs
2019-03-27 16:51:51,569 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.scope.jm, jobmanager
2019-03-27 16:51:51,569 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.mb, 2048
2019-03-27 16:51:51,569 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.scope.tm.job, taskmanager.<job_name>
2019-03-27 16:51:51,569 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: internal.cluster.execution-mode, DETACHED
2019-03-27 16:51:51,569 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability, zookeeper
2019-03-27 16:51:51,569 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: restart-strategy, failure-rate
2019-03-27 16:51:51,569 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: env.java.opts.taskmanager, -Dlog.file.external=/home/hadoop/var/log/flink/paul/5c9b2b70a312953453ce2005/taskmanager
2019-03-27 16:51:51,570 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 2048m
2019-03-27 16:51:51,570 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.checkpoints.dir, hdfs://cosmos/flink/paul/checkpoints/5c9b2b70a312953453ce2005
2019-03-27 16:51:51,570 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: akka.client.timeout, 60 s
2019-03-27 16:51:51,570 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: zookeeper.sasl.service-name, zookeeper
2019-03-27 16:51:51,583 INFO org.apache.flink.runtime.clusterframework.BootstrapTools - Setting directories for temporary files to: /home/disk1/yarn/local/usercache/paul/appcache/application_1552032656212_0200
2019-03-27 16:51:51,584 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - keytab path: /home/disk1/yarn/local/usercache/paul/appcache/application_1552032656212_0200/container_e05_1552032656212_0200_01_000002/krb5.keytab
2019-03-27 16:51:51,584 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - YARN daemon is running as: paul Yarn client user obtainer: paul/schedu...@nie.netease.com
2019-03-27 16:51:51,798 INFO org.apache.hadoop.security.UserGroupInformation - Login successful for user paul/schedu...@nie.netease.com using keytab file /home/disk1/yarn/local/usercache/paul/appcache/application_1552032656212_0200/container_e05_1552032656212_0200_01_000002/krb5.keytab
2019-03-27 16:51:51,800 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to paul/schedu...@nie.netease.com (auth:KERBEROS)
2019-03-27 16:51:52,310 INFO org.apache.flink.runtime.blob.FileSystemBlobStore - Creating highly available BLOB storage directory at hdfs://cosmos/flink/paul/ha/5c9b2b70a312953453ce2005/application_1552032656212_0200/blob
2019-03-27 16:51:52,400 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Enforcing default ACL for ZK connections
2019-03-27 16:51:52,400 INFO org.apache.flink.runtime.util.ZooKeeperUtils - Using '/flink/application_1552032656212_0200' as Zookeeper namespace.
2019-03-27 16:51:52,438 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
2019-03-27 16:51:52,443 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2019-03-27 16:51:52,443 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:host.name=gdc-dn38-cosmos.i.nease.net
2019-03-27 16:51:52,443 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_151
2019-03-27 16:51:52,443 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
2019-03-27 16:51:52,443 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.home=/home/hadoop/src/jdk-1.8.0/jre
2019-03-27 16:51:52,443 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=7a7740525e220135e43e55b524657ccb:lib/flink-shaded-hadoop2-uber-1.6.4-GDC1.0.0.jar:lib/log4j-1.2.17.jar:lib/log4j-utils-1.0.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:flink.jar:flink-conf.yaml:job.graph::/home/hadoop/gdcconf/hadoop/cosmos:/home/hadoop/hadoop/share/hadoop/common/hadoop-common-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/common/hadoop-common-2.6.0-cdh5.6.0-tests.jar:/home/hadoop/hadoop/share/hadoop/common/hadoop-nfs-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/activation-1.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/xz-1.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/hadoop-auth-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/asm-3.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jetty-6.1.26.cloudera.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/home/hadoop/hadoop/share/hadoop/common/lib/logredactor-1.0.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/avro-1.7.6-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jsch-0.1.54.jar:/home/hadoop/hadoop/share/hadoop/common/lib/hadoop-annotations-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/hadoop/hadoop/share/hadoop/common/lib/htrace-core4-4.0.1-incubating.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-collections-3.2.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop/share/hadoop/common/lib/junit-4.11.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-el-1.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/httpcore-4.2.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/home/hadoop/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/httpclient-4.2.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/curator-client-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/curator-framework-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/home/hadoop/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jsr305-3.0.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/home/hadoop/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/zookeeper-3.4.5-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/hadoop/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/gson-2.2.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.6.0-cdh5.6.0-tests.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.cloudera.4.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.cloudera.4.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/htrace-core4-4.0.1-incubating.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0-cdh5.6.0-tests.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/avro-1.7.6-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop/share/hadoop/mapreduce2/*:/home/hadoop/hadoop/share/hadoop/mapreduce2/lib/*:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/spark-2.0.2-yarn-shuffle.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-registry-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.6.0-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jetty-util-6.1.26.cloudera.4.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/activation-1.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jetty-6.1.26.cloudera.4.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jettison-1.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jline-2.11.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jackson-xc-1.8.8.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.5-cdh5.6.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:
2019-03-27 16:51:52,444 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/home/hadoop/hadoop/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-03-27 16:51:52,444 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
2019-03-27 16:51:52,444 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
2019-03-27 16:51:52,444 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2019-03-27 16:51:52,444 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2019-03-27 16:51:52,444 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.9.0-8-amd64
2019-03-27 16:51:52,444 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.name=paul
2019-03-27 16:51:52,444 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/paul
2019-03-27 16:51:52,444 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/disk1/yarn/local/usercache/paul/appcache/application_1552032656212_0200/container_e05_1552032656212_0200_01_000002
2019-03-27 16:51:52,445 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=cosmos.zk.gdc.x.netease.com:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.curator.org.apache.curator.ConnectionState@63cd604c
2019-03-27 16:51:52,462 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Using configured hostname/address for TaskManager: gdc-dn38-cosmos.i.nease.net.
2019-03-27 16:51:52,464 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Starting AkkaRpcService at gdc-dn38-cosmos.i.nease.net:0.
2019-03-27 16:51:52,470 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.Login - Client successfully logged in.
2019-03-27 16:51:52,471 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.Login - TGT refresh thread started.
2019-03-27 16:51:52,475 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.client.ZooKeeperSaslClient - Client will use GSSAPI as SASL mechanism.
2019-03-27 16:51:52,475 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.Login - TGT valid starting at: Wed Mar 27 16:51:52 HKT 2019
2019-03-27 16:51:52,476 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.Login - TGT expires: Sat Mar 30 16:51:52 HKT 2019
2019-03-27 16:51:52,476 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.Login - TGT refresh sleeping until: Sat Mar 30 03:47:16 HKT 2019
2019-03-27 16:51:52,476 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Opening socket connection to server gdc-zk01-cosmos.i.nease.net/10.90.128.45:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
2019-03-27 16:51:52,477 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Socket connection established to gdc-zk01-cosmos.i.nease.net/10.90.128.45:2181, initiating session
2019-03-27 16:51:52,482 INFO org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Session establishment complete on server gdc-zk01-cosmos.i.nease.net/10.90.128.45:2181, sessionid = 0x1695c599c9401e2, negotiated timeout = 60000
2019-03-27 16:51:52,487 INFO org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
2019-03-27 16:51:52,734 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2019-03-27 16:51:52,785 INFO akka.remote.Remoting - Starting remoting
2019-03-27 16:51:52,898 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://fl...@gdc-dn38-cosmos.i.nease.net:14367]
2019-03-27 16:51:52,974 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - Configuring jmx with {port=8800-8850, class=org.apache.flink.metrics.jmx.JMXReporter}.
2019-03-27 16:51:53,078 INFO org.apache.flink.metrics.jmx.JMXReporter - Started JMX server on port 8801.
2019-03-27 16:51:53,078 INFO org.apache.flink.metrics.jmx.JMXReporter - Configured JMXReporter with {port:8800-8850}
2019-03-27 16:51:53,078 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - Reporting metrics for reporter jmx of type org.apache.flink.metrics.jmx.JMXReporter.
2019-03-27 16:51:53,085 INFO org.apache.flink.runtime.blob.PermanentBlobCache - Created BLOB cache storage directory /home/disk1/yarn/local/usercache/paul/appcache/application_1552032656212_0200/blobStore-4a140672-1349-4b54-b0db-7562a82ff3b4
2019-03-27 16:51:53,088 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /home/disk1/yarn/local/usercache/paul/appcache/application_1552032656212_0200/blobStore-1a54963c-d9b9-4602-9187-41db39bc4bcd
2019-03-27 16:51:53,088 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Starting TaskManager with ResourceID: container_e05_1552032656212_0200_01_000002
2019-03-27 16:51:53,092 INFO org.apache.flink.runtime.io.network.netty.NettyConfig - NettyConfig [server address: gdc-dn38-cosmos.i.nease.net/10.90.129.49, server port: 0, ssl enabled: false, memory segment size (bytes): 32768, transport type: NIO, number of server threads: 1 (manual), number of client threads: 1 (manual), server connect backlog: 0 (use Netty's default), client connect timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's default)]
2019-03-27 16:51:53,157 INFO org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary file directory '/home/disk1/yarn/local/usercache/paul/appcache/application_1552032656212_0200': total 1082 GB, usable 1060 GB (97.97% usable)
2019-03-27 16:51:53,241 INFO org.apache.flink.runtime.io.network.buffer.NetworkBufferPool - Allocated 144 MB for network buffer pool (number of memory segments: 4636, bytes per segment: 32768).
2019-03-27 16:51:53,289 INFO org.apache.flink.runtime.query.QueryableStateUtils - Could not load Queryable State Client Proxy. Probable reason: flink-queryable-state-runtime is not in the classpath. To enable Queryable State, please move the flink-queryable-state-runtime jar from the opt to the lib folder.
2019-03-27 16:51:53,289 INFO org.apache.flink.runtime.query.QueryableStateUtils - Could not load Queryable State Server. Probable reason: flink-queryable-state-runtime is not in the classpath. To enable Queryable State, please move the flink-queryable-state-runtime jar from the opt to the lib folder.
2019-03-27 16:51:53,290 INFO org.apache.flink.runtime.io.network.NetworkEnvironment - Starting the network environment and its components.
2019-03-27 16:51:53,320 INFO org.apache.flink.runtime.io.network.netty.NettyClient - Successful initialization (took 29 ms).
2019-03-27 16:51:53,358 INFO org.apache.flink.runtime.io.network.netty.NettyServer - Successful initialization (took 37 ms). Listening on SocketAddress /10.90.129.49:25571.
2019-03-27 16:51:53,359 INFO org.apache.flink.runtime.taskexecutor.TaskManagerServices - Limiting managed memory to 0.7 of the currently free heap space (906 MB), memory will be allocated lazily.
2019-03-27 16:51:53,362 INFO org.apache.flink.runtime.io.disk.iomanager.IOManager - I/O manager uses directory /home/disk1/yarn/local/usercache/paul/appcache/application_1552032656212_0200/flink-io-dea3d641-44ec-4e3c-8c16-462c56ab52b9 for spill files.
2019-03-27 16:51:53,380 INFO org.apache.flink.runtime.taskexecutor.TaskManagerConfiguration - Messages have a max timeout of 10000 ms
2019-03-27 16:51:53,388 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.taskexecutor.TaskExecutor at akka://flink/user/taskmanager_0 .
2019-03-27 16:51:53,401 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock.
2019-03-27 16:51:53,420 INFO org.apache.flink.runtime.taskexecutor.JobLeaderService - Start job leader service.
2019-03-27 16:51:53,421 INFO org.apache.flink.runtime.filecache.FileCache - User file cache uses directory /home/disk1/yarn/local/usercache/paul/appcache/application_1552032656212_0200/flink-dist-cache-75bb3fd8-ca06-4daf-9c3d-f22cbc1b4ce7
2019-03-27 16:51:53,433 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Connecting to ResourceManager akka.tcp://fl...@gdc-dn54-cosmos.i.nease.net:21967/user/resourcemanager(968f755e5a32914c820a05db350844e8).
2019-03-27 16:51:53,675 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Resolved ResourceManager address, beginning registration
2019-03-27 16:51:53,676 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Registration at ResourceManager attempt 1 (timeout=100ms)
2019-03-27 16:51:53,752 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Successful registration at resource manager akka.tcp://fl...@gdc-dn54-cosmos.i.nease.net:21967/user/resourcemanager under registration id ca58233822a95ce3cd1bce2cfa71b44a.
2019-03-27 16:51:53,781 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Receive slot request AllocationID{4ea24bd9f331acf9fb9442bf4f030211} for job a7003392caf464aa73c145ec6b8decf6 from resource manager with leader id 968f755e5a32914c820a05db350844e8.
2019-03-27 16:51:53,782 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Allocated slot for AllocationID{4ea24bd9f331acf9fb9442bf4f030211}.
2019-03-27 16:51:53,782 INFO org.apache.flink.runtime.taskexecutor.JobLeaderService - Add job a7003392caf464aa73c145ec6b8decf6 for job leader monitoring.
2019-03-27 16:51:53,783 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Starting ZooKeeperLeaderRetrievalService /leader/a7003392caf464aa73c145ec6b8decf6/job_manager_lock.
2019-03-27 16:51:53,788 INFO org.apache.flink.runtime.taskexecutor.JobLeaderService - Try to register at job manager akka.tcp://fl...@gdc-dn54-cosmos.i.nease.net:21967/user/jobmanager_0 with leader id cb607f5b-36ce-461d-9ab3-57a393ef3770.
2019-03-27 16:51:53,805 INFO org.apache.flink.runtime.taskexecutor.JobLeaderService - Resolved JobManager address, beginning registration
2019-03-27 16:51:53,805 INFO org.apache.flink.runtime.taskexecutor.JobLeaderService - Registration at JobManager attempt 1 (timeout=100ms)
2019-03-27 16:51:53,833 INFO org.apache.flink.runtime.taskexecutor.JobLeaderService - Successful registration at job manager akka.tcp://fl...@gdc-dn54-cosmos.i.nease.net:21967/user/jobmanager_0 for job a7003392caf464aa73c145ec6b8decf6.
2019-03-27 16:51:53,836 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Establish JobManager connection for job a7003392caf464aa73c145ec6b8decf6.
2019-03-27 16:51:53,840 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Offer reserved slots to the leader of job a7003392caf464aa73c145ec6b8decf6.
2019-03-27 16:51:53,866 INFO org.apache.flink.runtime.taskexecutor.slot.TaskSlotTable - Activate slot AllocationID{4ea24bd9f331acf9fb9442bf4f030211}.
2019-03-27 16:51:53,909 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Received task Source: continuous-source (1/1).
2019-03-27 16:51:53,910 INFO org.apache.flink.runtime.taskmanager.Task - Source: continuous-source (1/1) (3d90bb25757b2c28aa67295e10eb60be) switched from CREATED to DEPLOYING.
2019-03-27 16:51:53,910 INFO org.apache.flink.runtime.taskmanager.Task - Creating FileSystem stream leak safety net for task Source: continuous-source (1/1) (3d90bb25757b2c28aa67295e10eb60be) [DEPLOYING]
2019-03-27 16:51:53,915 INFO org.apache.flink.runtime.taskmanager.Task - Loading JAR files for task Source: continuous-source (1/1) (3d90bb25757b2c28aa67295e10eb60be) [DEPLOYING].
2019-03-27 16:51:53,923 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Received task daily-distinct-map -> Filter (1/1).
2019-03-27 16:51:53,923 INFO org.apache.flink.runtime.taskmanager.Task - daily-distinct-map -> Filter (1/1) (8098fa4965779f6f547b33a1786e4084) switched from CREATED to DEPLOYING.
2019-03-27 16:51:53,924 INFO org.apache.flink.runtime.taskmanager.Task - Creating FileSystem stream leak safety net for task daily-distinct-map -> Filter (1/1) (8098fa4965779f6f547b33a1786e4084) [DEPLOYING]
2019-03-27 16:51:53,924 INFO org.apache.flink.runtime.taskmanager.Task - Loading JAR files for task daily-distinct-map -> Filter (1/1) (8098fa4965779f6f547b33a1786e4084) [DEPLOYING].
2019-03-27 16:51:53,928 INFO org.apache.flink.runtime.taskmanager.Task - Registering task at network: Source: continuous-source (1/1) (3d90bb25757b2c28aa67295e10eb60be) [DEPLOYING].
2019-03-27 16:51:53,928 INFO org.apache.flink.runtime.taskmanager.Task - Registering task at network: daily-distinct-map -> Filter (1/1) (8098fa4965779f6f547b33a1786e4084) [DEPLOYING].
2019-03-27 16:51:53,953 INFO org.apache.flink.runtime.taskmanager.Task - daily-distinct-map -> Filter (1/1) (8098fa4965779f6f547b33a1786e4084) switched from DEPLOYING to RUNNING.
2019-03-27 16:51:53,957 INFO org.apache.flink.streaming.runtime.tasks.StreamTask - Loading state backend via factory org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory
2019-03-27 16:51:53,961 INFO org.apache.flink.runtime.taskmanager.Task - Source: continuous-source (1/1) (3d90bb25757b2c28aa67295e10eb60be) switched from DEPLOYING to RUNNING.
2019-03-27 16:51:53,961 INFO org.apache.flink.streaming.runtime.tasks.StreamTask - Loading state backend via factory org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory
2019-03-27 16:51:54,009 INFO org.apache.flink.contrib.streaming.state.RocksDBStateBackend - Attempting to load RocksDB native library and store it under '/home/disk1/yarn/local/usercache/paul/appcache/application_1552032656212_0200'
2019-03-27 16:51:54,076 INFO org.apache.flink.contrib.streaming.state.RocksDBStateBackend - Successfully loaded RocksDB native library
2019-03-27 16:51:54,095 DEBUG org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Setting initial keyed backend uid for operator StreamMap_373db71a51735692e8ca760e647b2aa0_(1/1) to fdf5aafb-3233-4764-9eda-64d225a08057.
2019-03-27 16:51:54,095 INFO org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Initializing RocksDB keyed state backend.
2019-03-27 16:51:54,095 DEBUG org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Restoring snapshot from state handles: StateObjectCollection{[]}.
2019-03-27 16:51:56,534 TRACE org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Taking incremental snapshot for checkpoint 1. Snapshot is based on last completed checkpoint -1 assuming the following (shared) files as base: null.
2019-03-27 16:51:56,536 TRACE org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Local RocksDB checkpoint goes to backup path SnapshotDirectory{directory=/home/disk1/yarn/local/usercache/paul/appcache/application_1552032656212_0200/flink-io-dea3d641-44ec-4e3c-8c16-462c56ab52b9/job_a7003392caf464aa73c145ec6b8decf6_op_StreamMap_373db71a51735692e8ca760e647b2aa0__1_1__uuid_52f0710e-1e6e-4772-b386-d0992553e7b8/chk-1, state=ONGOING}.
2019-03-27 16:51:56,545 TRACE org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Local RocksDB checkpoint backup path created: /home/disk1/yarn/local/usercache/paul/appcache/application_1552032656212_0200/flink-io-dea3d641-44ec-4e3c-8c16-462c56ab52b9/job_a7003392caf464aa73c145ec6b8decf6_op_StreamMap_373db71a51735692e8ca760e647b2aa0__1_1__uuid_52f0710e-1e6e-4772-b386-d0992553e7b8/chk-1
2019-03-27 16:51:56,545 TRACE org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Directory content: CURRENT
MANIFEST-000006
000009.sst
OPTIONS-000008
2019-03-27 16:51:56,601 INFO org.apache.flink.runtime.taskmanager.Task - Attempting to fail task externally daily-distinct-map -> Filter (1/1) (8098fa4965779f6f547b33a1786e4084).
2019-03-27 16:51:56,602 INFO org.apache.flink.runtime.taskmanager.Task - daily-distinct-map -> Filter (1/1) (8098fa4965779f6f547b33a1786e4084) switched from RUNNING to FAILED.
AsynchronousException{java.lang.Exception: Could not materialize checkpoint 1 for operator daily-distinct-map -> Filter (1/1).}
at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointExceptionHandler.tryHandleCheckpointException(StreamTask.java:1153)
at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:947)
at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:884)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Exception: Could not materialize checkpoint 1 for operator daily-distinct-map -> Filter (1/1).
at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.handleExecutionException(StreamTask.java:942)
... 6 more
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Local backup directory SnapshotDirectory{directory=/home/disk1/yarn/local/usercache/paul/appcache/application_1552032656212_0200/flink-io-dea3d641-44ec-4e3c-8c16-462c56ab52b9/job_a7003392caf464aa73c145ec6b8decf6_op_StreamMap_373db71a51735692e8ca760e647b2aa0__1_1__uuid_52f0710e-1e6e-4772-b386-d0992553e7b8/chk-1, state=ONGOING} doesn't exist
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:53)
at org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.<init>(OperatorSnapshotFinalizer.java:47)
at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:853)
... 5 more
Caused by: java.lang.IllegalStateException: Local backup directory SnapshotDirectory{directory=/home/disk1/yarn/local/usercache/paul/appcache/application_1552032656212_0200/flink-io-dea3d641-44ec-4e3c-8c16-462c56ab52b9/job_a7003392caf464aa73c145ec6b8decf6_op_StreamMap_373db71a51735692e8ca760e647b2aa0__1_1__uuid_52f0710e-1e6e-4772-b386-d0992553e7b8/chk-1, state=ONGOING} doesn't exist
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBIncrementalSnapshotOperation.runSnapshot(RocksDBKeyedStateBackend.java:2574)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:50)
... 7 more
2019-03-27 16:51:56,614 INFO org.apache.flink.runtime.taskmanager.Task - Triggering cancellation of task code daily-distinct-map -> Filter (1/1) (8098fa4965779f6f547b33a1786e4084).
2019-03-27 16:51:56,625 INFO org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Deleting existing instance base directory /home/disk1/yarn/local/usercache/paul/appcache/application_1552032656212_0200/flink-io-dea3d641-44ec-4e3c-8c16-462c56ab52b9/job_a7003392caf464aa73c145ec6b8decf6_op_StreamMap_373db71a51735692e8ca760e647b2aa0__1_1__uuid_52f0710e-1e6e-4772-b386-d0992553e7b8.
2019-03-27 16:51:56,631 INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for daily-distinct-map -> Filter (1/1) (8098fa4965779f6f547b33a1786e4084).
2019-03-27 16:51:56,640 INFO org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem streams are closed for task daily-distinct-map -> Filter (1/1) (8098fa4965779f6f547b33a1786e4084) [FAILED]
2019-03-27 16:51:56,641 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and sending final execution state FAILED to JobManager for task daily-distinct-map -> Filter 8098fa4965779f6f547b33a1786e4084.
2019-03-27 16:51:56,696 INFO org.apache.flink.runtime.taskmanager.Task - Attempting to cancel task Source: continuous-source (1/1) (3d90bb25757b2c28aa67295e10eb60be).
2019-03-27 16:51:56,697 INFO org.apache.flink.runtime.taskmanager.Task - Source: continuous-source (1/1) (3d90bb25757b2c28aa67295e10eb60be) switched from RUNNING to CANCELING.
2019-03-27 16:51:56,698 INFO org.apache.flink.runtime.taskmanager.Task - Triggering cancellation of task code Source: continuous-source (1/1) (3d90bb25757b2c28aa67295e10eb60be).
2019-03-27 16:51:56,698 INFO com.netease.nie.gdc.flink.checkpointvalidate.function.ContinuousSource - canceling continuous source
2019-03-27 16:51:56,699 INFO com.netease.nie.gdc.flink.checkpointvalidate.function.ContinuousSource - received interruption signal, canceling continuous source
2019-03-27 16:51:56,700 INFO com.netease.nie.gdc.flink.checkpointvalidate.function.ContinuousSource - continuous source is canceled
2019-03-27 16:51:56,700 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Discarding the results produced by task execution 8098fa4965779f6f547b33a1786e4084.
2019-03-27 16:51:56,701 INFO org.apache.flink.runtime.taskmanager.Task - Source: continuous-source (1/1) (3d90bb25757b2c28aa67295e10eb60be) switched from CANCELING to CANCELED.
2019-03-27 16:51:56,701 INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for Source: continuous-source (1/1) (3d90bb25757b2c28aa67295e10eb60be).
2019-03-27 16:51:56,702 INFO org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem streams are closed for task Source: continuous-source (1/1) (3d90bb25757b2c28aa67295e10eb60be) [CANCELED]
2019-03-27 16:51:56,702 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and sending final execution state CANCELED to JobManager for task Source: continuous-source 3d90bb25757b2c28aa67295e10eb60be.