hi, guys I'm using samza in realtime process. After running for about 10 hours, some containers paused and not processing.
When I looked into the log, I found a lot of 2016-08-21 10:03:07 [WARN ](o.a.k.c.p.i.Sender :257) Got error produce response with correlation id 490345 on topic-partition test3_a2_mobileDictClient_android_uid_imei-3, retrying (17 attempts left). Error: NOT_LEADER_FOR_PARTITION 2016-08-21 10:03:07 [WARN ](o.a.k.c.p.i.Sender :257) Got error produce response with correlation id 490345 on topic-partition test3_a2_mobileDictClient_android_uid_imei-4, retrying (18 attempts left). Error: NOT_LEADER_FOR_PARTITION 2016-08-21 10:03:07 [WARN ](o.a.k.c.p.i.Sender :257) Got error produce response with correlation id 490345 on topic-partition test3_a2_mobileDictClient_android_uid_imei-6, retrying (18 attempts left). Error: NOT_LEADER_FOR_PARTITION 2016-08-21 10:03:07 [WARN ](o.a.k.c.p.i.Sender :257) Got error produce response with correlation id 490346 on topic-partition test3_a2_mobileDictClient_android_uid_imei-3, retrying (16 attempts left). Error: NOT_LEADER_FOR_PARTITION 2016-08-21 10:03:07 [WARN ](o.a.k.c.p.i.Sender :257) Got error produce response with correlation id 490346 on topic-partition test3_a2_mobileDictClient_android_uid_imei-4, retrying (17 attempts left). Error: NOT_LEADER_FOR_PARTITION 2016-08-21 10:03:07 [WARN ](o.a.k.c.p.i.Sender :257) Got error produce response with correlation id 490346 on topic-partition test3_a2_mobileDictClient_android_uid_imei-6, retrying (17 attempts left). Error: NOT_LEADER_FOR_PARTITION ... 2016-08-21 10:49:01 [WARN ](o.a.s.s.k.KafkaSystemProducer :66 ) Retrying send messsage due to RetriableException - org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.. Turn on debugging to get a full stack trace 2016-08-21 10:49:11 [WARN ](o.a.s.s.k.KafkaSystemProducer :66 ) Retrying send messsage due to RetriableException - org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.. Turn on debugging to get a full stack trace 2016-08-21 10:49:21 [WARN ](o.a.s.s.k.KafkaSystemProducer :66 ) Retrying send messsage due to RetriableException - org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.. Turn on debugging to get a full stack trace 2016-08-21 10:49:31 [WARN ](o.a.s.s.k.KafkaSystemProducer :66 ) Retrying send messsage due to RetriableException - org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.. Turn on debugging to get a full stack trace 2 This happens since "rush hour" for new messages produced to kafka. May be this is a bug of kafka / samza? kafka version: 0.10.0.0 kafka config and part of paused log are attached.
/disk1/eadop/jdk1.8.0_40/bin/java -Xmx3000m -Xms3000m -Xmn500m -XX:+UseConcMarkSweepGC -server -Dsamza.container.id=3 -Dsamza.container.name=samza-container-3 -Dlog4j.configuration=file:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/log4j.xml -Dsamza.log.dir=/disk9/eadop/hadoop-tmp/userlogs/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015 -Djava.io.tmpdir=/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/tmp -XX:+PrintGCDateStamps -Xloggc:/disk9/eadop/hadoop-tmp/userlogs/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10241024 -d64 -cp /disk1/eadop/hadoop-2.7.2/etc/hadoop:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/activation-1.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/akka-actor_2.10-2.1.2.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/aopalliance-1.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/apacheds-i18n-2.0.0-M15.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/api-asn1-api-1.0.0-M20.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/api-util-1.0.0-M20.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/asm-3.2.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/avro-1.7.5.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/avro-ipc-1.7.4-tests.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/avro-mapred-1.7.4-hadoop2-1.7.4-hadoop2.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/batik-css-1.7b1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/batik-ext-1.7b1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/batik-util-1.7b1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/bcmail-jdk14-1.32.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/bcprov-jdk14-1.32.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/bijection-avro_2.10-0.9.2.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/bijection-core_2.10-0.9.2.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/cglib-2.2.1-v20090111.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/cluster-r81292.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/collections-2.0.399745.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/commons-cli-1.2.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/commons-codec-1.3.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/commons-codec-1.8.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/commons-collections-3.2.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/commons-collections-3.2.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/commons-compress-1.4.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/commons-configuration-1.4.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/commons-configuration-1.9.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/commons-httpclient-3.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/commons-io-2.4.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/commons-lang-2.3.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/commons-lang-2.6.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/commons-logging-1.1.3.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/commons-logging-1.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/commons-math3-3.2.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/commons-net-3.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/config-1.0.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/cowork-6.1.500010.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/cowork-6.2.16.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/crawler-1.0.141757.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/curator-client-2.7.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/curator-framework-2.7.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/curator-recipes-2.7.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/dom4j-1.6.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/feed-tools-1.5.83487.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/filters-1.5.81879.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/flowtest-2.0.81941.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/fontbox-0.1.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/grizzled-slf4j_2.10-1.0.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/gson-2.2.4.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/guava-17.0-17.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/guava-17.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/hadoop-annotations-2.7.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/hadoop-auth-2.7.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/hadoop-client-2.7.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/hadoop-common-2.7.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/hadoop-hdfs-2.7.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/hadoop-mapreduce-client-app-2.7.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/hadoop-mapreduce-client-common-2.7.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/hadoop-mapreduce-client-core-2.7.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/hadoop-mapreduce-client-hs-2.7.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/hadoop-mapreduce-client-jobclient-2.7.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/hadoop-mapreduce-client-shuffle-2.7.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/hadoop-yarn-api-2.7.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/hadoop-yarn-client-2.7.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/hadoop-yarn-common-2.7.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/hadoop-yarn-server-common-2.7.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/hadoop-yarn-server-nodemanager-2.7.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/hamcrest-core-1.3.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/hsqldb-2.0.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/htrace-core-3.1.0-incubating.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/httpclient-4.2.5.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/httpclient-om-3.0.1.115005.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/httpcore-4.2.5.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/ip-location-2.1.417467.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jackson-annotations-2.7.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jackson-core-2.7.5.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jackson-core-asl-1.9.13.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jackson-databind-2.7.5.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jackson-jaxrs-1.9.13.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jackson-mapper-asl-1.9.13.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jackson-xc-1.9.13.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/javax.inject-1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/java-xmlbuilder-0.4.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/javax.servlet-3.0.0.v201112011016.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jaxb-api-2.2.2.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jaxb-impl-2.2.3-1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jaxen-1.1beta8.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jdom-1.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jersey-client-1.9.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jersey-core-1.9.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jersey-guice-1.9.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jersey-json-1.9.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jersey-server-1.9.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jets3t-0.9.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jettison-1.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jetty-6.1.26.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jetty-continuation-8.1.8.v20121106.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jetty-http-8.1.8.v20121106.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jetty-io-8.1.8.v20121106.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jetty-security-8.1.8.v20121106.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jetty-server-8.1.8.v20121106.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jetty-servlet-8.1.8.v20121106.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jetty-util-6.1.26.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jetty-util-8.1.8.v20121106.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jetty-webapp-8.1.8.v20121106.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jetty-xml-8.1.8.v20121106.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jline-2.9.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/joda-convert-1.2.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/joda-time-2.3.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jopt-simple-3.2.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jsch-0.1.42.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/json-1.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jsp-api-2.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/jsr305-3.0.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/juniversalchardet-1.0.3.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/kafka_2.10-0.8.2.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/kafka-avro-serializer-1.0.1-1.0.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/kafka-clients-0.8.2.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/kafka-schema-registry-client-1.0.1-1.0.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/leveldbjni-all-1.8.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/log4j-1.2.17.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/lz4-1.2.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/metrics-core-2.2.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/mime-tools-1.0.82161.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/mime-util-2.1.3.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/mina-core-1.1.6.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/misc-tools-3.0.478908.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/netty-3.6.2.Final.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/netty-all-4.0.23.Final.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/normalizers-1.0.81884.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/odfs-api-6.0.500010.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/odis-5.1.24.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/odis-app-2.1.237649.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/outback-data-1.0.280912.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/outlog-2.0.496136.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/paranamer-2.3.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/pdfbox-0.7.3.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/poi-3.0.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/poi-contrib-3.0.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/poi-scratchpad-3.0.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/protobuf-java-2.5.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/ps-2.0.278469.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/qdox-1.6.3.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/request_log_demo.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/rhino-1.7R1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/rl_2.10-0.4.4.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/rocksdbjni-3.13.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/rome-om-r71835-0.9b.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/samza-api-0.10.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/samza-core_2.10-0.10.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/samza-kafka_2.10-0.10.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/samza-kv_2.10-0.10.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/samza-kv-inmemory_2.10-0.10.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/samza-kv-rocksdb_2.10-0.10.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/samza-yarn_2.10-0.10.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/scala-compiler-2.10.4.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/scala-library-2.10.5.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/scala-reflect-2.10.4.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/scalate-core_2.10-1.6.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/scalate-util_2.10-1.6.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/scalatra_2.10-2.2.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/scalatra-common_2.10-2.2.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/scalatra-scalate_2.10-2.2.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/servlet-api-2.5-20081211.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/servlet-api-2.5.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/simplenet-1.1.82554.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/slf4j-api-1.7.10.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/slf4j-log4j12-1.7.10.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/slf4j-s-1.0.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/snappy-java-1.1.1.6.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/spring-2.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/spring-mock-2.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/stax-api-1.0-2.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/text-tools-3.0.404990.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/toolbox-validator-0.2.308523.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/tousy-1.6.137832.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/velocity-1.7.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/w3c-sac-1.3.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/web-tools-1.0.83345.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/xercesImpl-2.9.1.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/xml-apis-1.3.04.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/xmlenc-0.52.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/xz-1.0.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/zkclient-0.3.jar:/disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/__package/lib/zookeeper-3.4.6.jar org.apache.samza.container.SamzaContainer 2016-08-20 23:10:50 [INFO ](o.a.s.c.SamzaContainer$ :80 ) Got container ID: 3 2016-08-20 23:10:50 [INFO ](o.a.s.c.SamzaContainer$ :82 ) Got coordinator URL: http://hd012.corp.yodao.com:23030/ 2016-08-20 23:10:50 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Fetching configuration from: http://hd012.corp.yodao.com:23030/ 2016-08-20 23:10:51 [INFO ](o.a.s.m.JmxServer :55 ) According to Util.getLocalHost.getHostName we are hd033.corp.yodao.com 2016-08-20 23:10:51 [INFO ](o.a.s.m.JmxServer :55 ) Started JmxServer registry port=25873 server port=25874 url=service:jmx:rmi://localhost:25874/jndi/rmi://localhost:25873/jmxrmi 2016-08-20 23:10:51 [INFO ](o.a.s.m.JmxServer :55 ) If you are tunneling, you might want to try JmxServer registry port=25873 server port=25874 url=service:jmx:rmi://hd033.corp.yodao.com:25874/jndi/rmi://hd033.corp.yodao.com:25873/jmxrmi 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Setting up Samza container: samza-container-3 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Samza container PID: 107...@hd033.corp.yodao.com 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Using configuration: {yarn.container.count=8, serializers.registry.metrics.class=org.apache.samza.serializers.MetricsSnapshotSerdeFactory, serializers.registry.string.class=org.apache.samza.serializers.StringSerdeFactory, systems.kafkaAvro.producer.retries=30, systems.kafkaAvro.producer.compression.codec=gzip, task.checkpoint.system=kafkaStr, analyzer2.output.type=repart, systems.kafkaAvro.consumer.auto.offset.reset=smallest, task.checkpoint.factory=org.apache.samza.checkpoint.kafka.KafkaCheckpointManagerFactory, systems.kafkaStr.consumer.auto.offset.reset=smallest, systems.kafkaAvro.samza.offset.default=oldest, systems.kafkaStr.samza.msg.serde=string, metrics.reporters=streamMetrics,jmxMetrics, job.name=test3_a2_mobileDictClient_android_init_task, yarn.container.retry.window.ms=60000, systems.kafkaAvro.samza.key.serde=string, systems.kafkaStr.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory, job.coordinator.system=kafkaStr, systems.kafkaStr.producer.request.timeout.ms=3000, yarn.package.path=hdfs://ws001:8000/user/analysis/samza_job_test.jar, job.factory.class=org.apache.samza.job.yarn.YarnJobFactory, yarn.container.retry.count=300, systems.kafkaAvro.producer.buffer.memory=1000000, systems.kafkaAvro.producer.batch.size=10000, systems.kafkaStr.producer.bootstrap.servers=hd020:9092,hd035:9092,hd036:9092, yarn.container.memory.mb=5120, systems.kafkaAvro.producer.bootstrap.servers=hd020:9092,hd035:9092,hd036:9092, systems.kafkaStr.producer.acks=all, yarn.am.container.memory.mb=5120, analyzer2.output.keyField=imei, analyzer2.output.topic=test3_a2_mobileDictClient_android_uid_imei, yarn.am.opts=-Xmx3000m -Xms3000m -Xmn500m -XX:+UseConcMarkSweepGC, systems.kafkaAvro.producer.request.timeout.ms=3000, job.coordinator.replication.factor=3, systems.kafkaAvro.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory, systems.kafkaStr.consumer.zookeeper.connect=ns013:2185,ns014:2185,ns015:2185, analyzer2.output.system=kafkaAvro, metrics.reporter.streamMetrics.class=org.apache.samza.metrics.reporter.MetricsSnapshotReporterFactory, systems.kafkaStr.samza.offset.default=oldest, analyzer2.importUid.date=20160721, metrics.reporter.jmxMetrics.class=org.apache.samza.metrics.reporter.JmxReporterFactory, systems.kafkaStr.producer.retry.backoff.ms=1000, task.inputs=kafkaStr.outlog_MobileSearch,kafkaStr.outlog_mobileDictClient, analyzer2.isDebugging=false, systems.kafkaAvro.producer.acks=all, systems.kafkaStr.streams.samzaMetrics.samza.msg.serde=metrics, task.class=toolbox.analyzer2.realtime.InitTask, systems.kafkaAvro.consumer.zookeeper.connect=ns013:2185,ns014:2185,ns015:2185, systems.kafkaAvro.producer.retry.backoff.ms=1000, systems.kafkaStr.producer.retries=30, analyzer.product.name=mobileDictClient.android, systems.kafkaAvro.producer.linger.ms=100, task.opts=-Xmx3000m -Xms3000m -Xmn500m -XX:+UseConcMarkSweepGC, systems.kafkaAvro.samza.msg.serde=string, metrics.reporter.streamMetrics.stream=kafkaStr.samzaMetrics} 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Using container model: ContainerModel [containerId=3, tasks={Partition 3=TaskModel [taskName=Partition 3, systemStreamPartitions=[SystemStreamPartition [kafkaStr, outlog_mobileDictClient, 3], SystemStreamPartition [kafkaStr, outlog_MobileSearch, 3]], changeLogPartition=Partition [partition=2]]}] 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Got system names: Set(kafkaAvro, kafkaStr) 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Got serde names: Set(string, metrics) 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Verifying properties 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property auto.offset.reset is overridden to smallest 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property client.id is overridden to samza_admin-test3_a2_mobileDictClient_android_init_task-1-1471705851612-0 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property group.id is overridden to undefined-samza-consumer-group-a0a66822-828e-445d-a1a5-efc48956d613 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property zookeeper.connect is overridden to ns013:2185,ns014:2185,ns015:2185 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Verifying properties 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property auto.offset.reset is overridden to smallest 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property client.id is overridden to samza_admin-test3_a2_mobileDictClient_android_init_task-1-1471705851646-1 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property group.id is overridden to undefined-samza-consumer-group-84f9e4e3-271d-4402-acf1-012b1b613642 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property zookeeper.connect is overridden to ns013:2185,ns014:2185,ns015:2185 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Got system factories: Set(kafkaAvro, kafkaStr) 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Verifying properties 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property client.id is overridden to samza_admin-test3_a2_mobileDictClient_android_init_task-1-1471705851646-1 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property metadata.broker.list is overridden to hd020:9092,hd035:9092,hd036:9092 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property request.timeout.ms is overridden to 30000 2016-08-20 23:10:51 [INFO ](k.c.ClientUtils$ :68 ) Fetching metadata from broker id:0,host:hd020,port:9092 with correlation id 0 for 2 topic(s) Set(outlog_mobileDictClient, outlog_MobileSearch) 2016-08-20 23:10:51 [INFO ](k.p.SyncProducer :68 ) Connected to hd020:9092 for producing 2016-08-20 23:10:51 [INFO ](k.p.SyncProducer :68 ) Disconnecting from hd020:9092 2016-08-20 23:10:51 [INFO ](o.a.s.s.k.KafkaSystemAdmin$ :55 ) Got metadata: Map(outlog_mobileDictClient -> SystemStreamMetadata [streamName=outlog_mobileDictClient, partitionMetadata={Partition [partition=0]=SystemStreamPartitionMetadata [oldestOffset=3340666712, newestOffset=3498332367, upcomingOffset=3498332368], Partition [partition=5]=SystemStreamPartitionMetadata [oldestOffset=3338080233, newestOffset=3496188680, upcomingOffset=3496188681], Partition [partition=1]=SystemStreamPartitionMetadata [oldestOffset=3304752977, newestOffset=3461353346, upcomingOffset=3461353347], Partition [partition=6]=SystemStreamPartitionMetadata [oldestOffset=3340457389, newestOffset=3499875069, upcomingOffset=3499875070], Partition [partition=2]=SystemStreamPartitionMetadata [oldestOffset=3338016574, newestOffset=3494877594, upcomingOffset=3494877595], Partition [partition=7]=SystemStreamPartitionMetadata [oldestOffset=3287205740, newestOffset=3443616044, upcomingOffset=3443616045], Partition [partition=3]=SystemStreamPartitionMetadata [oldestOffset=3326753277, newestOffset=3482740040, upcomingOffset=3482740041], Partition [partition=4]=SystemStreamPartitionMetadata [oldestOffset=3335123218, newestOffset=3494219856, upcomingOffset=3494219857]}], outlog_MobileSearch -> SystemStreamMetadata [streamName=outlog_MobileSearch, partitionMetadata={Partition [partition=0]=SystemStreamPartitionMetadata [oldestOffset=25189579, newestOffset=28942973, upcomingOffset=28942974], Partition [partition=5]=SystemStreamPartitionMetadata [oldestOffset=27497291, newestOffset=29267686, upcomingOffset=29267687], Partition [partition=1]=SystemStreamPartitionMetadata [oldestOffset=27497654, newestOffset=29324396, upcomingOffset=29324397], Partition [partition=6]=SystemStreamPartitionMetadata [oldestOffset=27495723, newestOffset=29235872, upcomingOffset=29235873], Partition [partition=2]=SystemStreamPartitionMetadata [oldestOffset=27499260, newestOffset=29345228, upcomingOffset=29345229], Partition [partition=7]=SystemStreamPartitionMetadata [oldestOffset=27499813, newestOffset=29325432, upcomingOffset=29325433], Partition [partition=3]=SystemStreamPartitionMetadata [oldestOffset=27492528, newestOffset=29123454, upcomingOffset=29123455], Partition [partition=4]=SystemStreamPartitionMetadata [oldestOffset=27490635, newestOffset=29002356, upcomingOffset=29002357]}]) 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Got input stream metadata: Map(SystemStream [system=kafkaStr, stream=outlog_mobileDictClient] -> SystemStreamMetadata [streamName=outlog_mobileDictClient, partitionMetadata={Partition [partition=0]=SystemStreamPartitionMetadata [oldestOffset=3340666712, newestOffset=3498332367, upcomingOffset=3498332368], Partition [partition=5]=SystemStreamPartitionMetadata [oldestOffset=3338080233, newestOffset=3496188680, upcomingOffset=3496188681], Partition [partition=1]=SystemStreamPartitionMetadata [oldestOffset=3304752977, newestOffset=3461353346, upcomingOffset=3461353347], Partition [partition=6]=SystemStreamPartitionMetadata [oldestOffset=3340457389, newestOffset=3499875069, upcomingOffset=3499875070], Partition [partition=2]=SystemStreamPartitionMetadata [oldestOffset=3338016574, newestOffset=3494877594, upcomingOffset=3494877595], Partition [partition=7]=SystemStreamPartitionMetadata [oldestOffset=3287205740, newestOffset=3443616044, upcomingOffset=3443616045], Partition [partition=3]=SystemStreamPartitionMetadata [oldestOffset=3326753277, newestOffset=3482740040, upcomingOffset=3482740041], Partition [partition=4]=SystemStreamPartitionMetadata [oldestOffset=3335123218, newestOffset=3494219856, upcomingOffset=3494219857]}], SystemStream [system=kafkaStr, stream=outlog_MobileSearch] -> SystemStreamMetadata [streamName=outlog_MobileSearch, partitionMetadata={Partition [partition=0]=SystemStreamPartitionMetadata [oldestOffset=25189579, newestOffset=28942973, upcomingOffset=28942974], Partition [partition=5]=SystemStreamPartitionMetadata [oldestOffset=27497291, newestOffset=29267686, upcomingOffset=29267687], Partition [partition=1]=SystemStreamPartitionMetadata [oldestOffset=27497654, newestOffset=29324396, upcomingOffset=29324397], Partition [partition=6]=SystemStreamPartitionMetadata [oldestOffset=27495723, newestOffset=29235872, upcomingOffset=29235873], Partition [partition=2]=SystemStreamPartitionMetadata [oldestOffset=27499260, newestOffset=29345228, upcomingOffset=29345229], Partition [partition=7]=SystemStreamPartitionMetadata [oldestOffset=27499813, newestOffset=29325432, upcomingOffset=29325433], Partition [partition=3]=SystemStreamPartitionMetadata [oldestOffset=27492528, newestOffset=29123454, upcomingOffset=29123455], Partition [partition=4]=SystemStreamPartitionMetadata [oldestOffset=27490635, newestOffset=29002356, upcomingOffset=29002357]}]) 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Verifying properties 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property auto.offset.reset is overridden to smallest 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property client.id is overridden to samza_consumer-test3_a2_mobileDictClient_android_init_task-1-1471705851787-2 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property group.id is overridden to undefined-samza-consumer-group-0da8af23-d390-4ef6-a439-81eef12d1b5c 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property zookeeper.connect is overridden to ns013:2185,ns014:2185,ns015:2185 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Verifying properties 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property auto.offset.reset is overridden to smallest 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property client.id is overridden to samza_admin-test3_a2_mobileDictClient_android_init_task-1-1471705851795-3 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property group.id is overridden to undefined-samza-consumer-group-3f9f859b-5149-48b6-a2c4-2bc873a072ce 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property zookeeper.connect is overridden to ns013:2185,ns014:2185,ns015:2185 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Got system consumers: Set(kafkaStr) 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Got system producers: Set(kafkaAvro, kafkaStr) 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Got serdes: Set(string, metrics) 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Got change log system streams: Map() 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Setting up JVM metrics. 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Setting up message chooser. 2016-08-20 23:10:51 [INFO ](o.a.s.s.c.DefaultChooser :55 ) Building default chooser with: useBatching=false, useBootstrapping=false, usePriority=false 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Setting up metrics reporters. 2016-08-20 23:10:51 [INFO ](o.a.s.m.r.MetricsSnapshotReporterFactory:55 ) Creating new metrics snapshot reporter. 2016-08-20 23:10:51 [WARN ](o.a.s.m.r.MetricsSnapshotReporterFactory:66 ) Unable to find implementation version in jar's meta info. Defaulting to 0.0.1. 2016-08-20 23:10:51 [INFO ](o.a.s.m.r.MetricsSnapshotReporterFactory:55 ) Got system stream SystemStream [system=kafkaStr, stream=samzaMetrics]. 2016-08-20 23:10:51 [INFO ](o.a.s.m.r.MetricsSnapshotReporterFactory:55 ) Got system factory org.apache.samza.system.kafka.KafkaSystemFactory@5adb0db3. 2016-08-20 23:10:51 [INFO ](o.a.s.m.r.MetricsSnapshotReporterFactory:55 ) Got producer org.apache.samza.system.kafka.KafkaSystemProducer@7241a47d. 2016-08-20 23:10:51 [INFO ](o.a.s.m.r.MetricsSnapshotReporterFactory:55 ) Got serde org.apache.samza.serializers.MetricsSnapshotSerde@7e3181aa. 2016-08-20 23:10:51 [INFO ](o.a.s.m.r.MetricsSnapshotReporterFactory:55 ) Setting polling interval to 60 2016-08-20 23:10:51 [INFO ](o.a.s.m.r.MetricsSnapshotReporter :55 ) got metrics snapshot reporter properties [job name: test3_a2_mobileDictClient_android_init_task, job id: 1, containerName: samza-container-3, version: 0.0.1, samzaVersion: 0.10.0, host: hd033.corp.yodao.com, pollingInterval 60] 2016-08-20 23:10:51 [INFO ](o.a.s.m.r.MetricsSnapshotReporter :55 ) Registering MetricsSnapshotReporterFactory with producer. 2016-08-20 23:10:51 [INFO ](o.a.s.m.r.JmxReporterFactory :55 ) Creating JMX reporter with name jmxMetrics. 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Got metrics reporters: Set(streamMetrics, jmxMetrics) 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Verifying properties 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property auto.offset.reset is overridden to smallest 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property client.id is overridden to samza_admin-test3_a2_mobileDictClient_android_init_task-1-1471705851863-7 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property group.id is overridden to undefined-samza-consumer-group-487c5451-8665-4a18-b056-a420485bf20f 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property zookeeper.connect is overridden to ns013:2185,ns014:2185,ns015:2185 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Verifying properties 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property auto.offset.reset is overridden to smallest 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property client.id is overridden to samza_checkpoint_manager-test3_a2_mobileDictClient_android_init_task-1-1471705851869-9 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property group.id is overridden to undefined-samza-consumer-group-69cbbcf1-06d0-4c5b-b153-97b44a862740 2016-08-20 23:10:51 [INFO ](k.u.VerifiableProperties :68 ) Property zookeeper.connect is overridden to ns013:2185,ns014:2185,ns015:2185 2016-08-20 23:10:51 [INFO ](o.a.s.c.k.KafkaCheckpointManager :55 ) Creating KafkaCheckpointManager with: clientId=samza_checkpoint_manager-test3_a2_mobileDictClient_android_init_task-1-1471705851869-9, checkpointTopic=__samza_checkpoint_ver_1_for_test3-a2-mobileDictClient-android-init-task_1, systemName=kafkaStr 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Got checkpoint manager: KafkaCheckpointManager [systemName=kafkaStr, checkpointTopic=__samza_checkpoint_ver_1_for_test3-a2-mobileDictClient-android-init-task_1] 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Got offset manager: org.apache.samza.checkpoint.OffsetManager@79351f41 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Got storage engines: Set() 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Got stream task class: toolbox.analyzer2.realtime.InitTask 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Got window milliseconds: -1 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Got commit milliseconds: 60000 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Got shutdown timeout milliseconds: 5000 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Got store consumers: Map() 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Got default storage engine base directory: /disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/state 2016-08-20 23:10:51 [WARN ](o.a.s.c.SamzaContainer$ :66 ) No override was provided for logged store base directory. This disables local state re-use on application restart. If you want to enable this feature, set LOGGED_STORE_BASE_DIR as an environment variable in all machines running the Samza container 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Got base directory for logged data stores: /disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/state 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Got task stores: Map() 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Retrieved SystemStreamPartitions Set(SystemStreamPartition [kafkaStr, outlog_mobileDictClient, 3], SystemStreamPartition [kafkaStr, outlog_MobileSearch, 3]) for Partition 3 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer$ :55 ) Samza container setup complete. 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer :55 ) Starting container. 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer :55 ) Registering task instances with metrics. 2016-08-20 23:10:51 [INFO ](o.a.s.m.r.MetricsSnapshotReporter :55 ) Registering TaskName-Partition 3 with producer. 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer :55 ) Starting JVM metrics. 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer :55 ) Starting metrics reporters. 2016-08-20 23:10:51 [INFO ](o.a.s.m.r.MetricsSnapshotReporter :55 ) Registering samza-container-3 with producer. 2016-08-20 23:10:51 [INFO ](o.a.s.m.r.MetricsSnapshotReporter :55 ) Starting producer. 2016-08-20 23:10:51 [INFO ](o.a.s.m.r.MetricsSnapshotReporter :55 ) Starting reporter timer. 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer :55 ) Registering task instances with offsets. 2016-08-20 23:10:51 [INFO ](o.a.s.c.SamzaContainer :55 ) Starting offset manager. 2016-08-20 23:10:51 [INFO ](o.a.s.u.KafkaUtil :55 ) Attempting to create topic __samza_checkpoint_ver_1_for_test3-a2-mobileDictClient-android-init-task_1. 2016-08-20 23:10:51 [INFO ](o.I.z.ZkEventThread :64 ) Starting ZkClient event thread. 2016-08-20 23:10:51 [INFO ](o.a.s.s.k.KafkaSystemProducer :55 ) Creating a new producer for system kafkaStr. 2016-08-20 23:10:51 [WARN ](o.a.s.c.KafkaProducerConfig :66 ) Samza does not provide producer failure handling. Consider setting 'retries' to a large value, like Int.MAX. 2016-08-20 23:10:51 [INFO ](o.a.k.c.p.ProducerConfig :113) ProducerConfig values: compression.type = none metric.reporters = [] metadata.max.age.ms = 300000 metadata.fetch.timeout.ms = 60000 acks = all batch.size = 16384 reconnect.backoff.ms = 10 bootstrap.servers = [hd020:9092, hd035:9092, hd036:9092] receive.buffer.bytes = 32768 retry.backoff.ms = 1000 buffer.memory = 33554432 timeout.ms = 30000 key.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer retries = 30 max.request.size = 1048576 block.on.buffer.full = true value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer metrics.sample.window.ms = 30000 send.buffer.bytes = 131072 max.in.flight.requests.per.connection = 1 metrics.num.samples = 2 linger.ms = 0 client.id = samza_producer-test3_a2_mobileDictClient_android_init_task-1-1471705851855-6 2016-08-20 23:10:51 [INFO ](o.I.z.ZkClient :449) zookeeper state changed (SyncConnected) 2016-08-20 23:10:51 [WARN ](o.a.k.c.p.ProducerConfig :121) The configuration request.timeout.ms = null was supplied but isn't a known config. 2016-08-20 23:10:52 [INFO ](o.I.z.ZkEventThread :82 ) Terminate ZkClient event thread. 2016-08-20 23:10:52 [INFO ](o.a.s.u.KafkaUtil :55 ) Topic __samza_checkpoint_ver_1_for_test3-a2-mobileDictClient-android-init-task_1 already exists. 2016-08-20 23:10:52 [INFO ](o.a.s.u.KafkaUtil :55 ) Validating topic __samza_checkpoint_ver_1_for_test3-a2-mobileDictClient-android-init-task_1. Expecting partition count: 1 2016-08-20 23:10:52 [INFO ](k.u.VerifiableProperties :68 ) Verifying properties 2016-08-20 23:10:52 [INFO ](k.u.VerifiableProperties :68 ) Property client.id is overridden to samza_checkpoint_manager-test3_a2_mobileDictClient_android_init_task-1-1471705851869-9 2016-08-20 23:10:52 [INFO ](k.u.VerifiableProperties :68 ) Property metadata.broker.list is overridden to hd020:9092,hd035:9092,hd036:9092 2016-08-20 23:10:52 [INFO ](k.u.VerifiableProperties :68 ) Property request.timeout.ms is overridden to 30000 2016-08-20 23:10:52 [INFO ](k.c.ClientUtils$ :68 ) Fetching metadata from broker id:2,host:hd036,port:9092 with correlation id 0 for 1 topic(s) Set(__samza_checkpoint_ver_1_for_test3-a2-mobileDictClient-android-init-task_1) 2016-08-20 23:10:52 [INFO ](k.p.SyncProducer :68 ) Connected to hd036:9092 for producing 2016-08-20 23:10:52 [INFO ](k.p.SyncProducer :68 ) Disconnecting from hd036:9092 2016-08-20 23:10:52 [INFO ](o.a.s.u.KafkaUtil :55 ) Successfully validated topic __samza_checkpoint_ver_1_for_test3-a2-mobileDictClient-android-init-task_1. 2016-08-20 23:10:52 [INFO ](o.a.s.c.k.KafkaCheckpointManager :55 ) Reading checkpoint for taskName Partition 3 2016-08-20 23:10:52 [INFO ](o.a.s.c.k.KafkaCheckpointManager :55 ) No TaskName to checkpoint mapping provided. Reading for first time. 2016-08-20 23:10:52 [INFO ](o.a.s.c.k.KafkaCheckpointManager :55 ) Connecting to leader hd035:9092 for topic __samza_checkpoint_ver_1_for_test3-a2-mobileDictClient-android-init-task_1 and to fetch all checkpoint messages. 2016-08-20 23:10:52 [INFO ](o.a.s.c.k.KafkaCheckpointManager :55 ) Got offset 0 for topic __samza_checkpoint_ver_1_for_test3-a2-mobileDictClient-android-init-task_1 and partition 0. Attempting to fetch messages for checkpoint log. 2016-08-20 23:10:52 [INFO ](o.a.s.c.k.KafkaCheckpointManager :55 ) Get latest offset 352854 for topic __samza_checkpoint_ver_1_for_test3-a2-mobileDictClient-android-init-task_1 and partition 0. 2016-08-20 23:10:52 [INFO ](o.a.s.c.k.KafkaCheckpointManager :55 ) Got checkpoint state for taskName Partition 3: Checkpoint [offsets={SystemStreamPartition [kafkaStr, outlog_mobileDictClient, 3]=3482528738, SystemStreamPartition [kafkaStr, outlog_MobileSearch, 3]=29121197}] 2016-08-20 23:10:52 [INFO ](o.a.s.c.OffsetManager :55 ) Checkpointed offset is currently 3482528738 for SystemStreamPartition [kafkaStr, outlog_mobileDictClient, 3] 2016-08-20 23:10:52 [INFO ](o.a.s.c.OffsetManager :55 ) Checkpointed offset is currently 29121197 for SystemStreamPartition [kafkaStr, outlog_MobileSearch, 3] 2016-08-20 23:10:52 [INFO ](o.a.s.c.OffsetManager :55 ) Successfully loaded last processed offsets: Map(Partition 3 -> Map(SystemStreamPartition [kafkaStr, outlog_mobileDictClient, 3] -> 3482528738, SystemStreamPartition [kafkaStr, outlog_MobileSearch, 3] -> 29121197)) 2016-08-20 23:10:52 [INFO ](o.a.s.c.OffsetManager :55 ) Successfully loaded starting offsets: Map(Partition 3 -> Map(SystemStreamPartition [kafkaStr, outlog_mobileDictClient, 3] -> 3482528739, SystemStreamPartition [kafkaStr, outlog_MobileSearch, 3] -> 29121198)) 2016-08-20 23:10:52 [INFO ](o.a.s.c.SamzaContainer :55 ) Registering localityManager for the container 2016-08-20 23:10:52 [INFO ](o.a.s.c.s.CoordinatorStreamSystemProducer:83 ) Starting coordinator stream producer. 2016-08-20 23:10:52 [INFO ](o.a.s.c.SamzaContainer :55 ) Writing container locality and JMX address to Coordinator Stream 2016-08-20 23:10:52 [INFO ](o.a.s.c.LocalityManager :125) Container 3 started at hd033.corp.yodao.com 2016-08-20 23:10:52 [INFO ](o.a.s.s.k.KafkaSystemProducer :55 ) Creating a new producer for system kafkaStr. 2016-08-20 23:10:52 [WARN ](o.a.s.c.KafkaProducerConfig :66 ) Samza does not provide producer failure handling. Consider setting 'retries' to a large value, like Int.MAX. 2016-08-20 23:10:52 [INFO ](o.a.k.c.p.ProducerConfig :113) ProducerConfig values: compression.type = none metric.reporters = [] metadata.max.age.ms = 300000 metadata.fetch.timeout.ms = 60000 acks = all batch.size = 16384 reconnect.backoff.ms = 10 bootstrap.servers = [hd020:9092, hd035:9092, hd036:9092] receive.buffer.bytes = 32768 retry.backoff.ms = 1000 buffer.memory = 33554432 timeout.ms = 30000 key.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer retries = 30 max.request.size = 1048576 block.on.buffer.full = true value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer metrics.sample.window.ms = 30000 send.buffer.bytes = 131072 max.in.flight.requests.per.connection = 1 metrics.num.samples = 2 linger.ms = 0 client.id = samza_producer-test3_a2_mobileDictClient_android_init_task-1-1471705851865-8 2016-08-20 23:10:52 [WARN ](o.a.k.c.p.ProducerConfig :121) The configuration request.timeout.ms = null was supplied but isn't a known config. 2016-08-20 23:10:54 [INFO ](o.a.s.c.SamzaContainer :55 ) Starting task instance stores. 2016-08-20 23:10:54 [INFO ](o.a.s.s.TaskStorageManager :55 ) Validating change log streams 2016-08-20 23:10:54 [INFO ](o.a.s.s.TaskStorageManager :55 ) Got change log stream metadata: Map() 2016-08-20 23:10:54 [INFO ](o.a.s.s.TaskStorageManager :55 ) Assigning oldest change log offsets for taskName Partition 3: Map() 2016-08-20 23:10:54 [INFO ](o.a.s.c.SamzaContainer :55 ) Registering task instances with producers. 2016-08-20 23:10:54 [INFO ](o.a.s.c.SamzaContainer :55 ) Starting producer multiplexer. 2016-08-20 23:10:54 [INFO ](o.a.s.c.SamzaContainer :55 ) Initializing stream tasks. 2016-08-20 23:10:54 [INFO ](t.a.u.h.SamzaInitialize :32 ) moving dir /disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/./__package/conf to /disk10/eadop/hadoop-tmp/nm-local-dir/usercache/analysis/appcache/application_1470649775019_23388/container_e17_1470649775019_23388_01_000015/./conf 2016-08-20 23:10:56 [INFO ](o.a.s.c.SamzaContainer :55 ) Registering task instances with consumers. 2016-08-20 23:10:56 [INFO ](o.a.s.c.SamzaContainer :55 ) Starting consumer multiplexer. 2016-08-20 23:10:56 [INFO ](o.a.s.s.k.KafkaSystemConsumer :55 ) Refreshing brokers for: Map([outlog_MobileSearch,3] -> 29121198, [outlog_mobileDictClient,3] -> 3482528739) 2016-08-20 23:10:56 [INFO ](o.a.s.s.k.BrokerProxy :55 ) Creating new SimpleConsumer for host hd020:9092 for system kafkaStr 2016-08-20 23:10:56 [INFO ](o.a.s.s.k.GetOffset :55 ) Validating offset 29121198 for topic and partition [outlog_MobileSearch,3] 2016-08-20 23:10:56 [INFO ](o.a.s.s.k.GetOffset :55 ) Able to successfully read from offset 29121198 for topic and partition [outlog_MobileSearch,3]. Using it to instantiate consumer. 2016-08-20 23:10:56 [INFO ](o.a.s.s.k.BrokerProxy :55 ) Starting BrokerProxy for hd020:9092 2016-08-20 23:10:56 [INFO ](o.a.s.s.k.BrokerProxy :55 ) Creating new SimpleConsumer for host hd035:9092 for system kafkaStr 2016-08-20 23:10:56 [INFO ](o.a.s.s.k.GetOffset :55 ) Validating offset 3482528739 for topic and partition [outlog_mobileDictClient,3] 2016-08-20 23:10:56 [INFO ](o.a.s.s.k.GetOffset :55 ) Able to successfully read from offset 3482528739 for topic and partition [outlog_mobileDictClient,3]. Using it to instantiate consumer. 2016-08-20 23:10:56 [INFO ](o.a.s.s.k.BrokerProxy :55 ) Starting BrokerProxy for hd035:9092 2016-08-20 23:10:56 [INFO ](o.a.s.c.SamzaContainer :55 ) Entering run loop. 2016-08-20 23:10:56 [INFO ](o.a.s.c.TaskInstance :55 ) SystemStreamPartition [kafkaStr, outlog_MobileSearch, 3] is catched up. 2016-08-20 23:10:56 [INFO ](o.a.s.c.TaskInstance :55 ) SystemStreamPartition [kafkaStr, outlog_mobileDictClient, 3] is catched up. 2016-08-20 23:10:56 [INFO ](o.a.s.s.k.KafkaSystemProducer :55 ) Creating a new producer for system kafkaAvro. 2016-08-20 23:10:56 [WARN ](o.a.s.c.KafkaProducerConfig :66 ) Samza does not provide producer failure handling. Consider setting 'retries' to a large value, like Int.MAX. 2016-08-20 23:10:56 [INFO ](o.a.k.c.p.ProducerConfig :113) ProducerConfig values: compression.type = none metric.reporters = [] metadata.max.age.ms = 300000 metadata.fetch.timeout.ms = 60000 acks = all batch.size = 10000 reconnect.backoff.ms = 10 bootstrap.servers = [hd020:9092, hd035:9092, hd036:9092] receive.buffer.bytes = 32768 retry.backoff.ms = 1000 buffer.memory = 1000000 timeout.ms = 30000 key.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer retries = 30 max.request.size = 1048576 block.on.buffer.full = true value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer metrics.sample.window.ms = 30000 send.buffer.bytes = 131072 max.in.flight.requests.per.connection = 1 metrics.num.samples = 2 linger.ms = 100 client.id = samza_producer-test3_a2_mobileDictClient_android_init_task-1-1471705851802-4 2016-08-20 23:10:56 [WARN ](o.a.k.c.p.ProducerConfig :121) The configuration request.timeout.ms = null was supplied but isn't a known config. 2016-08-20 23:10:56 [WARN ](o.a.k.c.p.ProducerConfig :121) The configuration compression.codec = null was supplied but isn't a known config. 2016-08-20 23:11:52 [WARN ](o.a.s.c.KafkaProducerConfig :66 ) Samza does not provide producer failure handling. Consider setting 'retries' to a large value, like Int.MAX. 2016-08-20 23:11:52 [INFO ](o.a.k.c.p.ProducerConfig :113) ProducerConfig values: compression.type = none metric.reporters = [] metadata.max.age.ms = 300000 metadata.fetch.timeout.ms = 60000 acks = all batch.size = 16384 reconnect.backoff.ms = 10 bootstrap.servers = [hd020:9092, hd035:9092, hd036:9092] receive.buffer.bytes = 32768 retry.backoff.ms = 1000 buffer.memory = 33554432 timeout.ms = 30000 key.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer retries = 30 max.request.size = 1048576 block.on.buffer.full = true value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer metrics.sample.window.ms = 30000 send.buffer.bytes = 131072 max.in.flight.requests.per.connection = 1 metrics.num.samples = 2 linger.ms = 0 client.id = samza_checkpoint_manager-test3_a2_mobileDictClient_android_init_task-1-1471705851869-9 2016-08-20 23:11:52 [WARN ](o.a.k.c.p.ProducerConfig :121) The configuration request.timeout.ms = null was supplied but isn't a known config. 2016-08-20 23:20:52 [WARN ](o.a.k.c.n.Selector :276) Error in I/O with hd020/10.168.17.55 java.io.EOFException at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62) at org.apache.kafka.common.network.Selector.poll(Selector.java:248) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) at java.lang.Thread.run(Thread.java:745) 2016-08-20 23:20:54 [WARN ](o.a.k.c.n.Selector :276) Error in I/O with hd020/10.168.17.55 java.io.EOFException at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62) at org.apache.kafka.common.network.Selector.poll(Selector.java:248) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) at java.lang.Thread.run(Thread.java:745) 2016-08-20 23:20:58 [WARN ](o.a.k.c.n.Selector :276) Error in I/O with hd035/10.168.17.70 java.io.EOFException at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62) at org.apache.kafka.common.network.Selector.poll(Selector.java:248) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) at java.lang.Thread.run(Thread.java:745) 2016-08-20 23:21:54 [WARN ](o.a.k.c.n.Selector :276) Error in I/O with hd035/10.168.17.70 java.io.EOFException at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62) at org.apache.kafka.common.network.Selector.poll(Selector.java:248) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) at java.lang.Thread.run(Thread.java:745) 2016-08-21 03:31:00 [WARN ](o.a.k.c.n.Selector :276) Error in I/O with hd036/10.168.17.71 java.io.EOFException at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62) at org.apache.kafka.common.network.Selector.poll(Selector.java:248) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) at java.lang.Thread.run(Thread.java:745) 2016-08-21 03:46:54 [WARN ](o.a.k.c.n.Selector :276) Error in I/O with hd020/10.168.17.55 java.io.EOFException at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62) at org.apache.kafka.common.network.Selector.poll(Selector.java:248) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) at java.lang.Thread.run(Thread.java:745) 2016-08-21 05:01:01 [WARN ](o.a.k.c.n.Selector :276) Error in I/O with hd036/10.168.17.71 java.io.EOFException at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62) at org.apache.kafka.common.network.Selector.poll(Selector.java:248) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) at java.lang.Thread.run(Thread.java:745) 2016-08-21 08:36:04 [WARN ](o.a.k.c.n.Selector :276) Error in I/O with hd036/10.168.17.71 java.io.EOFException at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62) at org.apache.kafka.common.network.Selector.poll(Selector.java:248) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) at java.lang.Thread.run(Thread.java:745) 2016-08-21 09:21:57 [WARN ](o.a.k.c.n.Selector :276) Error in I/O with hd036/10.168.17.71 java.io.EOFException at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62) at org.apache.kafka.common.network.Selector.poll(Selector.java:248) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) at java.lang.Thread.run(Thread.java:745) 2016-08-21 10:02:23 [INFO ](o.a.s.s.k.KafkaSystemConsumer :55 ) Abdicating for [outlog_MobileSearch,3] 2016-08-21 10:02:23 [INFO ](o.a.s.s.k.KafkaSystemConsumer :55 ) Refreshing brokers for: Map([outlog_MobileSearch,3] -> 29437812) 2016-08-21 10:02:23 [INFO ](k.u.VerifiableProperties :68 ) Verifying properties 2016-08-21 10:02:23 [INFO ](k.u.VerifiableProperties :68 ) Property client.id is overridden to samza_consumer-test3_a2_mobileDictClient_android_init_task-1-1471705851787-2 2016-08-21 10:02:23 [INFO ](k.u.VerifiableProperties :68 ) Property metadata.broker.list is overridden to hd020:9092,hd035:9092,hd036:9092 2016-08-21 10:02:23 [INFO ](k.u.VerifiableProperties :68 ) Property request.timeout.ms is overridden to 30000 2016-08-21 10:02:23 [INFO ](k.c.ClientUtils$ :68 ) Fetching metadata from broker id:1,host:hd035,port:9092 with correlation id 0 for 1 topic(s) Set(outlog_MobileSearch) 2016-08-21 10:02:23 [INFO ](k.p.SyncProducer :68 ) Connected to hd035:9092 for producing 2016-08-21 10:02:23 [INFO ](k.p.SyncProducer :68 ) Disconnecting from hd035:9092 2016-08-21 10:02:23 [INFO ](o.a.s.s.k.BrokerProxy :55 ) Creating new SimpleConsumer for host hd036:9092 for system kafkaStr 2016-08-21 10:02:23 [INFO ](o.a.s.s.k.GetOffset :55 ) Validating offset 29437812 for topic and partition [outlog_MobileSearch,3] 2016-08-21 10:02:24 [INFO ](o.a.s.s.k.KafkaSystemConsumer :55 ) Refreshing brokers for: Map([outlog_MobileSearch,3] -> 29437812) 2016-08-21 10:02:24 [INFO ](o.a.s.s.k.GetOffset :55 ) Able to successfully read from offset 29437812 for topic and partition [outlog_MobileSearch,3]. Using it to instantiate consumer. 2016-08-21 10:02:24 [INFO ](o.a.s.s.k.BrokerProxy :55 ) Starting BrokerProxy for hd036:9092 2016-08-21 10:02:25 [WARN ](o.a.k.c.p.i.Sender :257) Got error produce response with correlation id 490003 on topic-partition test3_a2_mobileDictClient_android_uid_imei-1, retrying (29 attempts left). Error: REQUEST_TIMED_OUT 2016-08-21 10:02:25 [WARN ](o.a.k.c.p.i.Sender :257) Got error produce response with correlation id 490003 on topic-partition test3_a2_mobileDictClient_android_uid_imei-5, retrying (29 attempts left). Error: REQUEST_TIMED_OUT 2016-08-21 10:02:25 [WARN ](o.a.k.c.p.i.Sender :257) Got error produce response with correlation id 490092 on topic-partition test3_a2_mobileDictClient_android_uid_imei-1, retrying (28 attempts left). Error: NOT_LEADER_FOR_PARTITION 2016-08-21 10:02:25 [WARN ](o.a.k.c.p.i.Sender :257) Got error produce response with correlation id 490092 on topic-partition test3_a2_mobileDictClient_android_uid_imei-7, retrying (29 attempts left). Error: NOT_LEADER_FOR_PARTITION 2016-08-21 10:02:25 [WARN ](o.a.k.c.p.i.Sender :257) Got error produce response with correlation id 490092 on topic-partition test3_a2_mobileDictClient_android_uid_imei-5, retrying (28 attempts left). Error: NOT_LEADER_FOR_PARTITION 2016-08-21 10:02:50 [WARN ](o.a.k.c.p.i.Sender :257) Got error produce response with correlation id 2081 on topic-partition samzaMetrics-5, retrying (29 attempts left). Error: REQUEST_TIMED_OUT 2016-08-21 10:02:53 [WARN ](o.a.k.c.p.i.Sender :257) Got error produce response with correlation id 2082 on topic-partition samzaMetrics-5, retrying (28 attempts left). Error: NOT_LEADER_FOR_PARTITION 2016-08-21 10:03:04 [WARN ](o.a.k.c.p.i.Sender :257) Got error produce response with correlation id 490320 on topic-partition test3_a2_mobileDictClient_android_uid_imei-3, retrying (29 attempts left). Error: NOT_LEADER_FOR_PARTITION 2016-08-21 10:03:04 [INFO ](o.a.s.s.k.KafkaSystemConsumer :55 ) Abdicating for [outlog_mobileDictClient,3] 2016-08-21 10:03:04 [INFO ](o.a.s.s.k.KafkaSystemConsumer :55 ) Refreshing brokers for: Map([outlog_mobileDictClient,3] -> 3503408869) 2016-08-21 10:03:04 [INFO ](o.a.s.s.k.KafkaSystemConsumer :55 ) Refreshing brokers for: Map([outlog_mobileDictClient,3] -> 3503408869) 2016-08-21 10:03:04 [INFO ](k.u.VerifiableProperties :68 ) Verifying properties 2016-08-21 10:03:04 [INFO ](k.u.VerifiableProperties :68 ) Property client.id is overridden to samza_consumer-test3_a2_mobileDictClient_android_init_task-1-1471705851787-2 2016-08-21 10:03:04 [INFO ](k.u.VerifiableProperties :68 ) Property metadata.broker.list is overridden to hd020:9092,hd035:9092,hd036:9092 2016-08-21 10:03:04 [INFO ](k.u.VerifiableProperties :68 ) Property request.timeout.ms is overridden to 30000 2016-08-21 10:03:04 [INFO ](k.c.ClientUtils$ :68 ) Fetching metadata from broker id:1,host:hd035,port:9092 with correlation id 1 for 1 topic(s) Set(outlog_mobileDictClient) 2016-08-21 10:03:04 [INFO ](o.a.s.s.k.KafkaSystemConsumer :55 ) Refreshing brokers for: Map([outlog_mobileDictClient,3] -> 3503408869) 2016-08-21 10:03:04 [INFO ](k.p.SyncProducer :68 ) Connected to hd035:9092 for producing 2016-08-21 10:03:04 [INFO ](k.p.SyncProducer :68 ) Disconnecting from hd035:9092 2016-08-21 10:03:04 [INFO ](o.a.s.s.k.GetOffset :55 ) Validating offset 3503408869 for topic and partition [outlog_mobileDictClient,3] 2016-08-21 10:03:04 [INFO ](o.a.s.s.k.GetOffset :55 ) Validating offset 3503408869 for topic and partition [outlog_mobileDictClient,3] 2016-08-21 10:03:04 [INFO ](o.a.s.s.k.GetOffset :55 ) Validating offset 3503408869 for topic and partition [outlog_mobileDictClient,3] 2016-08-21 10:03:04 [WARN ](o.a.s.s.k.KafkaSystemConsumer :66 ) While refreshing brokers for [outlog_mobileDictClient,3]: kafka.common.NotLeaderForPartitionException. Retrying. 2016-08-21 10:03:04 [WARN ](o.a.s.s.k.KafkaSystemConsumer :66 ) While refreshing brokers for [outlog_mobileDictClient,3]: kafka.common.NotLeaderForPartitionException. Retrying. 2016-08-21 10:03:04 [WARN ](o.a.s.s.k.KafkaSystemConsumer :66 ) While refreshing brokers for [outlog_mobileDictClient,3]: kafka.common.NotLeaderForPartitionException. Retrying. 2016-08-21 10:03:04 [INFO ](o.a.s.s.k.GetOffset :55 ) Validating offset 3503408869 for topic and partition [outlog_mobileDictClient,3] 2016-08-21 10:03:05 [WARN ](o.a.s.s.k.KafkaSystemConsumer :66 ) While refreshing brokers for [outlog_mobileDictClient,3]: kafka.common.NotLeaderForPartitionException. Retrying. 2016-08-21 10:03:05 [INFO ](o.a.s.s.k.GetOffset :55 ) Validating offset 3503408869 for topic and partition [outlog_mobileDictClient,3] 2016-08-21 10:03:05 [WARN ](o.a.s.s.k.KafkaSystemConsumer :66 ) While refreshing brokers for [outlog_mobileDictClient,3]: kafka.common.NotLeaderForPartitionException. Retrying. 2016-08-21 10:03:05 [INFO ](o.a.s.s.k.GetOffset :55 ) Validating offset 3503408869 for topic and partition [outlog_mobileDictClient,3] 2016-08-21 10:03:05 [WARN ](o.a.s.s.k.KafkaSystemConsumer :66 ) While refreshing brokers for [outlog_mobileDictClient,3]: kafka.common.NotLeaderForPartitionException. Retrying. 2016-08-21 10:03:06 [INFO ](o.a.s.s.k.GetOffset :55 ) Validating offset 3503408869 for topic and partition [outlog_mobileDictClient,3] 2016-08-21 10:03:06 [WARN ](o.a.s.s.k.KafkaSystemConsumer :66 ) While refreshing brokers for [outlog_mobileDictClient,3]: kafka.common.NotLeaderForPartitionException. Retrying. 2016-08-21 10:03:06 [INFO ](o.a.s.s.k.GetOffset :55 ) Validating offset 3503408869 for topic and partition [outlog_mobileDictClient,3] 2016-08-21 10:03:06 [WARN ](o.a.s.s.k.KafkaSystemConsumer :66 ) While refreshing brokers for [outlog_mobileDictClient,3]: kafka.common.NotLeaderForPartitionException. Retrying. 2016-08-21 10:03:06 [INFO ](o.a.s.s.k.GetOffset :55 ) Validating offset 3503408869 for topic and partition [outlog_mobileDictClient,3] 2016-08-21 10:03:06 [WARN ](o.a.s.s.k.KafkaSystemConsumer :66 ) While refreshing brokers for [outlog_mobileDictClient,3]: kafka.common.NotLeaderForPartitionException. Retrying. 2016-08-21 10:03:06 [INFO ](o.a.s.s.k.GetOffset :55 ) Validating offset 3503408869 for topic and partition [outlog_mobileDictClient,3] 2016-08-21 10:03:06 [WARN ](o.a.s.s.k.KafkaSystemConsumer :66 ) While refreshing brokers for [outlog_mobileDictClient,3]: kafka.common.NotLeaderForPartitionException. Retrying. 2016-08-21 10:03:06 [INFO ](o.a.s.s.k.GetOffset :55 ) Validating offset 3503408869 for topic and partition [outlog_mobileDictClient,3] 2016-08-21 10:03:06 [INFO ](o.a.s.s.k.GetOffset :55 ) Validating offset 3503408869 for topic and partition [outlog_mobileDictClient,3] 2016-08-21 10:03:06 [WARN ](o.a.s.s.k.KafkaSystemConsumer :66 ) While refreshing brokers for [outlog_mobileDictClient,3]: kafka.common.NotLeaderForPartitionException. Retrying. 2016-08-21 10:03:06 [WARN ](o.a.s.s.k.KafkaSystemConsumer :66 ) While refreshing brokers for [outlog_mobileDictClient,3]: kafka.common.NotLeaderForPartitionException. Retrying. 2016-08-21 10:03:06 [WARN ](o.a.k.c.p.i.Sender :257) Got error produce response with correlation id 490330 on topic-partition test3_a2_mobileDictClient_android_uid_imei-3, retrying (28 attempts left). Error: NOT_LEADER_FOR_PARTITION 2016-08-21 10:03:06 [WARN ](o.a.k.c.p.i.Sender :257) Got error produce response with correlation id 490330 on topic-partition test3_a2_mobileDictClient_android_uid_imei-4, retrying (29 attempts left). Error: NOT_LEADER_FOR_PARTITION 2016-08-21 10:03:06 [WARN ](o.a.k.c.p.i.Sender :257) Got error produce response with correlation id 490330 on topic-partition test3_a2_mobileDictClient_android_uid_imei-6, retrying (29 attempts left). Error: NOT_LEADER_FOR_PARTITION 2016-08-21 10:03:07 [INFO ](o.a.s.s.k.GetOffset :55 ) Validating offset 3503408869 for topic and partition [outlog_mobileDictClient,3] 2016-08-21 10:03:07 [WARN ](o.a.s.s.k.KafkaSystemConsumer :66 ) While refreshing brokers for [outlog_mobileDictClient,3]: kafka.common.NotLeaderForPartitionException. Retrying. 2016-08-21 10:03:07 [INFO ](o.a.s.s.k.GetOffset :55 ) Validating offset 3503408869 for topic and partition [outlog_mobileDictClient,3] 2016-08-21 10:03:07 [WARN ](o.a.s.s.k.KafkaSystemConsumer :66 ) While refreshing brokers for [outlog_mobileDictClient,3]: kafka.common.NotLeaderForPartitionException. Retrying. 2016-08-21 10:03:07 [INFO ](o.a.s.s.k.GetOffset :55 ) Validating offset 3503408869 for topic and partition [outlog_mobileDictClient,3] 2016-08-21 10:03:07 [WARN ](o.a.s.s.k.KafkaSystemConsumer :66 ) While refreshing brokers for [outlog_mobileDictClient,3]: kafka.common.NotLeaderForPartitionException. Retrying. 2016-08-21 10:03:07 [WARN ](o.a.k.c.p.i.Sender :257) Got error produce response with correlation id 490335 on topic-partition test3_a2_mobileDictClient_android_uid_imei-3, retrying (27 attempts left). Error: NOT_LEADER_FOR_PARTITION 2016-08-21 10:03:07 [WARN ](o.a.k.c.p.i.Sender :257) Got error produce response with correlation id 490335 on topic-partition test3_a2_mobileDictClient_android_uid_imei-4, retrying (28 attempts left). Error: NOT_LEADER_FOR_PARTITION 2016-08-21 10:03:07 [WARN ](o.a.k.c.p.i.Sender :257) Got error produce response with correlation id 490335 on topic-partition test3_a2_mobileDictClient_android_uid_imei-6, retrying (28 attempts left). Error: NOT_LEADER_FOR_PARTITION 2016-08-21 10:03:07 [WARN ](o.a.k.c.p.i.Sender :257) Got error produce response with correlation id 490336 on topic-partition test3_a2_mobileDictClient_android_uid_imei-3, retrying (26 attempts left). Error: NOT_LEADER_FOR_PARTITION 2016-08-21 10:03:07 [WARN ](o.a.k.c.p.i.Sender :257) Got error produce response with correlation id 490336 on topic-partition test3_a2_mobileDictClient_android_uid_imei-4, retrying (27 attempts left). Error: NOT_LEADER_FOR_PARTITION
