Hi Andy, I remember that I was testing a job with almost exactly the same setup as part of the Flink 1.3.2 release testing. The command I used to start my job is roughly this:
HADOOP_CONF_DIR=/etc/hadoop/conf bin/flink run -c my.main.Class -m yarn-cluster -yn 5 -ys 2 ... i.e. I export the proper hadoop config dir and I run a per-job YARN cluster. I think I also exported the result of "hadoop classpath" as HADOOP_CLASSPATH. Best, Aljoscha > On 10. Oct 2017, at 16:43, Andy M. <ajm2...@gmail.com> wrote: > > Hello, > > Bowen: Unless I am missing something, it says there needs to be no setup > on EMR, Each topic says: "You don’t have to configure this manually if you > are running Flink on EMR." S3 access from CLI works fine on my clusters. > > Chen: Thank you for this, I will look into this if I am unable to get this > running on YARN successfully. > > Stephan: Removing the said library causes the flink > (flink-1.3.2/bin/flink) bash script to fail. The underlying Java needs > this to work. I tried explicitly setting the classpath for the java call > as well to point to the hadoop library jars. This is the original java > command that I was trying to run: > > java > -Dlog.file=/home/hadoop/flink-1.3.2/log/flink-hadoop-client-ip-172-31-19-27.log > -Dlog4j.configuration=file:/home/hadoop/flink-1.3.2/conf/log4j-cli.properties > -Dlogback.configurationFile=file:/home/hadoop/flink-1.3.2/conf/logback.xml > -classpath > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3.2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/hadoop/conf: > org.apache.flink.client.CliFrontend run -m yarn-cluster -yn 1 > /home/hadoop/flink-consumer.jar > > > This is what I changed it too(removing the shadded-hadoop2-uber jar and > adding in the hadoop folder): > > java > -Dlog.file=/home/hadoop/flink-1.3.2/log/flink-hadoop-client-ip-172-31-19-27.log > -Dlog4j.configuration=file:/home/hadoop/flink-1.3.2/conf/log4j-cli.properties > -Dlogback.configurationFile=file:/home/hadoop/flink-1.3.2/conf/logback.xml > -classpath > /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/curator-client-2.7.1.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/jersey-json-1.9.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.10.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/usr/lib/hadoop/lib/jersey-server-1.9.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/gson-2.2.4.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/httpcore-4.4.4.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/junit-4.11.jar:/etc/hadoop/conf > org.apache.flink.client.CliFrontend run -m yarn-cluster -yn 1 > /home/hadoop/flink-consumer.jar > > The later throws the following error: > > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/util/VersionInfo > at > org.apache.flink.runtime.util.EnvironmentInformation.logEnvironmentInfo(EnvironmentInformation.java:283) > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1124) > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.util.VersionInfo > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 2 more > > Simply removing the .jar from the folder causes the same error. > > Thank you > > > > > On Mon, Oct 9, 2017 at 5:46 AM, Stephan Ewen <se...@apache.org> wrote: > >> Hi! >> >> It looks like multiple Hadoop versions are in the classpath. Flink's hadoop >> jar and the EMR Hadoop jars. >> I would simply drop Flink's own Hadoop dependency and only use the EMR >> Hadoop jars. >> >> Delete the 'flink-shaded-hadoop2-uber' jar from Flink, and make sure the >> setup is such that the Hadoop lib environment variable is set. Then it >> should not have conflicts any more. >> >> >> >> On Sun, Oct 8, 2017 at 12:08 AM, Chen Qin <qinnc...@gmail.com> wrote: >> >>> Attached my side project verified working to deploy jobmanager and >>> taskmanager as stateless service(non yarn/mesos), configuration here >>> >>> https://github.com/chenqin/flink-jar/tree/master/config/hadoop >>> >>> more detail here >>> https://github.com/chenqin/flink-jar/blob/master/src/ >>> main/java/FlinkBootstrap.java#L49 >>> >>> On Fri, Oct 6, 2017 at 10:26 PM, Bowen Li <bowen...@offerupnow.com> >> wrote: >>> >>>> Hi Andy, >>>> >>>> I believe it's because you didn't set your s3 impl correctly. Try to >> set >>>> your core-site.xml by following https://ci.apache.org/ >>>> projects/flink/flink-docs-release-1.4/ops/deployment/ >>>> aws.html#s3afilesystem- >>>> recommended >>>> >>>> Bowen >>>> >>>> On Fri, Oct 6, 2017 at 7:59 AM, Andy M. <ajm2...@gmail.com> wrote: >>>> >>>>> Hi Till, >>>>> >>>>> Seems like everything is in line there. hadoop-common.jar -> >>>>> hadoop-common-2.7.3-amzn-3.jar >>>>> >>>>> And when i decompiled that jar I see public void >>>> addResource(Configuration >>>>> conf) in org/apache/hadoop/conf/Configuration.java >>>>> >>>>> I agree that an incorrect version of the jar is probably being run, >> is >>>>> there a way to limit the classpath for the TaskManager when starting >>> the >>>>> job? >>>>> >>>>> Thank you >>>>> >>>>> On Fri, Oct 6, 2017 at 6:49 AM, Till Rohrmann <trohrm...@apache.org> >>>>> wrote: >>>>> >>>>>> Hi Andy, >>>>>> >>>>>> could you check which Hadoop version this jar >>>>>> /usr/lib/hadoop/hadoop-common.jar is? Maybe also checking whether >>> the >>>>>> contained hadoop Configuration class has the method >>>>>> Configuration.addResource(Lorg/apache/hadoop/conf/ >> Configuration;)V. >>>>> Maybe >>>>>> this jar is the culprit because it comes from a different Hadoop >>>> version. >>>>>> >>>>>> Cheers, >>>>>> Till >>>>>> >>>>>> >>>>>> On Thu, Oct 5, 2017 at 4:22 PM, Andy M. <ajm2...@gmail.com> wrote: >>>>>> >>>>>>> Hi Till, >>>>>>> >>>>>>> I believe this is what you are looking for, classpath is much >>> bigger >>>>> for >>>>>>> the task manager. I can also post the whole log file if needed: >>>>>>> >>>>>>> 2017-10-05 14:17:53,038 INFO org.apache.flink.yarn. >>>>>> YarnTaskManagerRunner >>>>>>> - Classpath: >>>>>>> flink-consumer.jar:lib/flink-dist_2.11-1.3.2.jar:lib/flink- >>>>>>> python_2.11-1.3.2.jar:lib/flink-shaded-hadoop2-uber-1.3. >>>>>>> 2.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar: >>>>>>> log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/ >>>>>>> etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.7.3- >>>>>>> amzn-3-tests.jar:/usr/lib/hadoop/hadoop-annotations-2.7. >>>>>>> 3-amzn-3.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/ >>>>>>> hadoop/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- >>>>>>> nfs-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-streaming-2.7.3- >>>>>>> amzn-3.jar:/usr/lib/hadoop/hadoop-ant-2.7.3-amzn-3.jar:/ >>>>>>> usr/lib/hadoop/hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/ >>>>>>> hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop- >>>>>>> streaming.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/ >>>>>>> hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-sls.jar:/ >>>>>>> usr/lib/hadoop/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/ >>>>>>> hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-extras-2.7. >>>>>>> 3-amzn-3.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/ >>>>>>> hadoop/hadoop-common-2.7.3-amzn-3.jar:/usr/lib/hadoop/ >>>>>>> hadoop-annotations.jar:/usr/lib/hadoop/hadoop-openstack-2. >>>>>>> 7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-archives-2.7.3- >>>>>>> amzn-3.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/ >>>>>>> hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-openstack. >>>>>>> jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/ >>>>>>> hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- >>>>>>> datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop- >>>>>>> archives.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/ >>>>>>> hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-rumen-2.7.3- >>>>>>> amzn-3.jar:/usr/lib/hadoop/hadoop-sls-2.7.3-amzn-3.jar:/ >>>>>>> usr/lib/hadoop/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/lib/ >>>>>>> hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2. >>>>>>> 2.2.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating. >>>>>>> jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/ >>>>>>> lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/ >>>>>>> httpcore-4.4.4.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4. >>>>>>> 1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0. >>>>>>> jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/ >>>>>>> lib/activation-1.1.jar:/usr/lib/hadoop/lib/jersey-server- >>>>>>> 1.9.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/ >>>>>>> usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/ >>>>>>> gson-2.2.4.jar:/usr/lib/hadoop/lib/commons-digester-1. >>>>>>> 8.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/ >>>>>>> lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/ >>>>>>> apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons- >>>>>>> httpclient-3.1.jar:/usr/lib/hadoop/lib/curator-client-2.7. >>>>>>> 1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/ >>>>>>> lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/ >>>>>>> commons-net-3.1.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/ >>>>>>> usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/ >>>>>>> xmlenc-0.52.jar:/usr/lib/hadoop/lib/jersey-json-1.9. >>>>>>> jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/ >>>>>>> commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/log4j-1.2.17. >>>>>>> jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/ >>>>>>> usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/ >>>>>>> jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/netty-3.6.2. >>>>>>> Final.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/ >>>>>>> lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/api-asn1- >>>>>>> api-1.0.0-M20.jar:/usr/lib/hadoop/lib/jackson-mapper-asl- >>>>>>> 1.9.13.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar: >>>>>>> /usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/ >>>>>>> jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons- >>>>>>> cli-1.2.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/ >>>>>>> lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/ >>>>>>> apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/ >>>>>>> commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/junit-4. >>>>>>> 11.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar: >>>>>>> /usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/ >>>>>>> lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7. >>>>>>> 10.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/ >>>>>>> usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/ >>>>>>> lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10. >>>>>>> jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/ >>>>>>> hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/ >>>>>>> curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/jackson- >>>>>>> jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0. >>>>>>> 4.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/ >>>>>>> usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.7.3-amzn-3.jar:/ >>>>>>> usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3-amzn-3-tests.jar:/ >>>>>>> usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/ >>>>>>> hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3- >>>>>>> amzn-3.jar:/usr/lib/hadoop-hdfs/lib/htrace-core-3.1.0- >>>>>>> incubating.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java- >>>>>>> 2.5.0.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8. >>>>>>> jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/ >>>>>>> lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/ >>>>>>> lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/ >>>>>>> commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52. >>>>>>> jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/ >>>>>>> usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/ >>>>>>> hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/ >>>>>>> netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/xml-apis- >>>>>>> 1.3.04.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/ >>>>>>> hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/ >>>>>>> lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/ >>>>>>> hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop- >>>>>>> hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/ >>>>>>> lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/servlet- >>>>>>> api-2.5.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar: >>>>>>> /usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/ >>>>>>> hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/ >>>>>>> lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons- >>>>>>> logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/jettison-1.1. >>>>>>> jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop- >>>>>>> mapreduce/htrace-core-3.1.0-incubating.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/ >>>>>> hadoop-mapreduce/hadoop- >>>>>>> mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> httpclient-4.5.3.jar:/usr/lib/hadoop-mapreduce/hadoop- >>>>>>> mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> hadoop-mapreduce-client-common-2.7.3-amzn-3.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/httpcore-4.4.4.jar:/usr/lib/hadoop- >>>>>>> mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> hadoop-mapreduce-client-core-2.7.3-amzn-3.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> jmespath-java-1.11.160.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop- >>>>>>> mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> hadoop-mapreduce-client-jobclient-2.7.3-amzn-3-tests. >>>>>>> jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/hadoop-streaming-2.7.3-amzn-3.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/hadoop-ant-2.7.3-amzn-3.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> hadoop-mapreduce-client-app-2.7.3-amzn-3.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3-amzn- >>>>>>> 3.jar:/usr/lib/hadoop-mapreduce/commons-digester-1. >>>>>> 8.jar:/usr/lib/hadoop- >>>>>>> mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3-amzn- >>>>>>> 3.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13. >>>>>>> jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop- >>>>>>> mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/commons- >>>>>>> net-3.1.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/jersey-json- >>>>>>> 1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/avro-1.7.4.jar:/usr/lib/hadoop- >>>>>>> mapreduce/aws-java-sdk-core-1.11.160.jar:/usr/lib/hadoop- >>>>>>> mapreduce/aws-java-sdk-s3-1.11.160.jar:/usr/lib/hadoop- >>>>>>> mapreduce/jackson-dataformat-cbor-2.6.6.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/ion-java-1.0.2.jar:/usr/lib/hadoop- >>>>>>> mapreduce/azure-storage-2.0.0.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> hadoop-mapreduce-client-shuffle-2.7.3-amzn-3.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop- >>>>>>> mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop- >>>>>>> mapreduce/hadoop-extras-2.7.3-amzn-3.jar:/usr/lib/hadoop- >>>>>>> mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/jsch- >>>>>>> 0.1.42.jar:/usr/lib/hadoop-mapreduce/joda-time-2.8.1.jar: >>>>>>> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2. >>>>>>> 7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2. >>>>>>> 2.3-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce- >>>>>>> client-hs.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final. >>>>>>> jar:/usr/lib/hadoop-mapreduce/mockito-all-1.8.5.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/hadoop-openstack-2.7.3-amzn-3.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/hadoop-mapreduce-client- >>>>>>> common.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.7. >>>>>>> 3-amzn-3.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/hadoop-mapreduce-client- >>>>>>> jobclient.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop- >>>>>>> mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop- >>>>>>> mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> hadoop-mapreduce-client-jobclient-2.7.3-amzn-3.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/aws-java-sdk-kms-1.11.160.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop- >>>>>>> mapreduce/hadoop-datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop- >>>>>>> mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/hadoop- >>>>>>> archives.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1. >>>>>>> 9.13.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2. >>>>>>> jar:/usr/lib/hadoop-mapreduce/jackson-core-2.6.6.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop- >>>>>>> mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/apacheds- >>>>>>> kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> hadoop-aws.jar:/usr/lib/hadoop-mapreduce/hadoop-auth. >>>>>>> jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.7.3-amzn-3.jar: >>>>>>> /usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar: >>>>>>> /usr/lib/hadoop-mapreduce/hadoop-sls-2.7.3-amzn-3.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/jackson-annotations-2.6.6.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/junit-4.11.jar:/usr/lib/hadoop- >>>> mapreduce/commons- >>>>>>> configuration-1.6.jar:/usr/lib/hadoop-mapreduce/api-util- >>>>>>> 1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/hamcrest-core-1.3.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/jackson-databind-2.6.6.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> zookeeper-3.4.10.jar:/usr/lib/hadoop-mapreduce/commons-lang- >>>>>>> 2.6.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar: >>>>>>> /usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/ >>>>>>> lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/ >>>>>> lib/hadoop-mapreduce/lib/ >>>>>>> snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/ >>>>>>> jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/ >>>>>>> paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jersey- >>>>>>> guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4. >>>>>>> jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop- >>>>>>> mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop- >>>>>>> mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop- >>>>>>> mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/ >>>>>>> lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/jackson- >>>>>>> mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/ >>>>>>> guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/xz-1. >>>>>>> 0.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/ >>>>>>> usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/ >>>>>>> hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop- >>>>>>> mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/ >>>>>>> hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop- >>>> yarn-server- >>>>>>> sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn- >>>>>>> server-web-proxy-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ >>>>>>> hadoop-yarn-common-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ >>>>>>> hadoop-yarn-server-tests-2.7.3-amzn-3.jar:/usr/lib/hadoop- >>>>>>> yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/ >>>>>>> hadoop-yarn/hadoop-yarn-applications-distributedshell- >>>>>>> 2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/hadoop-yarn- >>>>>>> server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop- >>> yarn-server- >>>>>>> sharedcachemanager-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ >>>>>>> hadoop-yarn-server-applicationhistoryservice.jar: >>>>>>> /usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/ >>>>>>> lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.7.3-amzn- >>>>>>> 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.7.3-amzn- >>>>>>> 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- >>>>>>> applicationhistoryservice-2.7.3-amzn-3.jar:/usr/lib/hadoop- >>>>>>> yarn/hadoop-yarn-server-common-2.7.3-amzn-3.jar:/usr/ >>>>>>> lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop- >>>>>>> yarn/hadoop-yarn-api-2.7.3-amzn-3.jar:/usr/lib/hadoop- >>>>>>> yarn/hadoop-yarn-server-resourcemanager-2.7.3-amzn-3. >>>>>>> jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications- >>>>>>> unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop- >>>>>>> yarn-registry-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/ >>>>>>> hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn- >>>>>>> common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web- >>>>>>> proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server- >>>>>>> nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api. >>>>>>> jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications- >>>>>>> unmanaged-am-launcher-2.7.3-amzn-3.jar:/usr/lib/hadoop- >>>>>>> yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/jaxb- >>>>>>> api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0. >>>>>>> jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/ >>>>>>> lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/ >>>>>>> hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/ >>>>>>> lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1. >>>>>>> 9.13.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/ >>>>>>> usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/ >>>>>>> hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/ >>>>>>> lib/jersey-json-1.9.jar:/usr/lib/hadoop-yarn/lib/jersey- >>>>>>> client-1.9.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17. >>>>>>> jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/ >>>>>>> usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop- >>>>>>> yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/ >>>>>>> netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/ >>>>>>> aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/ >>>>>>> usr/lib/hadoop-yarn/lib/zookeeper-3.4.10-tests.jar:/ >>>>>>> usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar: >>>>>>> /usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/ >>>>>>> lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/ >>>>>>> hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/guice- >>>>>>> 3.0.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13. >>>>>>> jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/ >>>>>>> hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/ >>>>>>> lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/commons- >>>>>>> collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/stax-api- >>>>>>> 1.0-2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/ >>>>>>> usr/lib/hadoop-yarn/lib/zookeeper-3.4.10.jar:/usr/lib/ >>>>>>> hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/ >>>>>>> lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/jackson- >>>>>>> jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/commons- >>>>>>> logging-1.1.3.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar: >>>>>>> /usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/ >>>>>>> share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/ >>>>>>> jmespath-java-1.11.129.jar:/usr/share/aws/emr/emrfs/lib/ >>>>>>> bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/jcl- >>>>>>> over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/ion- >>>>>>> java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7. >>>>>>> 21.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/ >>>>>>> usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/ >>>>>>> share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/ >>>>>>> share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.18.0.jar:/ >>>>>>> usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/ >>>>>>> lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/ >>>>>>> share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/ >>>>>>> aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/ >>>>>>> aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.0.0.jar:/ >>>>>>> usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/ >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11. >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api- >>>>>>> gateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- >>>>>>> java-sdk-kinesis-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>> aws-java-sdk- >>>>>>> elasticloadbalancingv2-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-rekognition-1.11.160.jar:/usr/ >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.160. >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> elasticbeanstalk-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>> aws-java-sdk-cloudhsm-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-iam-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-cognitoidp-1.11.160.jar:/usr/ >>>>>>> share/aws/aws-java-sdk/aws-java-sdk- >> mechanicalturkrequester-1.11. >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> marketplacecommerceanalytics-1.11.160.jar:/usr/share/aws/ >>>>>>> aws-java-sdk/aws-java-sdk-snowball-1.11.160.jar:/usr/ >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.160.jar:/ >>>>>>> usr/share/aws/aws-java-sdk/jmespath-java-1.11.160.jar:/ >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11. >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1. >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> simpleworkflow-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>> aws-java-sdk-machinelearning-1.11.160.jar:/usr/share/aws/ >>>>>>> aws-java-sdk/aws-java-sdk-pinpoint-1.11.160.jar:/usr/ >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.160. >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.160. >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11. >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> greengrass-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- >>>>>>> java-sdk-ecs-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>> aws-java-sdk-xray-1.11.160.jar:/usr/share/aws/aws-java- >>>>>>> sdk/aws-java-sdk-cloudtrail-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.160. >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> elasticloadbalancing-1.11.160.jar:/usr/share/aws/aws-java- >>>>>>> sdk/aws-java-sdk-codepipeline-1.11.160.jar:/usr/share/aws/ >>>>>>> aws-java-sdk/aws-java-sdk-cloudwatch-1.11.160.jar:/usr/ >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.160.jar:/ >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.160. >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>> stepfunctions-1.11.160.jar:/ >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11. >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1. >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1. >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> datapipeline-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>> aws-java-sdk-s3-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>> aws-java-sdk-dax-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>> aws-java-sdk-opsworks-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-servicecatalog-1.11.160.jar:/ >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11. >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1. >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> directory-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- >>>>>>> java-sdk-opsworkscm-1.11.160.jar:/usr/share/aws/aws-java- >>>>>> sdk/aws-java-sdk- >>>>>>> cloudformation-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>> aws-java-sdk-cloudfront-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-budgets-1.11.160.jar:/usr/share/aws/ >>>>>>> aws-java-sdk/aws-java-sdk-clouddirectory-1.11.160.jar:/ >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1. >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1. >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> marketplaceentitlement-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-dynamodb-1.11.160.jar:/usr/ >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.160. >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>> elastictranscoder-1.11.160. >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>> organizations-1.11.160.jar:/ >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11. >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1. >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> servermigration-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>> aws-java-sdk-events-1.11.160.jar:/usr/share/aws/aws-java- >>>>>> sdk/aws-java-sdk- >>>>>>> applicationautoscaling-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-health-1.11.160.jar:/usr/share/aws/ >>>>>>> aws-java-sdk/aws-java-sdk-kms-1.11.160.jar:/usr/share/aws/ >>>>>>> aws-java-sdk/aws-java-sdk-logs-1.11.160.jar:/usr/share/ >>>>>>> aws/aws-java-sdk/aws-java-sdk-codestar-1.11.160.jar:/usr/ >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.160.jar: >>>>>>> /usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11. >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> marketplacemeteringservice-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-sns-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-batch-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-waf-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-simpledb-1.11.160.jar:/usr/ >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.160. >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.160. >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11. >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> devicefarm-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- >>>>>>> java-sdk-cognitoidentity-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-lexmodelbuilding-1.11.160.jar: >>>>>>> /usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1. >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> elasticache-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- >>>>>>> java-sdk-costandusagereport-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-discovery-1.11.160.jar:/usr/ >>>>>>> share/aws/aws-java-sdk/aws-java-sdk- >> resourcegroupstaggingapi-1.11. >>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1. >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda- >>>>>>> 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> workdocs-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws- >>>>>>> java-sdk-code-generator-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-cognitosync-1.11.160.jar:/usr/ >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.160.jar:/ >>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.160.jar: >>>>>>> /usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.160. >>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1. >>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> storagegateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>> aws-java-sdk-lightsail-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-acm-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-1.11.160.jar:/usr/share/aws/aws- >>>>>>> java-sdk/aws-java-sdk-glacier-1.11.160.jar:/usr/share/aws/ >>>>>>> aws-java-sdk/aws-java-sdk-ecr-1.11.160.jar:/usr/share/aws/ >>>>>>> aws-java-sdk/aws-java-sdk-support-1.11.160.jar:/usr/ >>>>>>> share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin- >>>>>>> 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk- >>>>>>> elasticsearch-1.11.160.jar:/usr/share/aws/aws-java-sdk/ >>>>>>> aws-java-sdk-iot-1.11.160.jar >>>>>>> >>>>>>> Thank you >>>>>>> >>>>>>> On Thu, Oct 5, 2017 at 5:25 AM, Till Rohrmann < >>> trohrm...@apache.org> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Andy, >>>>>>>> >>>>>>>> the CliFrontend is not executed via Yarn, thus, it is not >>> affected >>>> by >>>>>>>> dependencies which are added due to the underlying Yarn >> cluster. >>>>>>> Therefore, >>>>>>>> it would be helpful to look at the TaskManager logs. Either you >>>> have >>>>>>>> enabled log aggregation on your Yarn cluster, then you can >> obtain >>>> the >>>>>>> logs >>>>>>>> via `yarn logs -applicationId <application ID>` or you have to >>>>> retrieve >>>>>>>> them from the machines where they were running (either by going >>>>>> directly >>>>>>>> there or via the Yarn web interface). >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Till >>>>>>>> >>>>>>>> On Wed, Oct 4, 2017 at 4:27 PM, Andy M. <ajm2...@gmail.com> >>> wrote: >>>>>>>> >>>>>>>>> Hi Till, >>>>>>>>> >>>>>>>>> That is actually the classpath used by the flink bash >>> script(that >>>>>>>> launches >>>>>>>>> the jar using the java command). I changed the execute to an >>>> echo, >>>>>> and >>>>>>>>> grabbed that for the CLI arguments. >>>>>>>>> >>>>>>>>> I believe this is the class path from the log file(although >> it >>>>> might >>>>>>> not >>>>>>>> be >>>>>>>>> the taskmanager log, is that any different from what would be >>> in >>>> my >>>>>>>>> flink-1.3.2/log folder?): >>>>>>>>> >>>>>>>>> 2017-10-02 20:03:26,450 INFO org.apache.flink.client. >>>> CliFrontend >>>>>>>>> - Classpath: >>>>>>>>> /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/ >>>>>>>>> home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3. >>>>>>>>> 2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/ >>>>>>>>> hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/ >>>>>>>>> hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/ >>>>> hadoop/conf: >>>>>>>>> >>>>>>>>> If that doesn't seem right, and you can point me in the right >>>>>> direction >>>>>>>> as >>>>>>>>> to where the TaskManager logs would be, I would be happy to >>> grab >>>>> the >>>>>>>>> information your looking for. >>>>>>>>> >>>>>>>>> Thank you >>>>>>>>> >>>>>>>>> On Wed, Oct 4, 2017 at 3:27 AM, Till Rohrmann < >>>>> trohrm...@apache.org> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Andy, >>>>>>>>>> >>>>>>>>>> this looks to me indeed like a dependency problem. I assume >>>> that >>>>>> EMR >>>>>>> or >>>>>>>>>> something else is pulling in an incompatible version of >>> Hadoop. >>>>>>>>>> >>>>>>>>>> The classpath you've posted, is this the one logged in the >>> log >>>>>> files >>>>>>>>>> (TaskManager log) or did you compile it yourself? In the >>> latter >>>>>> case, >>>>>>>> it >>>>>>>>>> would also be helpful to get access to the TaskManager >> logs. >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Till >>>>>>>>>> >>>>>>>>>> On Mon, Oct 2, 2017 at 10:20 PM, Andy M. < >> ajm2...@gmail.com> >>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Fabian, >>>>>>>>>>> >>>>>>>>>>> 1) I have looked at the linked docs, and from what I can >>> tell >>>>> no >>>>>>>> setup >>>>>>>>>>> should really need to be done to get Flink working(Other >>> than >>>>>>>>> downloading >>>>>>>>>>> the correct binaries, which I believe I did) >>>>>>>>>>> 2) I have downloaded the Flink 1.3.2 >>>> binaries(flink-1.3.2-bin- >>>>>>>>>>> hadoop27-scala_2.11.tgz >>>>>>>>>>> <http://apache.claz.org/flink/ >> flink-1.3.2/flink-1.3.2-bin- >>>>>>>>>>> hadoop27-scala_2.11.tgz>) >>>>>>>>>>> This is for hadoop 2.7.X, which matches EMR 5.8.0. >>>>>>>>>>> >>>>>>>>>>> I appreciate any help or guidance you can provide me in >>>> fixing >>>>> my >>>>>>>>>> problems, >>>>>>>>>>> please let me know if there is anything else I can >> provide >>>> you. >>>>>>>>>>> >>>>>>>>>>> Thank you >>>>>>>>>>> >>>>>>>>>>> On Mon, Oct 2, 2017 at 4:12 PM, Fabian Hueske < >>>>> fhue...@gmail.com >>>>>>> >>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Andy, >>>>>>>>>>>> >>>>>>>>>>>> I'm not an AWS expert, so I'll just check on some >> common >>>>>> issues. >>>>>>>>>>>> >>>>>>>>>>>> I guess you already had a look at the Flink docs for >>>> AWS/EMR >>>>>> but >>>>>>>> I'll >>>>>>>>>>> post >>>>>>>>>>>> the link just be to sure [1]. >>>>>>>>>>>> >>>>>>>>>>>> Since you are using Flink 1.3.2 (EMR 5.8.0 comes with >>> Flink >>>>>>> 1.3.1) >>>>>>>>> did >>>>>>>>>>> you >>>>>>>>>>>> built Flink yourself or did you download the binaries? >>>>>>>>>>>> Does the Hadoop version of the Flink build match the >>> Hadoop >>>>>>> version >>>>>>>>> of >>>>>>>>>>> EMR >>>>>>>>>>>> 5.8.0, i.e., Hadoop 2.7.x? >>>>>>>>>>>> >>>>>>>>>>>> Best, Fabian >>>>>>>>>>>> >>>>>>>>>>>> [1] >>>>>>>>>>>> https://ci.apache.org/projects/flink/flink-docs- >>>>>>>>>>> release-1.3/setup/aws.html >>>>>>>>>>>> >>>>>>>>>>>> 2017-10-02 21:51 GMT+02:00 Andy M. <ajm2...@gmail.com >>> : >>>>>>>>>>>> >>>>>>>>>>>>> Hi Fabian, >>>>>>>>>>>>> >>>>>>>>>>>>> Sorry, I just realized I forgot to include that part. >>>> The >>>>>>> error >>>>>>>>>>> returned >>>>>>>>>>>>> is: >>>>>>>>>>>>> >>>>>>>>>>>>> java.lang.NoSuchMethodError: >>>>>>>>>>>>> org.apache.hadoop.conf.Configuration.addResource( >>>>>>>>>>>> Lorg/apache/hadoop/conf/ >>>>>>>>>>>>> Configuration;)V >>>>>>>>>>>>> at com.amazon.ws.emr.hadoop.fs. >>>>> EmrFileSystem.initialize( >>>>>>>>>>>>> EmrFileSystem.java:93) >>>>>>>>>>>>> at org.apache.flink.runtime.fs. >>>> hdfs.HadoopFileSystem. >>>>>>>>>>>>> initialize(HadoopFileSystem.java:328) >>>>>>>>>>>>> at org.apache.flink.core.fs.FileSystem. >>>>>>>> getUnguardedFileSystem( >>>>>>>>>>>>> FileSystem.java:350) >>>>>>>>>>>>> at org.apache.flink.core.fs. >>>> FileSystem.get(FileSystem. >>>>>>>>> java:389) >>>>>>>>>>>>> at org.apache.flink.core.fs.Path. >>>>>>>> getFileSystem(Path.java:293) >>>>>>>>>>>>> at org.apache.flink.runtime.state.filesystem. >>>>>>>>>>>>> FsCheckpointStreamFactory.<init>( >>>>> FsCheckpointStreamFactory. >>>>>>>>> java:99) >>>>>>>>>>>>> at org.apache.flink.runtime.state.filesystem. >>>>>>> FsStateBackend. >>>>>>>>>>>>> createStreamFactory(FsStateBackend.java:282) >>>>>>>>>>>>> at org.apache.flink.contrib.streaming.state. >>>>>>>>> RocksDBStateBackend. >>>>>>>>>>>>> createStreamFactory(RocksDBStateBackend.java:273 >>>>>>>>>>>>> >>>>>>>>>>>>> I believe it has something to do with the classpath, >>> but >>>> I >>>>> am >>>>>>>>> unsure >>>>>>>>>>> why >>>>>>>>>>>> or >>>>>>>>>>>>> how to fix it. The classpath being used during the >>>>> execution >>>>>>> is: >>>>>>>>>>>>> /home/hadoop/flink-1.3.2/lib/ >>>> flink-python_2.11-1.3.2.jar:/ >>>>>>>>>>>>> home/hadoop/flink-1.3.2/ >>> lib/flink-shaded-hadoop2- >>>>>>>>>>>>> uber-1.3.2.jar:/home/hadoop/ >>>>> flink-1.3.2/lib/log4j-1.2. >>>>>>>>>>>>> 17.jar:/home/hadoop/flink-1. >>>>> 3.2/lib/slf4j-log4j12-1.7.7. >>>>>>>>>>>>> jar:/home/hadoop/flink-1. >>>>> 3.2/lib/flink-dist_2.11-1.3. >>>>>>>>>>>>> 2.jar::/etc/hadoop/conf: >>>>>>>>>>>>> >>>>>>>>>>>>> I decompiled flink-shaded-hadoop2-uber-1.3.2.jar >>> and >>>>> it >>>>>>>> seems >>>>>>>>>> the >>>>>>>>>>>>> addResource function does seem to be there. >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Oct 2, 2017 at 3:43 PM, Fabian Hueske < >>>>>>> fhue...@gmail.com >>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Andy, >>>>>>>>>>>>>> >>>>>>>>>>>>>> can you describe in more detail what exactly isn't >>>>> working? >>>>>>>>>>>>>> Do you see error messages in the log files or on >> the >>>>>> console? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Fabian >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2017-10-02 15:52 GMT+02:00 Andy M. < >>> ajm2...@gmail.com >>>>> : >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I am about to deploy my first Flink projects to >>>>>>> production, >>>>>>>>> but >>>>>>>>>> I >>>>>>>>>>> am >>>>>>>>>>>>>>> running into a very big hurdle. I am unable to >>>> launch >>>>> my >>>>>>>>> project >>>>>>>>>>> so >>>>>>>>>>>> it >>>>>>>>>>>>>> can >>>>>>>>>>>>>>> write to an S3 bucket. My project is running on >> an >>>> EMR >>>>>>>>> cluster, >>>>>>>>>>>> where >>>>>>>>>>>>> I >>>>>>>>>>>>>>> have installed Flink 1.3.2. I am using Yarn to >>>> launch >>>>>> the >>>>>>>>>>>> application, >>>>>>>>>>>>>> and >>>>>>>>>>>>>>> it seems to run fine unless I am trying to enable >>>> check >>>>>>>>>>>> pointing(with a >>>>>>>>>>>>>> S3 >>>>>>>>>>>>>>> target). I am looking to use RocksDB as my >>>>>> check-pointing >>>>>>>>>> backend. >>>>>>>>>>>> I >>>>>>>>>>>>>> have >>>>>>>>>>>>>>> asked a few places, and I am still unable to >> find a >>>>>>> solution >>>>>>>> to >>>>>>>>>>> this >>>>>>>>>>>>>>> problem. Here are my steps for creating a >> cluster, >>>> and >>>>>>>>> launching >>>>>>>>>>> my >>>>>>>>>>>>>>> application, perhaps I am missing a step. I'd be >>>> happy >>>>>> to >>>>>>>>>> provide >>>>>>>>>>>> any >>>>>>>>>>>>>>> additional information if needed. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> AWS Portal: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1) EMR -> Create Cluster >>>>>>>>>>>>>>> 2) Advanced Options >>>>>>>>>>>>>>> 3) Release = emr-5.8.0 >>>>>>>>>>>>>>> 4) Only select Hadoop 2.7.3 >>>>>>>>>>>>>>> 5) Next -> Next -> Next -> Create Cluster ( I >>> do >>>>> fill >>>>>>> out >>>>>>>>>>>>>>> names/keys/etc) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Once the cluster is up I ssh into the Master and >> do >>>> the >>>>>>>>>> following: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1 wget >>>>>>>>>>>>>>> http://apache.claz.org/flink/ >>>>>> flink-1.3.2/flink-1.3.2-bin- >>>>>>>>>>>>>>> hadoop27-scala_2.11.tgz >>>>>>>>>>>>>>> 2 tar -xzf flink-1.3.2-bin-hadoop27- >>>>> scala_2.11.tgz >>>>>>>>>>>>>>> 3 cd flink-1.3.2 >>>>>>>>>>>>>>> 4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4 >> -d >>>>>>>>>>>>>>> 5 Change conf/flink-conf.yaml >>>>>>>>>>>>>>> 6 ./bin/flink run -m yarn-cluster -yn 1 >>>>>>>>> ~/flink-consumer.jar >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> My conf/flink-conf.yaml I add the following >> fields: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> state.backend: rocksdb >>>>>>>>>>>>>>> state.backend.fs.checkpointdir: >>>>> s3:/bucket/location >>>>>>>>>>>>>>> state.checkpoints.dir: s3:/bucket/location >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> My program's checkpointing setup: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> env.enableCheckpointing(getCheckpointRate, >>>>>>>>>>> CheckpointingMode.EXACTLY_ >>>>>>>>>>>>>> ONCE) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> env.getCheckpointConfig. >>>> enableExternalizedCheckpoints( >>>>>>>>>>>>>>> ExternalizedCheckpointCleanup. >>>> RETAIN_ON_CANCELLATION) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> env.getCheckpointConfig. >>>> setMinPauseBetweenCheckpoints( >>>>>>>>>>>>>>> getCheckpointMinPause) >>>>>>>>>>>>>>> env.getCheckpointConfig. >> setCheckpointTimeout( >>>>>>>>>>>> getCheckpointTimeout) >>>>>>>>>>>>>>> env.getCheckpointConfig. >>>>>> setMaxConcurrentCheckpoints(1) >>>>>>>>>>>>>>> env.setStateBackend(new >>>> RocksDBStateBackend("s3:// >>>>>>>>>>>>> bucket/location", >>>>>>>>>>>>>>> true)) >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>
