Hi,
I can't spot anything obviously wrong with your config. I've been quite busy
lately with traveling and release preparations. I will get back to this and try
it myself, though, once I have time again.
Best,
Aljoscha
> On 12. Oct 2017, at 17:07, Andy M. <ajm2...@gmail.com> wrote:
>
> Hi Ajoscha,
>
> That didn't seem to do the trick either. Do the following look correct?
>
> I see 5.9.0 is released with Flink 1.3.2, so I tried that, and got the same
> problem, all I did was upload my Scala .jar to the master, updated my
> flink-conf.yaml, set my env variables, and ran it with the following
> command: HADOOP_CONF_DIR=/etc/hadoop/conf flink run -m yarn-cluster -yn 5
> -ys 2 ~/flink-consumer.jar
>
> I am beginning to think something else may be wrong with my configuration.
> Does the following look correct?
>
> In flink-conf.yaml:
> state.backend: rocksdb
> state.backend.fs.checkpointdir: s3://org/flink-project/state
> state.checkpoints.dir: s3://org/flink-project/state
>
> In the code:
> env.enableCheckpointing(getCheckpointRate,CheckpointingMode.EXACTLY_ONCE)
> env.getCheckpointConfig.enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION)
> env.getCheckpointConfig.setMinPauseBetweenCheckpoints(1000)
> env.getCheckpointConfig.setCheckpointTimeout(5000)
> env.getCheckpointConfig.setMaxConcurrentCheckpoints(1)
> env.setStateBackend(new RocksDBStateBackend("s3://org/flink-project/state",
> true))
>
> Thank you
>
>
>
> On Wed, Oct 11, 2017 at 5:19 AM, Aljoscha Krettek <aljos...@apache.org>
> wrote:
>
>> Hi Andy,
>>
>> I remember that I was testing a job with almost exactly the same setup as
>> part of the Flink 1.3.2 release testing. The command I used to start my job
>> is roughly this:
>>
>> HADOOP_CONF_DIR=/etc/hadoop/conf bin/flink run -c my.main.Class -m
>> yarn-cluster -yn 5 -ys 2 ...
>>
>> i.e. I export the proper hadoop config dir and I run a per-job YARN
>> cluster. I think I also exported the result of "hadoop classpath" as
>> HADOOP_CLASSPATH.
>>
>> Best,
>> Aljoscha
>>
>>> On 10. Oct 2017, at 16:43, Andy M. <ajm2...@gmail.com> wrote:
>>>
>>> Hello,
>>>
>>> Bowen: Unless I am missing something, it says there needs to be no setup
>>> on EMR, Each topic says: "You don’t have to configure this manually if
>> you
>>> are running Flink on EMR." S3 access from CLI works fine on my clusters.
>>>
>>> Chen: Thank you for this, I will look into this if I am unable to get
>> this
>>> running on YARN successfully.
>>>
>>> Stephan: Removing the said library causes the flink
>>> (flink-1.3.2/bin/flink) bash script to fail. The underlying Java needs
>>> this to work. I tried explicitly setting the classpath for the java call
>>> as well to point to the hadoop library jars. This is the original java
>>> command that I was trying to run:
>>>
>>> java
>>> -Dlog.file=/home/hadoop/flink-1.3.2/log/flink-hadoop-client-
>> ip-172-31-19-27.log
>>> -Dlog4j.configuration=file:/home/hadoop/flink-1.3.2/conf/
>> log4j-cli.properties
>>> -Dlogback.configurationFile=file:/home/hadoop/flink-1.3.2/
>> conf/logback.xml
>>> -classpath
>>> /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/
>> home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3.
>> 2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/
>> hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/
>> hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/hadoop/conf:
>>> org.apache.flink.client.CliFrontend run -m yarn-cluster -yn 1
>>> /home/hadoop/flink-consumer.jar
>>>
>>>
>>> This is what I changed it too(removing the shadded-hadoop2-uber jar and
>>> adding in the hadoop folder):
>>>
>>> java
>>> -Dlog.file=/home/hadoop/flink-1.3.2/log/flink-hadoop-client-
>> ip-172-31-19-27.log
>>> -Dlog4j.configuration=file:/home/hadoop/flink-1.3.2/conf/
>> log4j-cli.properties
>>> -Dlogback.configurationFile=file:/home/hadoop/flink-1.3.2/
>> conf/logback.xml
>>> -classpath
>>> /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/
>> home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/hadoop/
>> flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/hadoop/flink-
>> 1.3.2/lib/flink-dist_2.11-1.3.2.jar:/usr/lib/hadoop/lib/
>> activation-1.1.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar:
>> /usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/
>> lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/
>> apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons-
>> lang-2.6.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/
>> usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/
>> lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/
>> lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/java-
>> xmlbuilder-0.4.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.
>> jar:/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/
>> lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/
>> jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/
>> usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/
>> lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.
>> 3-1.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/
>> lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/curator-
>> client-2.7.1.jar:/usr/lib/hadoop/lib/jersey-core-1.9.
>> jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/
>> hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/curator-
>> framework-2.7.1.jar:/usr/lib/hadoop/lib/jersey-json-1.9.
>> jar:/usr/lib/hadoop/lib/slf4j-api-1.7.10.jar:/usr/lib/
>> hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/
>> curator-recipes-2.7.1.jar:/usr/lib/hadoop/lib/jersey-
>> server-1.9.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.
>> 10.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.
>> jar:/usr/lib/hadoop/lib/gson-2.2.4.jar:/usr/lib/hadoop/lib/
>> jets3t-0.9.0.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.
>> 1.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/
>> hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/jettison-1.
>> 1.jar:/usr/lib/hadoop/lib/stax-api-1.0-2.jar:/usr/lib/
>> hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/
>> hamcrest-core-1.3.jar:/usr/lib/hadoop/lib/jetty-6.1.26-
>> emr.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/commons-
>> collections-3.2.2.jar:/usr/lib/hadoop/lib/htrace-core-3.
>> 1.0-incubating.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-
>> emr.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/
>> lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/
>> httpclient-4.5.3.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/
>> usr/lib/hadoop/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop/
>> lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/
>> httpcore-4.4.4.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/
>> usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/
>> hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/
>> jsr305-3.0.0.jar:/usr/lib/hadoop/lib/commons-httpclient-
>> 3.1.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/
>> lib/hadoop/lib/junit-4.11.jar:/etc/hadoop/conf
>>> org.apache.flink.client.CliFrontend run -m yarn-cluster -yn 1
>>> /home/hadoop/flink-consumer.jar
>>>
>>> The later throws the following error:
>>>
>>> SLF4J: Class path contains multiple SLF4J bindings.
>>> SLF4J: Found binding in
>>> [jar:file:/home/hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.
>> jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>> SLF4J: Found binding in
>>> [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/
>> slf4j/impl/StaticLoggerBinder.class]
>>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>>> explanation.
>>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> org/apache/hadoop/util/VersionInfo
>>> at
>>> org.apache.flink.runtime.util.EnvironmentInformation.logEnvironmentInfo(
>> EnvironmentInformation.java:283)
>>> at org.apache.flink.client.CliFrontend.main(CliFrontend.
>> java:1124)
>>> Caused by: java.lang.ClassNotFoundException:
>>> org.apache.hadoop.util.VersionInfo
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>> ... 2 more
>>>
>>> Simply removing the .jar from the folder causes the same error.
>>>
>>> Thank you
>>>
>>>
>>>
>>>
>>> On Mon, Oct 9, 2017 at 5:46 AM, Stephan Ewen <se...@apache.org> wrote:
>>>
>>>> Hi!
>>>>
>>>> It looks like multiple Hadoop versions are in the classpath. Flink's
>> hadoop
>>>> jar and the EMR Hadoop jars.
>>>> I would simply drop Flink's own Hadoop dependency and only use the EMR
>>>> Hadoop jars.
>>>>
>>>> Delete the 'flink-shaded-hadoop2-uber' jar from Flink, and make sure
>> the
>>>> setup is such that the Hadoop lib environment variable is set. Then it
>>>> should not have conflicts any more.
>>>>
>>>>
>>>>
>>>> On Sun, Oct 8, 2017 at 12:08 AM, Chen Qin <qinnc...@gmail.com> wrote:
>>>>
>>>>> Attached my side project verified working to deploy jobmanager and
>>>>> taskmanager as stateless service(non yarn/mesos), configuration here
>>>>>
>>>>> https://github.com/chenqin/flink-jar/tree/master/config/hadoop
>>>>>
>>>>> more detail here
>>>>> https://github.com/chenqin/flink-jar/blob/master/src/
>>>>> main/java/FlinkBootstrap.java#L49
>>>>>
>>>>> On Fri, Oct 6, 2017 at 10:26 PM, Bowen Li <bowen...@offerupnow.com>
>>>> wrote:
>>>>>
>>>>>> Hi Andy,
>>>>>>
>>>>>> I believe it's because you didn't set your s3 impl correctly. Try to
>>>> set
>>>>>> your core-site.xml by following https://ci.apache.org/
>>>>>> projects/flink/flink-docs-release-1.4/ops/deployment/
>>>>>> aws.html#s3afilesystem-
>>>>>> recommended
>>>>>>
>>>>>> Bowen
>>>>>>
>>>>>> On Fri, Oct 6, 2017 at 7:59 AM, Andy M. <ajm2...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Till,
>>>>>>>
>>>>>>> Seems like everything is in line there. hadoop-common.jar ->
>>>>>>> hadoop-common-2.7.3-amzn-3.jar
>>>>>>>
>>>>>>> And when i decompiled that jar I see public void
>>>>>> addResource(Configuration
>>>>>>> conf) in org/apache/hadoop/conf/Configuration.java
>>>>>>>
>>>>>>> I agree that an incorrect version of the jar is probably being run,
>>>> is
>>>>>>> there a way to limit the classpath for the TaskManager when starting
>>>>> the
>>>>>>> job?
>>>>>>>
>>>>>>> Thank you
>>>>>>>
>>>>>>> On Fri, Oct 6, 2017 at 6:49 AM, Till Rohrmann <trohrm...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Andy,
>>>>>>>>
>>>>>>>> could you check which Hadoop version this jar
>>>>>>>> /usr/lib/hadoop/hadoop-common.jar is? Maybe also checking whether
>>>>> the
>>>>>>>> contained hadoop Configuration class has the method
>>>>>>>> Configuration.addResource(Lorg/apache/hadoop/conf/
>>>> Configuration;)V.
>>>>>>> Maybe
>>>>>>>> this jar is the culprit because it comes from a different Hadoop
>>>>>> version.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Till
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Oct 5, 2017 at 4:22 PM, Andy M. <ajm2...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Till,
>>>>>>>>>
>>>>>>>>> I believe this is what you are looking for, classpath is much
>>>>> bigger
>>>>>>> for
>>>>>>>>> the task manager. I can also post the whole log file if needed:
>>>>>>>>>
>>>>>>>>> 2017-10-05 14:17:53,038 INFO org.apache.flink.yarn.
>>>>>>>> YarnTaskManagerRunner
>>>>>>>>> - Classpath:
>>>>>>>>> flink-consumer.jar:lib/flink-dist_2.11-1.3.2.jar:lib/flink-
>>>>>>>>> python_2.11-1.3.2.jar:lib/flink-shaded-hadoop2-uber-1.3.
>>>>>>>>> 2.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:
>>>>>>>>> log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/
>>>>>>>>> etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.7.3-
>>>>>>>>> amzn-3-tests.jar:/usr/lib/hadoop/hadoop-annotations-2.7.
>>>>>>>>> 3-amzn-3.jar:/usr/lib/hadoop/hadoop-distcp.jar:/usr/lib/
>>>>>>>>> hadoop/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-
>>>>>>>>> nfs-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-streaming-2.7.3-
>>>>>>>>> amzn-3.jar:/usr/lib/hadoop/hadoop-ant-2.7.3-amzn-3.jar:/
>>>>>>>>> usr/lib/hadoop/hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/
>>>>>>>>> hadoop/hadoop-datajoin.jar:/usr/lib/hadoop/hadoop-
>>>>>>>>> streaming.jar:/usr/lib/hadoop/hadoop-common.jar:/usr/lib/
>>>>>>>>> hadoop/hadoop-ant.jar:/usr/lib/hadoop/hadoop-sls.jar:/
>>>>>>>>> usr/lib/hadoop/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/
>>>>>>>>> hadoop/hadoop-nfs.jar:/usr/lib/hadoop/hadoop-extras-2.7.
>>>>>>>>> 3-amzn-3.jar:/usr/lib/hadoop/hadoop-gridmix.jar:/usr/lib/
>>>>>>>>> hadoop/hadoop-common-2.7.3-amzn-3.jar:/usr/lib/hadoop/
>>>>>>>>> hadoop-annotations.jar:/usr/lib/hadoop/hadoop-openstack-2.
>>>>>>>>> 7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-archives-2.7.3-
>>>>>>>>> amzn-3.jar:/usr/lib/hadoop/hadoop-azure.jar:/usr/lib/
>>>>>>>>> hadoop/hadoop-extras.jar:/usr/lib/hadoop/hadoop-openstack.
>>>>>>>>> jar:/usr/lib/hadoop/hadoop-rumen.jar:/usr/lib/hadoop/
>>>>>>>>> hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-
>>>>>>>>> datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop/hadoop-
>>>>>>>>> archives.jar:/usr/lib/hadoop/hadoop-aws.jar:/usr/lib/
>>>>>>>>> hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-rumen-2.7.3-
>>>>>>>>> amzn-3.jar:/usr/lib/hadoop/hadoop-sls-2.7.3-amzn-3.jar:/
>>>>>>>>> usr/lib/hadoop/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/lib/
>>>>>>>>> hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.
>>>>>>>>> 2.2.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating.
>>>>>>>>> jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/
>>>>>>>>> lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/
>>>>>>>>> httpcore-4.4.4.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.
>>>>>>>>> 1.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.
>>>>>>>>> jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/
>>>>>>>>> lib/activation-1.1.jar:/usr/lib/hadoop/lib/jersey-server-
>>>>>>>>> 1.9.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/
>>>>>>>>> usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/
>>>>>>>>> gson-2.2.4.jar:/usr/lib/hadoop/lib/commons-digester-1.
>>>>>>>>> 8.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/
>>>>>>>>> lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/
>>>>>>>>> apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/commons-
>>>>>>>>> httpclient-3.1.jar:/usr/lib/hadoop/lib/curator-client-2.7.
>>>>>>>>> 1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/
>>>>>>>>> lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/
>>>>>>>>> commons-net-3.1.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/
>>>>>>>>> usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/
>>>>>>>>> xmlenc-0.52.jar:/usr/lib/hadoop/lib/jersey-json-1.9.
>>>>>>>>> jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/
>>>>>>>>> commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/log4j-1.2.17.
>>>>>>>>> jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/
>>>>>>>>> usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/
>>>>>>>>> jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/netty-3.6.2.
>>>>>>>>> Final.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/
>>>>>>>>> lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/api-asn1-
>>>>>>>>> api-1.0.0-M20.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-
>>>>>>>>> 1.9.13.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar:
>>>>>>>>> /usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/
>>>>>>>>> jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons-
>>>>>>>>> cli-1.2.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/
>>>>>>>>> lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/
>>>>>>>>> apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/
>>>>>>>>> commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/junit-4.
>>>>>>>>> 11.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:
>>>>>>>>> /usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/
>>>>>>>>> lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.
>>>>>>>>> 10.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/
>>>>>>>>> usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/
>>>>>>>>> lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10.
>>>>>>>>> jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/
>>>>>>>>> hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/
>>>>>>>>> curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/jackson-
>>>>>>>>> jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0.
>>>>>>>>> 4.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/
>>>>>>>>> usr/lib/hadoop-hdfs/hadoop-hdfs-nfs-2.7.3-amzn-3.jar:/
>>>>>>>>> usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3-amzn-3-tests.jar:/
>>>>>>>>> usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/
>>>>>>>>> hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs-2.7.3-
>>>>>>>>> amzn-3.jar:/usr/lib/hadoop-hdfs/lib/htrace-core-3.1.0-
>>>>>>>>> incubating.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-
>>>>>>>>> 2.5.0.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8.
>>>>>>>>> jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/
>>>>>>>>> lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/
>>>>>>>>> lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/
>>>>>>>>> commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.
>>>>>>>>> jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/
>>>>>>>>> usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/
>>>>>>>>> hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/
>>>>>>>>> netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/xml-apis-
>>>>>>>>> 1.3.04.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/
>>>>>>>>> hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/
>>>>>>>>> lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/
>>>>>>>>> hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-
>>>>>>>>> hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/
>>>>>>>>> lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/servlet-
>>>>>>>>> api-2.5.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar:
>>>>>>>>> /usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/
>>>>>>>>> hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/
>>>>>>>>> lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons-
>>>>>>>>> logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/jettison-1.1.
>>>>>>>>> jar:/usr/lib/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/lib/
>>>>>>>>> hadoop-mapreduce/hadoop-distcp.jar:/usr/lib/hadoop-
>>>>>>>>> mapreduce/htrace-core-3.1.0-incubating.jar:/usr/lib/
>>>>>>>>> hadoop-mapreduce/hadoop-auth-2.7.3-amzn-3.jar:/usr/lib/
>>>>>>>>> hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/lib/
>>>>>>>> hadoop-mapreduce/hadoop-
>>>>>>>>> mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/
>>>>>>>>> httpclient-4.5.3.jar:/usr/lib/hadoop-mapreduce/hadoop-
>>>>>>>>> mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/
>>>>>>>>> hadoop-mapreduce-client-common-2.7.3-amzn-3.jar:/usr/
>>>>>>>>> lib/hadoop-mapreduce/httpcore-4.4.4.jar:/usr/lib/hadoop-
>>>>>>>>> mapreduce/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/
>>>>>>>>> commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/
>>>>>>>>> hadoop-mapreduce-client-core-2.7.3-amzn-3.jar:/usr/lib/
>>>>>>>>> hadoop-mapreduce/jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/
>>>>>>>>> jmespath-java-1.11.160.jar:/usr/lib/hadoop-mapreduce/
>>>>>>>>> hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop-
>>>>>>>>> mapreduce/activation-1.1.jar:/usr/lib/hadoop-mapreduce/
>>>>>>>>> hadoop-mapreduce-client-jobclient-2.7.3-amzn-3-tests.
>>>>>>>>> jar:/usr/lib/hadoop-mapreduce/jersey-server-1.9.jar:/usr/
>>>>>>>>> lib/hadoop-mapreduce/hadoop-streaming-2.7.3-amzn-3.jar:/
>>>>>>>>> usr/lib/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/
>>>>>>>>> usr/lib/hadoop-mapreduce/hadoop-ant-2.7.3-amzn-3.jar:/
>>>>>>>>> usr/lib/hadoop-mapreduce/guava-11.0.2.jar:/usr/lib/
>>>>>>>>> hadoop-mapreduce/gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/
>>>>>>>>> hadoop-distcp-2.7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/
>>>>>>>>> hadoop-mapreduce-client-app-2.7.3-amzn-3.jar:/usr/lib/
>>>>>>>>> hadoop-mapreduce/hadoop-mapreduce-examples-2.7.3-amzn-
>>>>>>>>> 3.jar:/usr/lib/hadoop-mapreduce/commons-digester-1.
>>>>>>>> 8.jar:/usr/lib/hadoop-
>>>>>>>>> mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3-amzn-
>>>>>>>>> 3.jar:/usr/lib/hadoop-mapreduce/jackson-xc-1.9.13.
>>>>>>>>> jar:/usr/lib/hadoop-mapreduce/paranamer-2.3.jar:/usr/lib/
>>>>>>>>> hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/lib/
>>>>>>>>> hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/lib/
>>>>>>>>> hadoop-mapreduce/curator-client-2.7.1.jar:/usr/lib/
>>>>>>>>> hadoop-mapreduce/hadoop-datajoin.jar:/usr/lib/hadoop-
>>>>>>>>> mapreduce/hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/
>>>>>>>>> commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/commons-
>>>>>>>>> net-3.1.jar:/usr/lib/hadoop-mapreduce/jets3t-0.9.0.jar:/
>>>>>>>>> usr/lib/hadoop-mapreduce/commons-io-2.4.jar:/usr/lib/
>>>>>>>>> hadoop-mapreduce/hadoop-ant.jar:/usr/lib/hadoop-mapreduce/
>>>>>>>>> xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/jersey-json-
>>>>>>>>> 1.9.jar:/usr/lib/hadoop-mapreduce/hadoop-sls.jar:/usr/
>>>>>>>>> lib/hadoop-mapreduce/avro-1.7.4.jar:/usr/lib/hadoop-
>>>>>>>>> mapreduce/aws-java-sdk-core-1.11.160.jar:/usr/lib/hadoop-
>>>>>>>>> mapreduce/aws-java-sdk-s3-1.11.160.jar:/usr/lib/hadoop-
>>>>>>>>> mapreduce/jackson-dataformat-cbor-2.6.6.jar:/usr/lib/
>>>>>>>>> hadoop-mapreduce/hadoop-azure-2.7.3-amzn-3.jar:/usr/lib/
>>>>>>>>> hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/lib/
>>>>>>>>> hadoop-mapreduce/ion-java-1.0.2.jar:/usr/lib/hadoop-
>>>>>>>>> mapreduce/azure-storage-2.0.0.jar:/usr/lib/hadoop-mapreduce/
>>>>>>>>> hadoop-mapreduce-client-shuffle-2.7.3-amzn-3.jar:/usr/
>>>>>>>>> lib/hadoop-mapreduce/log4j-1.2.17.jar:/usr/lib/hadoop-
>>>>>>>>> mapreduce/commons-compress-1.4.1.jar:/usr/lib/hadoop-
>>>>>>>>> mapreduce/hadoop-extras-2.7.3-amzn-3.jar:/usr/lib/hadoop-
>>>>>>>>> mapreduce/hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/jsch-
>>>>>>>>> 0.1.42.jar:/usr/lib/hadoop-mapreduce/joda-time-2.8.1.jar:
>>>>>>>>> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-hs-2.
>>>>>>>>> 7.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/jaxb-impl-2.
>>>>>>>>> 2.3-1.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-
>>>>>>>>> client-hs.jar:/usr/lib/hadoop-mapreduce/netty-3.6.2.Final.
>>>>>>>>> jar:/usr/lib/hadoop-mapreduce/mockito-all-1.8.5.jar:/usr/
>>>>>>>>> lib/hadoop-mapreduce/hadoop-openstack-2.7.3-amzn-3.jar:/
>>>>>>>>> usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-
>>>>>>>>> common.jar:/usr/lib/hadoop-mapreduce/hadoop-archives-2.7.
>>>>>>>>> 3-amzn-3.jar:/usr/lib/hadoop-mapreduce/hadoop-azure.jar:/
>>>>>>>>> usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-
>>>>>>>>> jobclient.jar:/usr/lib/hadoop-mapreduce/hadoop-extras.jar:/
>>>>>>>>> usr/lib/hadoop-mapreduce/asm-3.2.jar:/usr/lib/hadoop-
>>>>>>>>> mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop-
>>>>>>>>> mapreduce/hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/
>>>>>>>>> hadoop-mapreduce-client-jobclient-2.7.3-amzn-3.jar:/
>>>>>>>>> usr/lib/hadoop-mapreduce/hadoop-rumen.jar:/usr/lib/
>>>>>>>>> hadoop-mapreduce/aws-java-sdk-kms-1.11.160.jar:/usr/lib/
>>>>>>>>> hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/
>>>>>>>>> lib/hadoop-mapreduce/hadoop-aws-2.7.3-amzn-3.jar:/usr/lib/
>>>>>>>>> hadoop-mapreduce/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-
>>>>>>>>> mapreduce/hadoop-datajoin-2.7.3-amzn-3.jar:/usr/lib/hadoop-
>>>>>>>>> mapreduce/xz-1.0.jar:/usr/lib/hadoop-mapreduce/hadoop-
>>>>>>>>> archives.jar:/usr/lib/hadoop-mapreduce/jackson-core-asl-1.
>>>>>>>>> 9.13.jar:/usr/lib/hadoop-mapreduce/commons-lang3-3.3.2.
>>>>>>>>> jar:/usr/lib/hadoop-mapreduce/jackson-core-2.6.6.jar:/usr/
>>>>>>>>> lib/hadoop-mapreduce/commons-cli-1.2.jar:/usr/lib/hadoop-
>>>>>>>>> mapreduce/servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/
>>>>>>>>> jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/apacheds-
>>>>>>>>> kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/
>>>>>>>>> hadoop-aws.jar:/usr/lib/hadoop-mapreduce/hadoop-auth.
>>>>>>>>> jar:/usr/lib/hadoop-mapreduce/hadoop-rumen-2.7.3-amzn-3.jar:
>>>>>>>>> /usr/lib/hadoop-mapreduce/commons-collections-3.2.2.jar:
>>>>>>>>> /usr/lib/hadoop-mapreduce/hadoop-sls-2.7.3-amzn-3.jar:/
>>>>>>>>> usr/lib/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/
>>>>>>>>> lib/hadoop-mapreduce/jackson-annotations-2.6.6.jar:/usr/
>>>>>>>>> lib/hadoop-mapreduce/junit-4.11.jar:/usr/lib/hadoop-
>>>>>> mapreduce/commons-
>>>>>>>>> configuration-1.6.jar:/usr/lib/hadoop-mapreduce/api-util-
>>>>>>>>> 1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/stax-api-1.0-2.jar:/
>>>>>>>>> usr/lib/hadoop-mapreduce/curator-recipes-2.7.1.jar:/
>>>>>>>>> usr/lib/hadoop-mapreduce/hamcrest-core-1.3.jar:/usr/
>>>>>>>>> lib/hadoop-mapreduce/jackson-databind-2.6.6.jar:/usr/lib/
>>>>>>>>> hadoop-mapreduce/jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/
>>>>>>>>> hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/
>>>>>>>>> zookeeper-3.4.10.jar:/usr/lib/hadoop-mapreduce/commons-lang-
>>>>>>>>> 2.6.jar:/usr/lib/hadoop-mapreduce/jersey-core-1.9.jar:
>>>>>>>>> /usr/lib/hadoop-mapreduce/curator-framework-2.7.1.jar:/
>>>>>>>>> usr/lib/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/
>>>>>>>>> lib/hadoop-mapreduce/hadoop-gridmix-2.7.3-amzn-3.jar:/usr/
>>>>>>>>> lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/
>>>>>>>>> lib/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/lib/
>>>>>>>>> hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/lib/
>>>>>>>>> hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/
>>>>>>>>> hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/
>>>>>>>> lib/hadoop-mapreduce/lib/
>>>>>>>>> snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/
>>>>>>>>> jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/
>>>>>>>>> paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/jersey-
>>>>>>>>> guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4.
>>>>>>>>> jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/
>>>>>>>>> hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-
>>>>>>>>> mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/
>>>>>>>>> hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop-
>>>>>>>>> mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-
>>>>>>>>> mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/
>>>>>>>>> lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/jackson-
>>>>>>>>> mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/
>>>>>>>>> guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/xz-1.
>>>>>>>>> 0.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/
>>>>>>>>> usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/
>>>>>>>>> usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/
>>>>>>>>> hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop-
>>>>>>>>> mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/
>>>>>>>>> hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/hadoop-
>>>>>> yarn-server-
>>>>>>>>> sharedcachemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-
>>>>>>>>> server-web-proxy-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/
>>>>>>>>> hadoop-yarn-common-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/
>>>>>>>>> hadoop-yarn-server-tests-2.7.3-amzn-3.jar:/usr/lib/hadoop-
>>>>>>>>> yarn/hadoop-yarn-applications-distributedshell.jar:/usr/lib/
>>>>>>>>> hadoop-yarn/hadoop-yarn-applications-distributedshell-
>>>>>>>>> 2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-
>>>>>>>>> server-resourcemanager.jar:/usr/lib/hadoop-yarn/hadoop-
>>>>> yarn-server-
>>>>>>>>> sharedcachemanager-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/
>>>>>>>>> hadoop-yarn-server-applicationhistoryservice.jar:
>>>>>>>>> /usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/
>>>>>>>>> lib/hadoop-yarn/hadoop-yarn-server-nodemanager-2.7.3-amzn-
>>>>>>>>> 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-client-2.7.3-amzn-
>>>>>>>>> 3.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-
>>>>>>>>> applicationhistoryservice-2.7.3-amzn-3.jar:/usr/lib/hadoop-
>>>>>>>>> yarn/hadoop-yarn-server-common-2.7.3-amzn-3.jar:/usr/
>>>>>>>>> lib/hadoop-yarn/hadoop-yarn-registry.jar:/usr/lib/hadoop-
>>>>>>>>> yarn/hadoop-yarn-api-2.7.3-amzn-3.jar:/usr/lib/hadoop-
>>>>>>>>> yarn/hadoop-yarn-server-resourcemanager-2.7.3-amzn-3.
>>>>>>>>> jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-
>>>>>>>>> unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/hadoop-
>>>>>>>>> yarn-registry-2.7.3-amzn-3.jar:/usr/lib/hadoop-yarn/
>>>>>>>>> hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/hadoop-yarn-
>>>>>>>>> common.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-web-
>>>>>>>>> proxy.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-
>>>>>>>>> nodemanager.jar:/usr/lib/hadoop-yarn/hadoop-yarn-api.
>>>>>>>>> jar:/usr/lib/hadoop-yarn/hadoop-yarn-applications-
>>>>>>>>> unmanaged-am-launcher-2.7.3-amzn-3.jar:/usr/lib/hadoop-
>>>>>>>>> yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/jaxb-
>>>>>>>>> api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0.
>>>>>>>>> jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/
>>>>>>>>> lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/
>>>>>>>>> hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/
>>>>>>>>> lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1.
>>>>>>>>> 9.13.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/
>>>>>>>>> usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/
>>>>>>>>> hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/
>>>>>>>>> lib/jersey-json-1.9.jar:/usr/lib/hadoop-yarn/lib/jersey-
>>>>>>>>> client-1.9.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17.
>>>>>>>>> jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/
>>>>>>>>> usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop-
>>>>>>>>> yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/
>>>>>>>>> netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/
>>>>>>>>> aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/
>>>>>>>>> usr/lib/hadoop-yarn/lib/zookeeper-3.4.10-tests.jar:/
>>>>>>>>> usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar:
>>>>>>>>> /usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/
>>>>>>>>> lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/
>>>>>>>>> hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/guice-
>>>>>>>>> 3.0.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13.
>>>>>>>>> jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/
>>>>>>>>> hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/
>>>>>>>>> lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/commons-
>>>>>>>>> collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/stax-api-
>>>>>>>>> 1.0-2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/
>>>>>>>>> usr/lib/hadoop-yarn/lib/zookeeper-3.4.10.jar:/usr/lib/
>>>>>>>>> hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/
>>>>>>>>> lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/jackson-
>>>>>>>>> jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/commons-
>>>>>>>>> logging-1.1.3.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar:
>>>>>>>>> /usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/
>>>>>>>>> share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/
>>>>>>>>> jmespath-java-1.11.129.jar:/usr/share/aws/emr/emrfs/lib/
>>>>>>>>> bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/jcl-
>>>>>>>>> over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/ion-
>>>>>>>>> java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7.
>>>>>>>>> 21.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/
>>>>>>>>> usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/
>>>>>>>>> share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/
>>>>>>>>> share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.18.0.jar:/
>>>>>>>>> usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/
>>>>>>>>> lib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/
>>>>>>>>> share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/
>>>>>>>>> aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/
>>>>>>>>> aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.0.0.jar:/
>>>>>>>>> usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/
>>>>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11.
>>>>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api-
>>>>>>>>> gateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-
>>>>>>>>> java-sdk-kinesis-1.11.160.jar:/usr/share/aws/aws-java-sdk/
>>>>>>> aws-java-sdk-
>>>>>>>>> elasticloadbalancingv2-1.11.160.jar:/usr/share/aws/aws-
>>>>>>>>> java-sdk/aws-java-sdk-rekognition-1.11.160.jar:/usr/
>>>>>>>>> share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.160.
>>>>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
>>>>>>>>> elasticbeanstalk-1.11.160.jar:/usr/share/aws/aws-java-sdk/
>>>>>>>>> aws-java-sdk-cloudhsm-1.11.160.jar:/usr/share/aws/aws-
>>>>>>>>> java-sdk/aws-java-sdk-iam-1.11.160.jar:/usr/share/aws/aws-
>>>>>>>>> java-sdk/aws-java-sdk-cognitoidp-1.11.160.jar:/usr/
>>>>>>>>> share/aws/aws-java-sdk/aws-java-sdk-
>>>> mechanicalturkrequester-1.11.
>>>>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
>>>>>>>>> marketplacecommerceanalytics-1.11.160.jar:/usr/share/aws/
>>>>>>>>> aws-java-sdk/aws-java-sdk-snowball-1.11.160.jar:/usr/
>>>>>>>>> share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.160.jar:/
>>>>>>>>> usr/share/aws/aws-java-sdk/jmespath-java-1.11.160.jar:/
>>>>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11.
>>>>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1.
>>>>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
>>>>>>>>> simpleworkflow-1.11.160.jar:/usr/share/aws/aws-java-sdk/
>>>>>>>>> aws-java-sdk-machinelearning-1.11.160.jar:/usr/share/aws/
>>>>>>>>> aws-java-sdk/aws-java-sdk-pinpoint-1.11.160.jar:/usr/
>>>>>>>>> share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.160.
>>>>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.160.
>>>>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11.
>>>>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
>>>>>>>>> greengrass-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-
>>>>>>>>> java-sdk-ecs-1.11.160.jar:/usr/share/aws/aws-java-sdk/
>>>>>>>>> aws-java-sdk-xray-1.11.160.jar:/usr/share/aws/aws-java-
>>>>>>>>> sdk/aws-java-sdk-cloudtrail-1.11.160.jar:/usr/share/aws/aws-
>>>>>>>>> java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.160.
>>>>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
>>>>>>>>> elasticloadbalancing-1.11.160.jar:/usr/share/aws/aws-java-
>>>>>>>>> sdk/aws-java-sdk-codepipeline-1.11.160.jar:/usr/share/aws/
>>>>>>>>> aws-java-sdk/aws-java-sdk-cloudwatch-1.11.160.jar:/usr/
>>>>>>>>> share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.160.jar:/
>>>>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.160.
>>>>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
>>>>>>>> stepfunctions-1.11.160.jar:/
>>>>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11.
>>>>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1.
>>>>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1.
>>>>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
>>>>>>>>> datapipeline-1.11.160.jar:/usr/share/aws/aws-java-sdk/
>>>>>>>>> aws-java-sdk-s3-1.11.160.jar:/usr/share/aws/aws-java-sdk/
>>>>>>>>> aws-java-sdk-dax-1.11.160.jar:/usr/share/aws/aws-java-sdk/
>>>>>>>>> aws-java-sdk-opsworks-1.11.160.jar:/usr/share/aws/aws-
>>>>>>>>> java-sdk/aws-java-sdk-servicecatalog-1.11.160.jar:/
>>>>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11.
>>>>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1.
>>>>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
>>>>>>>>> directory-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-
>>>>>>>>> java-sdk-opsworkscm-1.11.160.jar:/usr/share/aws/aws-java-
>>>>>>>> sdk/aws-java-sdk-
>>>>>>>>> cloudformation-1.11.160.jar:/usr/share/aws/aws-java-sdk/
>>>>>>>>> aws-java-sdk-cloudfront-1.11.160.jar:/usr/share/aws/aws-
>>>>>>>>> java-sdk/aws-java-sdk-budgets-1.11.160.jar:/usr/share/aws/
>>>>>>>>> aws-java-sdk/aws-java-sdk-clouddirectory-1.11.160.jar:/
>>>>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1.
>>>>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1.
>>>>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
>>>>>>>>> marketplaceentitlement-1.11.160.jar:/usr/share/aws/aws-
>>>>>>>>> java-sdk/aws-java-sdk-dynamodb-1.11.160.jar:/usr/
>>>>>>>>> share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.160.
>>>>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
>>>>>>> elastictranscoder-1.11.160.
>>>>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
>>>>>>>> organizations-1.11.160.jar:/
>>>>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11.
>>>>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1.
>>>>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
>>>>>>>>> servermigration-1.11.160.jar:/usr/share/aws/aws-java-sdk/
>>>>>>>>> aws-java-sdk-events-1.11.160.jar:/usr/share/aws/aws-java-
>>>>>>>> sdk/aws-java-sdk-
>>>>>>>>> applicationautoscaling-1.11.160.jar:/usr/share/aws/aws-
>>>>>>>>> java-sdk/aws-java-sdk-health-1.11.160.jar:/usr/share/aws/
>>>>>>>>> aws-java-sdk/aws-java-sdk-kms-1.11.160.jar:/usr/share/aws/
>>>>>>>>> aws-java-sdk/aws-java-sdk-logs-1.11.160.jar:/usr/share/
>>>>>>>>> aws/aws-java-sdk/aws-java-sdk-codestar-1.11.160.jar:/usr/
>>>>>>>>> share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.160.jar:
>>>>>>>>> /usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11.
>>>>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
>>>>>>>>> marketplacemeteringservice-1.11.160.jar:/usr/share/aws/aws-
>>>>>>>>> java-sdk/aws-java-sdk-sns-1.11.160.jar:/usr/share/aws/aws-
>>>>>>>>> java-sdk/aws-java-sdk-batch-1.11.160.jar:/usr/share/aws/aws-
>>>>>>>>> java-sdk/aws-java-sdk-waf-1.11.160.jar:/usr/share/aws/aws-
>>>>>>>>> java-sdk/aws-java-sdk-simpledb-1.11.160.jar:/usr/
>>>>>>>>> share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.160.
>>>>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.160.
>>>>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11.
>>>>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
>>>>>>>>> devicefarm-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-
>>>>>>>>> java-sdk-cognitoidentity-1.11.160.jar:/usr/share/aws/aws-
>>>>>>>>> java-sdk/aws-java-sdk-lexmodelbuilding-1.11.160.jar:
>>>>>>>>> /usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1.
>>>>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
>>>>>>>>> elasticache-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-
>>>>>>>>> java-sdk-costandusagereport-1.11.160.jar:/usr/share/aws/aws-
>>>>>>>>> java-sdk/aws-java-sdk-discovery-1.11.160.jar:/usr/
>>>>>>>>> share/aws/aws-java-sdk/aws-java-sdk-
>>>> resourcegroupstaggingapi-1.11.
>>>>>>>>> 160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1.
>>>>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda-
>>>>>>>>> 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
>>>>>>>>> workdocs-1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-
>>>>>>>>> java-sdk-code-generator-1.11.160.jar:/usr/share/aws/aws-
>>>>>>>>> java-sdk/aws-java-sdk-cognitosync-1.11.160.jar:/usr/
>>>>>>>>> share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.160.jar:/
>>>>>>>>> usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.160.jar:
>>>>>>>>> /usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.160.
>>>>>>>>> jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1.
>>>>>>>>> 11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
>>>>>>>>> storagegateway-1.11.160.jar:/usr/share/aws/aws-java-sdk/
>>>>>>>>> aws-java-sdk-lightsail-1.11.160.jar:/usr/share/aws/aws-
>>>>>>>>> java-sdk/aws-java-sdk-acm-1.11.160.jar:/usr/share/aws/aws-
>>>>>>>>> java-sdk/aws-java-sdk-1.11.160.jar:/usr/share/aws/aws-
>>>>>>>>> java-sdk/aws-java-sdk-glacier-1.11.160.jar:/usr/share/aws/
>>>>>>>>> aws-java-sdk/aws-java-sdk-ecr-1.11.160.jar:/usr/share/aws/
>>>>>>>>> aws-java-sdk/aws-java-sdk-support-1.11.160.jar:/usr/
>>>>>>>>> share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin-
>>>>>>>>> 1.11.160.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-
>>>>>>>>> elasticsearch-1.11.160.jar:/usr/share/aws/aws-java-sdk/
>>>>>>>>> aws-java-sdk-iot-1.11.160.jar
>>>>>>>>>
>>>>>>>>> Thank you
>>>>>>>>>
>>>>>>>>> On Thu, Oct 5, 2017 at 5:25 AM, Till Rohrmann <
>>>>> trohrm...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Andy,
>>>>>>>>>>
>>>>>>>>>> the CliFrontend is not executed via Yarn, thus, it is not
>>>>> affected
>>>>>> by
>>>>>>>>>> dependencies which are added due to the underlying Yarn
>>>> cluster.
>>>>>>>>> Therefore,
>>>>>>>>>> it would be helpful to look at the TaskManager logs. Either you
>>>>>> have
>>>>>>>>>> enabled log aggregation on your Yarn cluster, then you can
>>>> obtain
>>>>>> the
>>>>>>>>> logs
>>>>>>>>>> via `yarn logs -applicationId <application ID>` or you have to
>>>>>>> retrieve
>>>>>>>>>> them from the machines where they were running (either by going
>>>>>>>> directly
>>>>>>>>>> there or via the Yarn web interface).
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Till
>>>>>>>>>>
>>>>>>>>>> On Wed, Oct 4, 2017 at 4:27 PM, Andy M. <ajm2...@gmail.com>
>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Till,
>>>>>>>>>>>
>>>>>>>>>>> That is actually the classpath used by the flink bash
>>>>> script(that
>>>>>>>>>> launches
>>>>>>>>>>> the jar using the java command). I changed the execute to an
>>>>>> echo,
>>>>>>>> and
>>>>>>>>>>> grabbed that for the CLI arguments.
>>>>>>>>>>>
>>>>>>>>>>> I believe this is the class path from the log file(although
>>>> it
>>>>>>> might
>>>>>>>>> not
>>>>>>>>>> be
>>>>>>>>>>> the taskmanager log, is that any different from what would be
>>>>> in
>>>>>> my
>>>>>>>>>>> flink-1.3.2/log folder?):
>>>>>>>>>>>
>>>>>>>>>>> 2017-10-02 20:03:26,450 INFO org.apache.flink.client.
>>>>>> CliFrontend
>>>>>>>>>>> - Classpath:
>>>>>>>>>>> /home/hadoop/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/
>>>>>>>>>>> home/hadoop/flink-1.3.2/lib/flink-shaded-hadoop2-uber-1.3.
>>>>>>>>>>> 2.jar:/home/hadoop/flink-1.3.2/lib/log4j-1.2.17.jar:/home/
>>>>>>>>>>> hadoop/flink-1.3.2/lib/slf4j-log4j12-1.7.7.jar:/home/
>>>>>>>>>>> hadoop/flink-1.3.2/lib/flink-dist_2.11-1.3.2.jar::/etc/
>>>>>>> hadoop/conf:
>>>>>>>>>>>
>>>>>>>>>>> If that doesn't seem right, and you can point me in the right
>>>>>>>> direction
>>>>>>>>>> as
>>>>>>>>>>> to where the TaskManager logs would be, I would be happy to
>>>>> grab
>>>>>>> the
>>>>>>>>>>> information your looking for.
>>>>>>>>>>>
>>>>>>>>>>> Thank you
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Oct 4, 2017 at 3:27 AM, Till Rohrmann <
>>>>>>> trohrm...@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Andy,
>>>>>>>>>>>>
>>>>>>>>>>>> this looks to me indeed like a dependency problem. I assume
>>>>>> that
>>>>>>>> EMR
>>>>>>>>> or
>>>>>>>>>>>> something else is pulling in an incompatible version of
>>>>> Hadoop.
>>>>>>>>>>>>
>>>>>>>>>>>> The classpath you've posted, is this the one logged in the
>>>>> log
>>>>>>>> files
>>>>>>>>>>>> (TaskManager log) or did you compile it yourself? In the
>>>>> latter
>>>>>>>> case,
>>>>>>>>>> it
>>>>>>>>>>>> would also be helpful to get access to the TaskManager
>>>> logs.
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Till
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Oct 2, 2017 at 10:20 PM, Andy M. <
>>>> ajm2...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Fabian,
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1) I have looked at the linked docs, and from what I can
>>>>> tell
>>>>>>> no
>>>>>>>>>> setup
>>>>>>>>>>>>> should really need to be done to get Flink working(Other
>>>>> than
>>>>>>>>>>> downloading
>>>>>>>>>>>>> the correct binaries, which I believe I did)
>>>>>>>>>>>>> 2) I have downloaded the Flink 1.3.2
>>>>>> binaries(flink-1.3.2-bin-
>>>>>>>>>>>>> hadoop27-scala_2.11.tgz
>>>>>>>>>>>>> <http://apache.claz.org/flink/
>>>> flink-1.3.2/flink-1.3.2-bin-
>>>>>>>>>>>>> hadoop27-scala_2.11.tgz>)
>>>>>>>>>>>>> This is for hadoop 2.7.X, which matches EMR 5.8.0.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I appreciate any help or guidance you can provide me in
>>>>>> fixing
>>>>>>> my
>>>>>>>>>>>> problems,
>>>>>>>>>>>>> please let me know if there is anything else I can
>>>> provide
>>>>>> you.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Oct 2, 2017 at 4:12 PM, Fabian Hueske <
>>>>>>> fhue...@gmail.com
>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Andy,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm not an AWS expert, so I'll just check on some
>>>> common
>>>>>>>> issues.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I guess you already had a look at the Flink docs for
>>>>>> AWS/EMR
>>>>>>>> but
>>>>>>>>>> I'll
>>>>>>>>>>>>> post
>>>>>>>>>>>>>> the link just be to sure [1].
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Since you are using Flink 1.3.2 (EMR 5.8.0 comes with
>>>>> Flink
>>>>>>>>> 1.3.1)
>>>>>>>>>>> did
>>>>>>>>>>>>> you
>>>>>>>>>>>>>> built Flink yourself or did you download the binaries?
>>>>>>>>>>>>>> Does the Hadoop version of the Flink build match the
>>>>> Hadoop
>>>>>>>>> version
>>>>>>>>>>> of
>>>>>>>>>>>>> EMR
>>>>>>>>>>>>>> 5.8.0, i.e., Hadoop 2.7.x?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best, Fabian
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>> https://ci.apache.org/projects/flink/flink-docs-
>>>>>>>>>>>>> release-1.3/setup/aws.html
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2017-10-02 21:51 GMT+02:00 Andy M. <ajm2...@gmail.com
>>>>> :
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Fabian,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sorry, I just realized I forgot to include that part.
>>>>>> The
>>>>>>>>> error
>>>>>>>>>>>>> returned
>>>>>>>>>>>>>>> is:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.lang.NoSuchMethodError:
>>>>>>>>>>>>>>> org.apache.hadoop.conf.Configuration.addResource(
>>>>>>>>>>>>>> Lorg/apache/hadoop/conf/
>>>>>>>>>>>>>>> Configuration;)V
>>>>>>>>>>>>>>> at com.amazon.ws.emr.hadoop.fs.
>>>>>>> EmrFileSystem.initialize(
>>>>>>>>>>>>>>> EmrFileSystem.java:93)
>>>>>>>>>>>>>>> at org.apache.flink.runtime.fs.
>>>>>> hdfs.HadoopFileSystem.
>>>>>>>>>>>>>>> initialize(HadoopFileSystem.java:328)
>>>>>>>>>>>>>>> at org.apache.flink.core.fs.FileSystem.
>>>>>>>>>> getUnguardedFileSystem(
>>>>>>>>>>>>>>> FileSystem.java:350)
>>>>>>>>>>>>>>> at org.apache.flink.core.fs.
>>>>>> FileSystem.get(FileSystem.
>>>>>>>>>>> java:389)
>>>>>>>>>>>>>>> at org.apache.flink.core.fs.Path.
>>>>>>>>>> getFileSystem(Path.java:293)
>>>>>>>>>>>>>>> at org.apache.flink.runtime.state.filesystem.
>>>>>>>>>>>>>>> FsCheckpointStreamFactory.<init>(
>>>>>>> FsCheckpointStreamFactory.
>>>>>>>>>>> java:99)
>>>>>>>>>>>>>>> at org.apache.flink.runtime.state.filesystem.
>>>>>>>>> FsStateBackend.
>>>>>>>>>>>>>>> createStreamFactory(FsStateBackend.java:282)
>>>>>>>>>>>>>>> at org.apache.flink.contrib.streaming.state.
>>>>>>>>>>> RocksDBStateBackend.
>>>>>>>>>>>>>>> createStreamFactory(RocksDBStateBackend.java:273
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I believe it has something to do with the classpath,
>>>>> but
>>>>>> I
>>>>>>> am
>>>>>>>>>>> unsure
>>>>>>>>>>>>> why
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>> how to fix it. The classpath being used during the
>>>>>>> execution
>>>>>>>>> is:
>>>>>>>>>>>>>>> /home/hadoop/flink-1.3.2/lib/
>>>>>> flink-python_2.11-1.3.2.jar:/
>>>>>>>>>>>>>>> home/hadoop/flink-1.3.2/
>>>>> lib/flink-shaded-hadoop2-
>>>>>>>>>>>>>>> uber-1.3.2.jar:/home/hadoop/
>>>>>>> flink-1.3.2/lib/log4j-1.2.
>>>>>>>>>>>>>>> 17.jar:/home/hadoop/flink-1.
>>>>>>> 3.2/lib/slf4j-log4j12-1.7.7.
>>>>>>>>>>>>>>> jar:/home/hadoop/flink-1.
>>>>>>> 3.2/lib/flink-dist_2.11-1.3.
>>>>>>>>>>>>>>> 2.jar::/etc/hadoop/conf:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I decompiled flink-shaded-hadoop2-uber-1.3.2.jar
>>>>> and
>>>>>>> it
>>>>>>>>>> seems
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> addResource function does seem to be there.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thank you
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Oct 2, 2017 at 3:43 PM, Fabian Hueske <
>>>>>>>>> fhue...@gmail.com
>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Andy,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> can you describe in more detail what exactly isn't
>>>>>>> working?
>>>>>>>>>>>>>>>> Do you see error messages in the log files or on
>>>> the
>>>>>>>> console?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks, Fabian
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2017-10-02 15:52 GMT+02:00 Andy M. <
>>>>> ajm2...@gmail.com
>>>>>>> :
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I am about to deploy my first Flink projects to
>>>>>>>>> production,
>>>>>>>>>>> but
>>>>>>>>>>>> I
>>>>>>>>>>>>> am
>>>>>>>>>>>>>>>>> running into a very big hurdle. I am unable to
>>>>>> launch
>>>>>>> my
>>>>>>>>>>> project
>>>>>>>>>>>>> so
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>> write to an S3 bucket. My project is running on
>>>> an
>>>>>> EMR
>>>>>>>>>>> cluster,
>>>>>>>>>>>>>> where
>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>> have installed Flink 1.3.2. I am using Yarn to
>>>>>> launch
>>>>>>>> the
>>>>>>>>>>>>>> application,
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> it seems to run fine unless I am trying to enable
>>>>>> check
>>>>>>>>>>>>>> pointing(with a
>>>>>>>>>>>>>>>> S3
>>>>>>>>>>>>>>>>> target). I am looking to use RocksDB as my
>>>>>>>> check-pointing
>>>>>>>>>>>> backend.
>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>> asked a few places, and I am still unable to
>>>> find a
>>>>>>>>> solution
>>>>>>>>>> to
>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> problem. Here are my steps for creating a
>>>> cluster,
>>>>>> and
>>>>>>>>>>> launching
>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>> application, perhaps I am missing a step. I'd be
>>>>>> happy
>>>>>>>> to
>>>>>>>>>>>> provide
>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>> additional information if needed.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> AWS Portal:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 1) EMR -> Create Cluster
>>>>>>>>>>>>>>>>> 2) Advanced Options
>>>>>>>>>>>>>>>>> 3) Release = emr-5.8.0
>>>>>>>>>>>>>>>>> 4) Only select Hadoop 2.7.3
>>>>>>>>>>>>>>>>> 5) Next -> Next -> Next -> Create Cluster ( I
>>>>> do
>>>>>>> fill
>>>>>>>>> out
>>>>>>>>>>>>>>>>> names/keys/etc)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Once the cluster is up I ssh into the Master and
>>>> do
>>>>>> the
>>>>>>>>>>>> following:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 1 wget
>>>>>>>>>>>>>>>>> http://apache.claz.org/flink/
>>>>>>>> flink-1.3.2/flink-1.3.2-bin-
>>>>>>>>>>>>>>>>> hadoop27-scala_2.11.tgz
>>>>>>>>>>>>>>>>> 2 tar -xzf flink-1.3.2-bin-hadoop27-
>>>>>>> scala_2.11.tgz
>>>>>>>>>>>>>>>>> 3 cd flink-1.3.2
>>>>>>>>>>>>>>>>> 4 ./bin/yarn-session.sh -n 2 -tm 5120 -s 4
>>>> -d
>>>>>>>>>>>>>>>>> 5 Change conf/flink-conf.yaml
>>>>>>>>>>>>>>>>> 6 ./bin/flink run -m yarn-cluster -yn 1
>>>>>>>>>>> ~/flink-consumer.jar
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> My conf/flink-conf.yaml I add the following
>>>> fields:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> state.backend: rocksdb
>>>>>>>>>>>>>>>>> state.backend.fs.checkpointdir:
>>>>>>> s3:/bucket/location
>>>>>>>>>>>>>>>>> state.checkpoints.dir: s3:/bucket/location
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> My program's checkpointing setup:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> env.enableCheckpointing(getCheckpointRate,
>>>>>>>>>>>>> CheckpointingMode.EXACTLY_
>>>>>>>>>>>>>>>> ONCE)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> env.getCheckpointConfig.
>>>>>> enableExternalizedCheckpoints(
>>>>>>>>>>>>>>>>> ExternalizedCheckpointCleanup.
>>>>>> RETAIN_ON_CANCELLATION)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> env.getCheckpointConfig.
>>>>>> setMinPauseBetweenCheckpoints(
>>>>>>>>>>>>>>>>> getCheckpointMinPause)
>>>>>>>>>>>>>>>>> env.getCheckpointConfig.
>>>> setCheckpointTimeout(
>>>>>>>>>>>>>> getCheckpointTimeout)
>>>>>>>>>>>>>>>>> env.getCheckpointConfig.
>>>>>>>> setMaxConcurrentCheckpoints(1)
>>>>>>>>>>>>>>>>> env.setStateBackend(new
>>>>>> RocksDBStateBackend("s3://
>>>>>>>>>>>>>>> bucket/location",
>>>>>>>>>>>>>>>>> true))
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>
>>
