Hi, did you check if the "org.apache.hadoop.fs.s3native.NativeS3FileSystem" class is in the flink-dist.jar in the lib/ folder?
On Sun, Mar 20, 2016 at 10:19 AM, Ashutosh Kumar <ashutosh.disc...@gmail.com > wrote: > I have setup a 3 node YARN based cluster on EC2. I am running flink in > cluster mode. I added these lines in core-site.xml > > <configuration> > > <property> > <name>fs.s3n.awsAccessKeyId</name> > <value>accesskey</value> > </property> > > <property> > <name>fs.s3n.awsSecretAccessKey</name> > <value>secret key</value> > </property> > <property> > <name>fs.s3n.impl</name> > <value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value> > </property> > </configuration> > > Also I added this line in conf/flink-conf.yaml > fs.hdfs.hadoopconf: /usr/local/hadoop/etc/hadoop > > > However I am getting class not found error while accessing s3 through s3n. > I am using flink 1.0.0. > > Caused by: org.apache.flink.runtime.JobException: Creating the input > splits caused an error: java.lang.RuntimeException: > java.lang.ClassNotFoundException: Class > org.apache.hadoop.fs.s3native.NativeS3FileSystem not found > at > org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:172) > at > org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:696) > at org.apache.flink.runtime.jobmanager.JobManager.org > $apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1023) > ... 25 more > Caused by: java.lang.RuntimeException: java.lang.RuntimeException: > java.lang.ClassNotFoundException: Class > org.apache.hadoop.fs.s3native.NativeS3FileSystem not found > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227) > at > org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.getHadoopWrapperClassNameForFileSystem(HadoopFileSystem.java:460) > at > org.apache.flink.core.fs.FileSystem.getHadoopWrapperClassNameForFileSystem(FileSystem.java:352) > at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:280) > at org.apache.flink.core.fs.Path.getFileSystem(Path.java:311) > at > org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:450) > at > org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:57) > at > org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:156) > ... 27 more > Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: > Class org.apache.hadoop.fs.s3native.NativeS3FileSystem not found > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195) > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2219) > ... 34 more > Caused by: java.lang.ClassNotFoundException: Class > org.apache.hadoop.fs.s3native.NativeS3FileSystem not found > at > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101) > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193) > ... 35 more > > > Thanks > Ashutosh > >
