Re: Access to S3 from YARN on EC2

2016-03-21 Thread Ashutosh Kumar
Adding required jar files in flink/lib resolves the issue. On Mar 21, 2016 12:57 PM, "Balaji Rajagopalan" < balaji.rajagopa...@olacabs.com> wrote: > This kind of class not found exception is a little bit misleading, it is > not the class is not found is the real problem rather than the combination

Re: Access to S3 from YARN on EC2

2016-03-21 Thread Balaji Rajagopalan
This kind of class not found exception is a little bit misleading, it is not the class is not found is the real problem rather than the combination of the different libraries that are using there is a version compatibility mismatch, so you will have to go back and check if there is any version mism

Re: Access to S3 from YARN on EC2

2016-03-20 Thread Ashutosh Kumar
Hi Tim, I have this dependency in my pom file . This jar is present in my jar with dependencies. I exploded the jar and checked it. The class NativeS3FileSystem.class is present there. Thanks Ashutosh On Mon, Mar 21, 2016 at 7:20 AM, Timothy Farkas < timothytiborfar...@gmail.com> wrote: > Hi A

Re: Access to S3 from YARN on EC2

2016-03-20 Thread Timothy Farkas
Hi Ashutosh, I believe you need to add the hadoop-aws jar to your project. http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/2.6.0 Thanks, Tim On Sun, Mar 20, 2016 at 9:39 AM, Ashutosh Kumar wrote: > Do I need to add some jars in lib ? > > Thanks > Ashutosh > > On Sun, Mar 20, 20

Re: Access to S3 from YARN on EC2

2016-03-20 Thread Ashutosh Kumar
Do I need to add some jars in lib ? Thanks Ashutosh On Sun, Mar 20, 2016 at 4:30 PM, Ashutosh Kumar wrote: > It is not there. > > Thanks > Ashutosh > > On Sun, Mar 20, 2016 at 2:58 PM, Robert Metzger > wrote: > >> Hi, >> >> did you check if the "org.apache.hadoop.fs.s3native.NativeS3FileSystem

Re: Access to S3 from YARN on EC2

2016-03-20 Thread Ashutosh Kumar
It is not there. Thanks Ashutosh On Sun, Mar 20, 2016 at 2:58 PM, Robert Metzger wrote: > Hi, > > did you check if the "org.apache.hadoop.fs.s3native.NativeS3FileSystem" > class is in the flink-dist.jar in the lib/ folder? > > > On Sun, Mar 20, 2016 at 10:19 AM, Ashutosh Kumar < > ashutosh.disc

Re: Access to S3 from YARN on EC2

2016-03-20 Thread Robert Metzger
Hi, did you check if the "org.apache.hadoop.fs.s3native.NativeS3FileSystem" class is in the flink-dist.jar in the lib/ folder? On Sun, Mar 20, 2016 at 10:19 AM, Ashutosh Kumar wrote: > I have setup a 3 node YARN based cluster on EC2. I am running flink in > cluster mode. I added these lines in

Access to S3 from YARN on EC2

2016-03-20 Thread Ashutosh Kumar
I have setup a 3 node YARN based cluster on EC2. I am running flink in cluster mode. I added these lines in core-site.xml fs.s3n.awsAccessKeyId accesskey fs.s3n.awsSecretAccessKey secret key fs.s3n.impl org.apache.hadoop.f