No with s3a, I have the following error : java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:285)
2015-07-27 11:17 GMT+02:00 Akhil Das <[email protected]>: > So you are able to access your AWS S3 with s3a now? What is the error that > you are getting when you try to access the custom storage with > fs.s3a.endpoint? > > Thanks > Best Regards > > On Mon, Jul 27, 2015 at 2:44 PM, Schmirr Wurst <[email protected]> > wrote: >> >> I was able to access Amazon S3, but for some reason, the Endpoint >> parameter is ignored, and I'm not able to access to storage from my >> provider... : >> >> sc.hadoopConfiguration.set("fs.s3a.endpoint","test") >> sc.hadoopConfiguration.set("fs.s3a.awsAccessKeyId","") >> sc.hadoopConfiguration.set("fs.s3a.awsSecretAccessKey","") >> >> Any Idea why it doesn't work ? >> >> 2015-07-20 18:11 GMT+02:00 Schmirr Wurst <[email protected]>: >> > Thanks, that is what I was looking for... >> > >> > Any Idea where I have to store and reference the corresponding >> > hadoop-aws-2.6.0.jar ?: >> > >> > java.io.IOException: No FileSystem for scheme: s3n >> > >> > 2015-07-20 8:33 GMT+02:00 Akhil Das <[email protected]>: >> >> Not in the uri, but in the hadoop configuration you can specify it. >> >> >> >> <property> >> >> <name>fs.s3a.endpoint</name> >> >> <description>AWS S3 endpoint to connect to. An up-to-date list is >> >> provided in the AWS Documentation: regions and endpoints. Without >> >> this >> >> property, the standard region (s3.amazonaws.com) is assumed. >> >> </description> >> >> </property> >> >> >> >> >> >> Thanks >> >> Best Regards >> >> >> >> On Sun, Jul 19, 2015 at 9:13 PM, Schmirr Wurst <[email protected]> >> >> wrote: >> >>> >> >>> I want to use pithos, were do I can specify that endpoint, is it >> >>> possible in the url ? >> >>> >> >>> 2015-07-19 17:22 GMT+02:00 Akhil Das <[email protected]>: >> >>> > Could you name the Storage service that you are using? Most of them >> >>> > provides >> >>> > a S3 like RestAPI endpoint for you to hit. >> >>> > >> >>> > Thanks >> >>> > Best Regards >> >>> > >> >>> > On Fri, Jul 17, 2015 at 2:06 PM, Schmirr Wurst >> >>> > <[email protected]> >> >>> > wrote: >> >>> >> >> >>> >> Hi, >> >>> >> >> >>> >> I wonder how to use S3 compatible Storage in Spark ? >> >>> >> If I'm using s3n:// url schema, the it will point to amazon, is >> >>> >> there >> >>> >> a way I can specify the host somewhere ? >> >>> >> >> >>> >> >> >>> >> --------------------------------------------------------------------- >> >>> >> To unsubscribe, e-mail: [email protected] >> >>> >> For additional commands, e-mail: [email protected] >> >>> >> >> >>> > >> >>> >> >>> --------------------------------------------------------------------- >> >>> To unsubscribe, e-mail: [email protected] >> >>> For additional commands, e-mail: [email protected] >> >>> >> >> > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
