> On 30 Dec 2015, at 19:31, KOSTIANTYN Kudriavtsev > <[email protected]> wrote: > > Hi Jerry, > > I want to run different jobs on different S3 buckets - different AWS creds - > on the same instances. Could you shed some light if it's possible to achieve > with hdfs-site? > > Thank you, > Konstantin Kudryavtsev >
The Hadoop s3a client doesn't have much (anything?) in the way for multiple logins. It'd be possible to do it by hand (create a Hadoop Configuration object, fill with the credential, and set "fs.s3a.impl.disable.cache"= true to make sure you weren't getting an existing version. I don't know how you'd hook that up to spark jobs. maybe try setting the credentials and that fs.s3a.impl.disable.cache flag in your spark context to see if together they get picked up --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
