On Sun, 12 Jul 2020 at 01:45, gpongracz <gpongr...@gmail.com> wrote: > As someone who mainly operates in AWS it would be very welcome to have the > option to use an updated version of hadoop using pyspark sourced from pypi. > > Acknowledging the issues of backwards compatability... > > The most vexing issue is the lack of ability to use s3a STS, ie > org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider. > > This prevents the use of AWS temporary credentials, hampering local > development against s3. > > I'd personally worry about other issue related to performance, security, Joda Time and java 8, etc. Hadoop 2.7.x is EOL and doesn't get security fixes any more.
If you do want that temporary credentials provider -you can stick a copy of the class on your classpath and just list it on the option fs.s3a.aws.credentials.provider > Whilst this would be solved by bumping the hadoop version to anything >= > 2.8.x, the 3.x option would also allow for the writing of data using KMS. > > Regards, > > George Pongracz > > > > -- > Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
