As someone who mainly operates in AWS it would be very welcome to have the option to use an updated version of hadoop using pyspark sourced from pypi.
Acknowledging the issues of backwards compatability... The most vexing issue is the lack of ability to use s3a STS, ie org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider. This prevents the use of AWS temporary credentials, hampering local development against s3. Whilst this would be solved by bumping the hadoop version to anything >= 2.8.x, the 3.x option would also allow for the writing of data using KMS. Regards, George Pongracz -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
