Well, the issue I'm trying to solve is slow writing due to S3's implementation of move as copy/delete. It seems like your S3 committers and S3Guard both ameliorate that somewhat by parallelizing the copy. I assume there's no better way to solve this issue without sacrificing safety. Even if there were, I couldn't use it, because I'm stuck on Spark 1.5 and there doesn't seem to be a way to force the use of a given output committer.
-- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Output-Committers-for-S3-tp21033p21049.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
