Hi,
I am using Spark on AWS and want to write the output to S3. It is a
relatively small file and I don't want them to output as multiple parts. So
I use
result.repartition(1).saveAsTextFile("s3://...")
However as long as I am using the saveAsTextFile method, the output doesn't
keep the original order. But if I use BufferedWriter in Java to write the
output, I could only write to the master machine instead of S3 directly. Is
there a way that I could write to S3 and the same time keep the order?
Thanks a lot!
Anny
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-output-to-S3-and-keep-the-order-tp21246.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
