What was the elapsed time for IO. I.e. The sort ran in 23 min, how long did it take to read and write the file?
The sort is only part of the process.... <snip> http://opensource.com/business/15/1/apache-spark-new-world-record <quote> In October 2014, Databricks participated in the Sort Benchmark and set a new world record for sorting 100 terabytes (TB) of data, or 1 trillion 100-byte records. The team used Apache Spark <http://spark.apache.org/> on 207 EC2 virtual machines and sorted 100 TB of data in 23 minutes. </quote> Impressive to me. </snip> ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO IBM-MAIN
