Hi, I wonder if I do wordcount on two different files, like this:
val file1 = sc.textFile("/...")
val file2 = sc.textFile("/...")
val wc1= file.flatMap(..).reduceByKey(_ + _,1)
val wc2= file.flatMap(...).reduceByKey(_ + _,1)
wc1.saveAsTextFile("titles.out")
wc2.saveAsTextFile("tables.out")
Would the two reduceByKey stages run in parallel given sufficient
capacity?
Best regards,
Wei
---------------------------------
Wei Tan, PhD
Research Staff Member
IBM T. J. Watson Research Center
http://researcher.ibm.com/person/us-wtan
